Dominic Battré

Nephele/PACTs: a programming model and execution framework for web-scale analytical processing

We present a parallel data processor centered around a programming model of so called Parallelization Contracts (PACTs) and the scalable parallel execution engine Nephele [18]. The PACT programming model is a generalization of the well-known map/reduce programming model, extending it with further second-order functions, as well as with Output Contracts that give guarantees about the behavior of a function. We describe methods to transform a PACT program into a data flow for Nephele, which executes its sequential building blocks in parallel and deals with communication, synchronization and fault tolerance. Our definition of PACTs allows to apply several types of optimizations on the data flow during the transformation. The system as a whole is designed to be as generic as (and compatible to) map/reduce systems, while overcoming several of their major weaknesses: 1) The functions map and reduce alone are not sufficient to express many data processing tasks both naturally and efficiently. 2) Map/reduce ties a program to a single fixed execution strategy, which is robust but highly suboptimal for many tasks. 3) Map/reduce makes no assumptions about the behavior of the functions. Hence, it offers only very limited optimization opportunities. With a set of examples and experiments, we illustrate how our system is able to naturally represent and efficiently execute several tasks that do not fit the map/reduce model well.

On triple dissemination, forward-chaining, and load balancing in DHT based RDF stores

The Resource Description Framework provides a powerful model for structured knowledge representation that allows the inference of new knowledge. Because of the anticipated scope of semantic information available in the future, centralized databases will become incapable of handling the load. Peer-to-Peer based distributed databases offer better scalability and integration of many different data sources. In this paper we present a detailed data management strategy for a DHT based RDF store that provides reasoning, robustness, and load-balancing.

A proposal for WS-Agreement Negotiation

The Web Services Agreement specification defines a normative language to formulate Service Level Agreements and a basic protocol to expose service-level descriptions, validate service-level requests, and come to an agreement. This protocol, often called “take-it-or-leave-it”, allows a service provider and a service consumer to decide whether to accept or reject a service offer. Although this approach is sufficient for a number of use cases, others exist with requirements for multi-step negotiation or the adaptation of an existing agreement. In this paper, we describe the Web Services Agreement Negotiation protocol, a proposal by the Open Grid Forum to extend the existing specification. This proposal is the result of combining various research activities that have been conducted to define protocols for negotiating service levels or to supersede the existing “take-it-or-leave-it” protocol. The main characteristics of this proposal are the multi-round negotiation capability, renegotiation capability, and compliance with the original specification.

Detecting bottlenecks in parallel DAG-based data flow programs

In recent years, several frameworks have been introduced to facilitate massively-parallel data processing on shared-nothing architectures like compute clouds. While these frameworks generally offer good support in terms of task deployment and fault tolerance, they only provide poor assistance in finding reasonable degrees of parallelization for the tasks to be executed. However, as billing models of clouds enable the utilization of many resources for a short period of time for the same cost as utilizing few resources for a long time, proper levels of parallelization are crucial to achieve short processing times while maintaining good resource utilization and therefore good cost efficiency. In this paper, we present and evaluate a solution for detecting CPU and I/O bottlenecks in parallel DAG-based data flow programs assuming capacity constrained communication channels. The detection of bottlenecks represents an important foundation for manually or automatically scaling out and tuning parallel data flow programs in order to increase performance and cost efficiency.

Evaluation of Network Topology Inference in Opaque Compute Clouds through End-to-End Measurements

Modern Infrastructure-as-a-Service (IaaS) clouds offer an unprecedented flexibility and elasticity in terms of resource provisioning through the use of hardware virtualization. However, for the cloud customer, this virtualization also introduces an opaqueness which imposes serious obstacles for data-intensive distributed applications. In particular, the lack of network topology information, i.e. information on how the rented virtual machines are physically interconnected, can easily cause network bottlenecks as common techniques to exploit data locality cannot be applied. In this paper we study to what extent the underlying network topology of virtual machines inside an IaaS cloud can be inferred based on end-to-end measurements. Therefore, we experimentally evaluate the impact of hardware virtualization on the measurable link characteristics packet loss and delay using the popular open source hyper visors KVM and XEN. Afterwards, we compare the accuracy of different topology inference approaches and propose an extension to improve the inference accuracy for typical network structures in datacenters. We found that common assumptions for end-to-end measurements do not hold in presence of virtualization and that RTT-based measurements in Para virtualized environments lead to the most accurate inference results.

Top k RDF query evaluation in structured P2P networks

Berners-Lees vision of the Semantic Web describes the idea of providing machine readable and processable information using key technologies such as ontologies and automated reasoning in order to create intelligent agents. The prospective amount of machine readable information available in the future will be large. Thus, heterogeneity and scalability will be central issues, rendering exhaustive searches and central storage of data infeasible. This paper presents a scalable peer-to-peer based approach to distributed querying of Semantic Web information that allows ordering of entries in result sets and limiting the size of result sets which is necessary to prevent results with millions of matches. The system relies on the graph-based W3C standard Resource Description Framework (RDF) for knowledge description. Thereby, it enables queries on large, distributed RDF graphs.

Caching of intermediate results in DHT-based RDF stores

In the recent past, several peer-to-peer based RDF stores have emerged in order to cope with large amounts of RDF data that are queried by many clients concurrently. Efficient query evaluation is a central issue in order to exploit the benefits of such a decentralised approach. This paper shows how caching of intermediate results can significantly reduce the utilisation of network communication, which is currently among the most serious bottlenecks of Distributed Hash Table (DHT)-based RDF stores. The approach presented uses a combination of query shipping and online query planning to follow paths of cached intermediate results from previous queries.

Quality assurance of Grid service provisioning by risk aware managing of resource failures

Quality assurance is a key aspect in scope of the provisioning of grid services since end-users ask for specific quality of service (QoS) criteria defined in service level agreements (SLA). To commit to an SLA, grid providers need a risk analysis during SLA negotiation in order to estimate the probability of an SLA violation. In addition, such a risk analysis is necessary in the post-negotiation phase, in order to find the most profitable solution if not all SLAs can be fulfilled. Current job failure rates in grids (10-45%) highlight the necessity of fault-tolerance mechanisms. If not enough resources exist to compensate for all resource outages, the provider has to prefer those jobs which are in expectation the most profitable ones. Hence, this quality assurance ensures that obligations from the most important jobs will be fulfilled.

Lessons Learned from Implementing WS-Agreement

WS-Agreement describes a protocol and structure for creating and representing service level agreements. In order to remain domain independent, the authors of the WS-Agreement specification have provided many extension points for domain specific content. This creates high degrees of freedoms for programmers to implement the specification. Many attempts to do this have been made in the past. In this paper, we explain what we have learned from our own and other projects’ attempts of implementing WS-Agreement. The paper presents a set of guidelines how the features of WS-Agreement can be used in a sound way that allows transferring large parts of the WS-Agreement logic into a generic and domain-independent WS-Agreement framework.

Papnet: A Proximity-aware Alphanumeric Overlay Supporting Ganesan On-Line Load Balancing

Many structured Peer-to-Peer-Systems (P2P) have been developed over the past years. However, most of them rely on hash-functions and thus put major restrictions on applications being implemented on top of them. In this paper we present a very detailed description of Papnet, a hash-free P2P overlay-network that supports range-queries and realizes an infinite alphanumeric address space that can be used to store arbitrarily skewed data. We show how Papnet is able to distribute load amongst its nodes using the Ganesan On-Line Load Balancing providing a constant imbalance ratio while still being able to perform proximity routing, reaching each participating node with a latency being only roughly twice the direct latency. Further, we introduce a join-algorithm that provides Papnet with high fault-tolerance properties. We evaluated Papnet in a real distributed environment by setting up a network consisting of 50,000 nodes.