Gilles Fedak

XtremWeb: a generic global computing system

Global computing achieves high throughput computing by harvesting a very large number of unused computing resources connected to the Internet. This parallel computing model targets a parallel architecture defined by a very high number of nodes, poor communication performance and continuously varying resources. The unprecedented scale of the global computing architecture paradigm requires us to revisit many basic issues related to parallel architecture programming models, performance models, and class of applications or algorithms suitable for this architecture. XtremWeb is an experimental global computing platform dedicated to provide a tool for such studies. The paper presents the design of XtremWeb. Two essential features of this design are multi-applications and high-performance. Accepting multiple applications allows institutions or enterprises to set up their own global computing applications or experiments. High-performance is ensured by scalability, fault tolerance, efficient scheduling and a large base of volunteer PCs. We also present an implementation of the first global application running on XtremWeb.

Characterizing resource availability in enterprise desktop grids

Desktop grids, which use the idle cycles of many desktop PCs, are one of the largest distributed systems in the world. Despite the popularity and success of many desktop grid projects, the heterogeneity and volatility of hosts within desktop grids have been poorly understood. Yet, resource characterization is essential for accurate simulation and modelling of such platforms. In this paper, we present application-level traces of four real desktop grids that can be used for simulation and modelling purposes. In addition, we describe aggregate and per host statistics that reflect the heterogeneity and volatility of desktop grid resources. Finally, we apply our characterization to develop a performance model for desktop grid applications for various task granularities, and then use a cluster equivalence metric to quantify the utility of the desktop grid relative to that of a dedicated cluster for task-parallel applications.

EDGeS: Bridging EGEE to BOINC and XtremWeb

Desktop Grids, such as XtremWeb and BOINC, and Service Grids, such as EGEE, are two different approaches for science communities to gather computing power from a large number of computing resources. Nevertheless, little work has been done to combine these two Grid technologies in order to establish a seamless and vast Grid resource pool. In this paper we present the EGEE Service Grid, the BOINC and XtremWeb Desktop Grids. Then, we present the EDGeS solution to bridge the EGEE Service Grid with the BOINC and XtremWeb Desktop Grids.

Characterizing result errors in internet desktop grids

Desktop grids use the free resources in Intranet and Internet environments for large-scale computation and storage. While desktop grids offer a high return on investment, one critical issue is the validation of results returned by participating hosts. Several mechanisms for result validation have been previously proposed. However, the characterization of errors is poorly understood. To study error rates, we implemented and deployed a desktop grid application across several thousand hosts distributed over the Internet. We then analyzed the results to give quantitative and empirical characterization of errors stemming from input or output (I/O) failures. We find that in practice, error rates are widespread across hosts but occur relatively infrequently. Moreover, we find that error rates tend to not be stationary over time nor correlated between hosts. In light of these characterization results, we evaluated state-of-the-art error detection mechanisms and describe the trade-offs for using each mechanism.

Towards MapReduce for Desktop Grid Computing

MapReduce is an emerging programming model for data-intense application proposed by Google, which has attracted a lot of attention recently. MapReduce borrows from functional programming, where programmer defines Map and Reduce tasks executed on large set of distributed data. In this paper we propose an implementation of the MapReduce programming model. We present the architecture of the prototype based on Bit Dew, a middleware for large scale data management on Desktop Grid. We describe the set of features which makes our approach suitable for large scale and loosely connected Internet Desktop Grid: massive fault tolerance, replica management, barriers-free execution, latency-hiding optimisation as well as distributed result checking. We also present performance evaluation of the prototype both against micro-benchmarks and real MapReduce application. The scalability test shows that we achieve linear speedup on the classical Word Count benchmark. Several scenarios involving lagger hosts and host crashes demonstrate that the prototype is able to cope with an experimental context similar to real-world Internet.

XtremWeb & Condor : sharing resources between Internet connected Condor pool

Grid computing presents two major challenges for deploying large scale applications across wide area networks gathering volunteers PC and clusters/parallel computers as computational resources: security and fault tolerance. This paper presents a lightweight Grid solution for the deployment of multi-parameters applications on a set of clusters protected by firewalls. The system uses a hierarchical design based on Condor for managing each cluster locally and XtremWeb for enabling resource sharing among the clusters. We discuss the security and fault tolerance mechanisms used for this design and demonstrate the usefulness of the approach measuring the performances of a multi-parameters bio-chemistry application deployed on two sites: University of Wisconsin/Madison and Paris South University. This experiment shows that we can efficiently and safely harness the computational power of about 200 PC distributed on two geographic sites.

BitDew: A data management and distribution service with multi-protocol file transfer and metadata abstraction

Desktop Grids use the computing, network and storage resources from idle desktop PCs distributed over multiple-LANs or the Internet to compute a large variety of resource-demanding distributed applications. While these applications need to access, compute, store and circulate large volumes of data, little attention has been paid to data management in such large-scale, dynamic, heterogeneous, volatile and highly distributed Grids. In most cases, data management relies on ad hoc solutions, and providing a general approach is still a challenging issue. A new class of data management service is desirable to deal with such a variety of file transfer protocols than client/server, P2P or the new and emerging Cloud storage service. To address this problem, we propose the BitDew framework, a programmable environment for automatic and transparent data management on computational Desktop Grids. This paper describes the BitDew programming interface, its architecture, and the performance evaluation of its runtime components. BitDew relies on a specific set of metadata to drive key data management operations, namely life cycle, distribution, placement, replication and fault tolerance with a high level of abstraction. The BitDew runtime environment is a flexible distributed service architecture that integrates modular P2P components such as DHTs (Distributed Hash Tables) for a Distributed Data Catalog and collaborative transport protocols for data distribution. We explain how to plug-in new or existing protocols and we give evidence of the versatility of the framework by implementing HTTP, FTP and BitTorrent protocols and access to the Amazon S3 and IBP Wide Area Storage. We describe the mechanisms used to provide asynchronous and reliable multi-protocols transfers. Through several examples, we describe how application programmers and BitDew users can exploit BitDews features. We report on performance evaluation using micro-benchmarks, various usage scenarios and data-intense bioinformatics application, both in the Grid context and on the Internet. The performance evaluation demonstrates that the high level of abstraction and transparency is obtained with a reasonable overhead, while offering the benefit of scalability, performance and fault tolerance with little programming cost.

Scheduling independent tasks sharing large data distributed with BitTorrent

Data-centric applications are still a challenging issue for large scale distributed computing systems. The emergence of new protocols and software for collaborative content distribution over Internet offers a new opportunity for efficient and fast delivery of high volume of data. In a previous paper, we have investigated BitTorrent as a protocol for data diffusion in the context of computational desktop grid. We showed that BitTorrent is efficient for large file transfers, scalable when the number of nodes increases but suffers from a high overhead when transmitting small files. This paper investigates two approach to overcome these limitations. First, we propose a performance model to select the best of FTP and BitTorrent protocols according to the size of the file to distribute and the number of receiver nodes. Next we propose enhancement of the BitTorrent protocol which provides more predictable communication patterns. We design a model for communication performance and evaluate BitTorrent-aware versions BT-MinMin, BT-MaxMin and BT-Sufferage scheduling heuristics against a synthetic parameter-sweep application.

Distributed Results Checking for MapReduce in Volunteer Computing

MapReduce is a promising approach to support data-intensive applications on Volunteer Computing Systems. Existent middleware like Bit Dew allows running MapReduce applications in a Desktop Grid environment. If the Desktop Grid is deployed in the Internet under the Volunteer Computing paradigm, it harnesses untrustable, volatile and heterogeneous resources and the results produced by MapReduce applications can be subject of sabotage. However, the implementation of large-scale MapReduce presents significant challenges with respect to the state of the art in Desktop Grid. A key issue is the design of the result certification, an operation needed to verify that malicious volunteers do not tamper with the results of computations. Because the volume of data produced and processed is so large that cannot be sent back to the server, the result certification cannot be centralized as it is currently implemented in Desktop Grid systems. In this paper we present a distributed result checker based on the Majority Voting method. We evaluate the efficiency of our approach using a model for characterizing errors and sabotage in the MapReduce paradigm. With this model, we can compute the aggregated probability with which a MapReduce implementation produces an erroneous result. The challenge is to capture the aggregated probability for the entire system, composed from probabilities resulted from the two phases of computation: Map and Reduce. We provide a detailed analysis on the performance of the result verification method and also discuss the generated overhead of managing security. We also give guidelines about how the result verification phase should be configured, given a MapReduce application.

Collaborative Data Distribution with BitTorrent for Computational Desktop Grids

Data-centric applications are still a challenging issue for large scale distributed computing systems. The emergence of new protocols and software for collaborative content distribution over Internet offers a new opportunity for efficient and fast delivery of high volume of data. This paper presents an evaluation of the BitTorrent protocol for computational desktop grids. We first present a prototype of a generic subsystem dedicated to data management and designed to serve as a building block for any desktop grid system. Based on this prototype we conduct experimentations to evaluate the potential of BitTorrent compared to a classical approach based on FTP data server. The preliminary results obtained with a 65-nodes cluster measure the basic characteristics of BitTorrent in terms of latency and bandwidth and evaluate the scalability of BitTorrent for the delivery of large input files. Moreover, we show that BitTorrent has a considerable latency overhead compared to FTP but clearly outperforms FTP when distributing large files or files to a high number of nodes. Tests on a synthetic application show that BitTorrent significantly increases the communication/computation ratio of the applications eligible to run on a desktop grid system