Heithem Abbes

PastryGrid : decentralisation of the execution of distributed applications in desktop grid

This paper proposes a decentralised system for managing Desktop Grid (DG). The idea is to bypass the main drawback of existing systems putting all the control on a single master that can fails. Here, each node can play alternatively the role of client or server. Our main contribution is to design the PastryGrid protocol (based on Pastry) for DG in order to decentralise the execution of a distributed application with precedence between tasks. Comparing to a centralised system, we evaluate our approach over 205 machines executing 2500 tasks. The results show that our decentralised system runs better than the same system configured as a master/slave because it gives less overhead.

Fault tolerance based on the publish-subscribe paradigm for the BonjourGrid middleware

How to federate the machines of all Boinc, Condor and XtremWeb projects? If you believe in volunteer computing and want to share more than one project then BonjourGrid may help. In previous works, we proposed a novel approach, called BonjourGrid, to orchestrate multiple instances of Institutional Desktop Grid middleware. It is our way to remove the risk of bottleneck and failure, and to guarantee the continuity of services in a distributed manner. Indeed, BonjourGrid can create a specific environment for each user based on a given computing system of his choice such as XtremWeb, Condor or Boinc. This work investigates, first, the procedure to deploy Boinc and Condor on top of BonjourGrid and, second, proposes a fault tolerant approach based on passive replication and virtualization to tolerate the crash of coordinators. The novelty resides here in an integrated environment based on Bonjour (publication-subscription mecanism) for both the coordination protocol and for the fault tolerance issues. In particular, it is not so frequent to our knowledge to describe and to implement a fault tolerant protocol according to the pub-sub paradigm. Experiments, conducted on the Grid5000 testbed, illustrate a comparative study between Boinc (respectively Condor) on top of BonjourGrid and a centralized system using Boinc (respectively Condor) and second prove the robustness of the fault tolerant mechanism.

Toward a Meta-Grid Middleware

Institutional Desktop Grid systems are attractive for running distributed applications with significant computational requirements. While the rapid increasing number of users and applications running on such systems does demonstrate the potential of Desktop Grid, current implementations follow the old-fashioned master-worker paradigm. Obviously, vulnerability to failures and permanent administrative monitoring are the disadvantages of client-server architectures. Moreover, it is important to exploit existing computing systems in order to make a meta-grid middleware able to support any kind of applications. To bypass this, we propose a novel system, called BonjourGrid, able to orchestrate multiple instances of Institutional Desktop Grid middlewares, able to remove the risk of single-source bottleneck and failure, and able to guaranty the continuity of services in a distributed manner. We choose XtremWeb-CH, Boinc and Condor as computing systems in BonjourGrid. Thus, BonjourGrid can create a specific environment for each user based on XtremWeb-CH, Boinc or Condor. In addition, BonjourGrid can be adapted to fulfill all the requirements of a decentralized job scheduler.nIn this paper, we set experimentations using the three cited systems: Boinc, Condor and XtremeWeb-CH, in order to analyze the overhead generated by BonjourGrid and its capacity to manage multiple instances of Desktop Grid. The evaluation proves that BonjourGrid is able to manage more than 400 applications instantiated in a concurrent way on an Institutional Desktop Grid using more than 1000 machines. Analyzing the execution of 405 applications with 2110 tasks during 3 hours demonstrates the potential of BonjourGrid concept and shows that, comparing to a classical Desktop Grid, such as Condor, Boinc and XtremWeb-CH, with one central master, BonjourGrid gives an acceptable overhead.

Wide Area BonjourGrid as a Data Desktop Grid: Modeling and Implementation on Top of Redis

Desktop Grid is among the success stories during last years by using volunteers nodes participating into projects. Now, with the emergence of Cloud Computing, the questions become where to take resources? and how to coordinate the resources? Our assumption is that Desktop Grid will continue to survive if we are able to transform the old-fashioned client/server architecture to new web oriented architecture to deliver services on demand. This paper revisits and extends the coordination protocol of BonjourGrid, a decentralized desktop grid system, based on the Publish-Subscribe paradigm and including a new tier for data management. The new protocol is designed according to a formal modelling using colored Petri nets. The protocol is veried and proved by CPN-Tools and implemented with Redis, a polpular net technology. We conducted out experiments on the Grid5000 testbed using 300 nodes. We analyze the Redis performance and we demonstrate that the extended version of BonjourGrid system is fully operational.

LXCloud-CR: Towards LinuX Containers Distributed Hash Table based Checkpoint-Restart

Abstract Infrastructure-as-a-Service (IaaS) container-based virtualization technology is gaining, over these years, significant interest in industry as an alternative platform for running distributed applications. They present an interesting alternative to virtual machines in the Cloud due to newer advances in container-based virtualization as a new technology that simplify the deployment of applications. As the cloud architectures continue to grow in scale and complexity, faults become very recurrent which make reliability a true challenge. Given the dynamic nature of IaaS clouds and the pay-as-you-go cloud model where the costs are directly proportional to the resource usage, a Checkpoint Restart mechanism is essential in this context. We propose LXCloud-CR our new decentralized Checkpoint-Restart model based on a distributed checkpoints repository using key–value store on a Distributed Hash Table (DHT). It is able to take snapshots of the whole Linux Container (LXC) instances. LXCloud-CR aims to reduce the runtime and storage overheads of checkpointing. Large scale experiments on the Grid’5000 testbed demonstrate the benefits of our proposal. Obtained results validate our model and improve the performance of applications.

PastryGridCP: A Decentralized Rollback-Recovery Protocol for Desktop Grid Systems

Desktop Grids are composed of several thousands of resources. They are characterized by high volatility of resources, due to voluntary disconnections or failures. This could affect the proper termination of applications execution. PastryGrid is a decentralized system which manages desktop grid resources and user applications over a fully decentralized P2P network. In this paper we present PastryGridCP: our rollback-recovery protocol, which is based on checkpoints designed for the decentralized Desktop Grid system PastryGrid. It provides fault tolerance for grid applications and ensures the termination of the execution of applications in a transparent way to users. We have conducted out experimentations on 110 nodes of Grid5000. Obtained results validate our protocol and improve the performance of applications.

LXCloudFT: Towards high availability, fault tolerant Cloud system based Linux Containers

Abstract Infrastructure-as-a-Service container-based virtualization is gaining interest as a platform for running distributed applications. With increasing scale of Cloud architectures, faults are becoming a frequent occurrence, which makes availability a challenge. LXCloudFT is a fault tolerant Cloud system, which is composed of LXCloud-CR, a Checkpoint–Restart model and GC-CR, a garbage collector component that eliminates old snapshots of containers. LXCloudFT is designed, originally, for scientific applications and all its components are decentralized. We want to adapt it to serve stateless loosely coupled applications such as web applications. Replication is a method to survive failures for such applications. This paper addresses the issue of replication and contributes with a novel replication model, LXCloud-Rep, in LXCloudFT. LXCloud-Rep is a replication model with versioning and garbage collection, which is able to replicate Linux Container instances on several nodes in a decentralized manner. Following a node failure, LXCloud-Rep restarts failed containers on a new node from distributed images of containers not from snapshots. It optimizes the use of storage space. Large-scale experiments on Grid’5000 improve the performance of applications.

PGTrust: a decentralized free-riding prevention model for DG systems

Desktop grids (DG) offer large amounts of computing power coming from internet-based volunteer networks. They suffer from the free-riding phenomenon. It may be possible for users to free ride, consuming resources donated by others but not donating any of their own. In this paper, we present PGTrust: our decentralized free-riding prevention model designed for PastryGrid. PastryGrid is a decentralized DG system which manages resources over a decentralized P2P network. PGTrust relies on the notion of score which is a metric of reputation used to evaluate the level of QoS of a peer. We have conducted out experimentations on Grid’5000 testbed. Obtained results prove the benefits of our free-riding prevention model. PGTrust is able to improve application running time by discouraging free-riders and motivating selfish peers to contribute. It offers a considerable speedup over distributed applications.

D3-MapReduce: Towards MapReduce for Distributed and Dynamic Data Sets

Since its introduction in 2004 by Google, MapReduce has become the programming model of choice for processing large data sets. Although MapReduce was originally developed for use by web enterprises in large data-centers, this technique has gained a lot of attention from the scientific community for its applicability in large parallel data analysis (including geographic, high energy physics, genomics, etc.). So far MapReduce has been mostly designed for batch processing of bulk data. The ambition of D3-MapReduce is to extend the MapReduce programming model and propose efficient implementation of this model to: i) cope with distributed data sets, i.e. that span over multiple distributed infrastructures or stored on network of loosely connected devices, ii) cope with dynamic data sets, i.e. which dynamically change over time or can be either incomplete or partially available. In this paper, we draw the path towards this ambitious goal. Our approach leverages Data Life Cycle as a key concept to provide MapReduce for distributed and dynamic data sets on heterogeneous and distributed infrastructures. We first report on our attempts at implementing the MapReduce programming model for Hybrid Distributed Computing Infrastructures (Hybrid DCIs). We present the architecture of the prototype based on BitDew, a middleware for large scale data management, and Active Data, a programming model for data life cycle management. Second, we outline the challenges in term of methodology and present our approaches based on simulation and emulation on the Grid5000 experimental testbed. We conduct performance evaluations and compare our prototype with Hadoop, the industry reference MapReduce implementation. We present our work in progress on dynamic data sets that has lead us to implement an incremental MapReduce framework. Finally, we discuss our achievements and outline the challenges that remain to be addressed before obtaining a complete D3-MapReduce environment.

Towards Privacy for MapReduce on Hybrid Clouds Using Information Dispersal Algorithm

MapReduce is a powerful model for parallel data processing. The motivation of this work is to allow running map-reduce jobs partially on untrusted infrastructures, such as public clouds and desktop grid, while using a trusted infrastructure, such as private cloud, to ensure that no outsider could get the ’entire’ information. Our idea is to break data into meaningless chunks and spread them on a combination of public and private clouds so that the compromise would not allow the attacker to reconstruct the whole data-set. To realize this, we use the Information Dispersion Algorithms (IDA), which allows to split a file into pieces so that, by carefully dispersing the pieces, there is no method for a single node to reconstruct the data if it cannot collaborate with other nodes. We propose a protocol that allows MapReduce computing nodes to exchange the data and perform IDA-aware MapReduce computation. We conduct experiments on the Grid’5000 testbed and report on performance evaluation of the prototype.