Cyril Briquet

Scheduling data-intensive bags of tasks in P2P grids with bittorrent-enabled data distribution

Scheduling Data-Intensive Bags of Tasks in P2P Grids leads to transfers of large input data files, which cause delays in completion times. We propose to combine several existing technologies and patterns to perform efficient data-aware scheduling: (1) use of the BitTorrent P2P file sharing protocol to transfer data, (2) data caching on computational Resources, (3) use of a data-aware Resource selection scheduling algorithm similar to Storage Affinity, (4) a new Task selection scheduling algorithm (Temporal Tasks Grouping), based on the temporally grouped scheduling of Tasks sharing input data files. Data replication is also discusse. The proposed approach does not need an overlay network or Predictive Communications Ordering, making our operational implementation of a P2P Grid middleware easily deployable in unstructured P2P networks. Experiments show that performance gains are achieved by combining BitTorrent, caching, Storage Affinity and Temporal Tasks Grouping. This work can be summarized as combining P2P Grid computing and P2P data transfer technologies.

A grid-enabled Lattice-Boltzmann-based modelling system

Lattice-Boltzmann (LB) methods are a well-known technique in the context of computational fluid dynamics. By nature, they can easily be parallelized but their adaptation to the Grid environment is not trivial due to hardware heterogeneity (CPU, memory...) in a Grid. A load balancing method to dynamically handle the differences in terms of CPU number and power among the machines of a Grid is presented. The CPU power is dynamically estimated using a benchmark. An estimation method of execution time is also given.

Reproducible testing of distributed software with middleware virtualization and simulation

P2P Grids are Grids organized into P2P networks where participant exchange computing time so as to complete computational tasks. Evaluating the performance of scheduling algorithms enables one to deploy those that are efficient. Performance is often evaluated experimentally or through simulation because these algorithms (typically heuristics) are too complex to model analytically. Testing the implementation of P2P Grid middleware before it is deployed is also important: Reproducing configurations or conditions that lead to unexpected outcomes is thus valuable. A P2P Grid environment exhibits multiple sources of failure and is typically dynamic and uncontrollable. Reproducing even basic behavior of Grid nodes in a controllable and repeatable manner is thus exceedingly difficult. Such lack of control over the environment is a major challenge in the software engineering of P2P Grid middleware [7]. Simulators have been proposed to evaluate the performance of scheduling algorithms, but are often limited in scope, reusability and accuracy, i.e. they rely on simplified models. We introduce a software engineering pattern - that we call code once, deploy twice - to both reduce the distance between simulated and implemented algorithms and reproduce, at will, Grid configurations and environments: A simulator implementation of a Grid architecture is built by virtualizing its middleware implementation. An immediate benefit is that most of the code can be reused between both implementations; only communications between Grid nodes, multithreading within Grid nodes and actual task execution are coded differently. As a derived benefit, most of the code of the middleware can be tested within the controlled environment of the simulator, before it is deployed as-is. Another benefit is high simulation accuracy. We describe the implementation of a P2P Grid following the code once, deploy twice pattern, that we believe is also relevant to other Grid types (certainly Volunteer Grids [5, 4] and Desktop Grids [22], and possibly Globus-based Grids [3]).

Approximate policy iteration for closed-loop learning of visual tasks

Approximate Policy Iteration (API) is a reinforcement learning paradigm that is able to solve high-dimensional, continuous control problems. We propose to exploit API for the closed-loop learning of mappings from images to actions. This approach requires a family of function approximators that maps visual percepts to a real-valued function. For this purpose, we use Regression Extra-Trees, a fast, yet accurate and versatile machine learning algorithm. The inputs of the Extra-Trees consist of a set of visual features that digest the informative patterns in the visual signal. We also show how to parallelize the Extra-Tree learning process to further reduce the computational expense, which is often essential in visual tasks. Experimental results on real-world images are given that indicate that the combination of API with Extra-Trees is a promising framework for the interactive learning of visual tasks.

LBG-SQUARE Fault-Tolerant, Locality-Aware Co-Allocation in P2P Grids

In this paper, the deployment and execution of iterative stencil applications on a P2P grid middleware are investigated. So-called iterative stencil applications are composed of sets of heavily-communicating, long-running tasks. They thus require co-allocation of multiple reliable resources for extended periods of time. P2P grids are totally decentralized and provide on-demand, transparent access to edge resources, e.g. Internet-connected, non-dedicated desktop computers. A P2P grid has the potential to provide access to a large number of resources at the fraction of the cost of a dedicated cluster. However, edge resources are heterogeneous in performance and intrinsically unreliable: task execution failures are common due to resource preemption or resource failure. Furthermore, P2P grid schedulers usually target sets of independent computational Tasks, i.e. so-called Bags of Tasks applications. It is therefore not trivial to deploy and run an iterative stencil application on a P2P grid. Checkpointing is a common fault-tolerance mechanism in high performance distributed computing, often based on a centralized architecture. Locality-aware co-allocation in P2P grids has been recently investigated. Checkpointing and locality-aware co-allocation yet have to be integrated in P2P grids. We propose to provide co-allocation through an existing middleware-level Bag of Tasks scheduling mechanism. We also introduce a layer of fault-tolerance for the iterative stencils that relies on a scalable, application-level, P2P checkpointing mechanism. Finally, LBG-SQUARE is described. This software results from the combination of a specific Iterative Stencil application (a computational fluid dynamics simulation software called LaBoGrid) with a P2P grid middleware (Lightweight Bartering Grid).

P2P file sharing for P2P computing

The transfer of large input data files in P2P computing Grids often leads to delays in Task completion times. Existing research related to this topic has been focused on the spatial grouping of Tasks, i.e. reuse of available data through data caching and data-aware scheduling. However, it tends to decrease the level of parallelism of Task execution. In this paper, this issue is addressed by integrating the BitTorrent P2P file sharing protocol, a novel Task selection scheduling algorithm, an existing online, data-aware Resource selection algorithm (similar to Storage Affinity), and caching support. These algorithms have been implemented in the Lightweight Bartering Grid middleware. The Java implementation relies exclusively on Free and Open Source data transfer software (Azureus, Apache FTP server, edtFTPj). The proposed data transfer architecture does not need Predictive Communications Ordering or an explicit deployment of an overlay network. It is also easily deployable. Our main contribution is the joint use of P2P computing and P2P file sharing technologies, enabling a highly scalable and adaptive data transfer architecture to support P2P computing.