Francisco Vilar Brasileiro

Trading Cycles for Information: Using Replication to Schedule Bag-of-Tasks Applications on Computational Grids

Scheduling independent tasks on heterogeneous environments, like grids, is not trivial. To make a good scheduling plan on this kind of environments, the scheduler usually needs some information such as host speed, host load, and task size. This kind of information is not always available and is often difficult to obtain. In this paper we propose a scheduling approach that does not use any kind of information but still delivers good performance. Our approach uses task replication to cope with the dynamic and heterogeneous nature of grids without depending on any information about machines or tasks. Our results show that task replication can deliver good and stable performance at the expense of additional resource consumption. By limiting replication, however, additional resource consumption can be controlled with little effect on performance.

OurGrid: An Approach to Easily Assemble Grids with Equitable Resource Sharing

Available grid technologies like the Globus Toolkit make possible for one to run a parallel application on resources distributed across several administrative domains. Most grid computing users, however, don’t have access to more than a handful of resources onto which they can use this technologies. This happens mainly because gaining access to resources still depends on personal negotiations between the user and each resource owner of resources. To address this problem, we are developing the OurGrid resources sharing system, a peer-to-peer network of sites that share resources equitably in order to form a grid to which they all have access. The resources are shared accordingly to a network of favors model, in which each peer prioritizes those who have credit in their past history of bilateral interactions. The emergent behavior in the system is that peers that contribute more to the community are prioritized when they request resources. We expect, with OurGrid, to solve the access gaining problem for users of bag-of-tasks applications (those parallel applications whose tasks are independent).

Labs of the World, Unite!!!

AbstracteScience is rapidly changing the way we do research. As a result, many research labs now need non-trivial computational power. Grid and voluntary computing are well-established solutions for this need. However, not all labs can effectively benefit from these technologies. In particular, small and medium research labs (which are the majority of the labs in the world) have a hard time using these technologies as they demand high visibility projects and/or high-qualified computer personnel. This paper describes OurGrid, a system designed to fill this gap. OurGrid is an open, free-to-join, cooperative Grid in which labs donate their idle computational resources in exchange for accessing other labs’ idle resources when needed. It relies on an incentive mechanism that makes it in the best interest of participants to collaborate with the system, employs a novel application scheduling technique that demands very little information, and uses virtual machines to isolate applications and thus provide security. The vision is that OurGrid enables labs to combine their resources in a massive worldwide computing platform. OurGrid is in production since December 2004. Any lab can join it by downloading its software from http://www.ourgrid.org.

Running Bag-of-Tasks applications on computational grids: the MyGrid approach

We here discuss how to run Bag-of-Tasks applications on computational grids. Bag-of-Tasks applications (those parallel applications whose tasks are independent) are both relevant and amendable for execution on grids. However, few users currently execute their Bag-of-Tasks applications on grids. We investigate the reason for this state of affairs and introduce MyGrid, a system designed to overcome the identified difficulties. MyGrid provides a simple, complete and secure way for a user to run Bag-of-Tasks applications on all resources she has access to. Besides putting together a complete solution useful for real users, MyGrid embeds two important research contributions to grid computing. First, we introduce some simple working environment abstractions that hide machine configuration heterogeneity from the user. Second, we introduce work queue with replication (WQR), a scheduling heuristics that attains good performance without relying on information about the grid or the application, although consuming a few more cycles. Note that not depending on information makes WQR much easier to deploy in practice

Faults in grids: why are they so bad and what can be done about it?

Computational grids have the potential to become the main execution platform for high performance and distributed applications. However, such systems are extremely complex and prone to failures. We present a survey with the grid community on which several people shared their actual experience regarding fault treatment. The survey reveals that, nowadays, users have to be highly involved in diagnosing failures, that most failures are due to configuration problems (a hint of the areas immaturity), and that solutions for dealing with failures are mainly application-dependent. Going further, we identify two main reasons for this state of affairs. First, grid components that provide high-level abstractions when working, do expose all gory details when broken. Since there are no appropriate mechanisms to deal with the complexity exposed (configuration, middleware, hardware and software issues), users need to be deeply involved in the diagnosis and correction of failures. To address this problem, one needs a way to coordinate different support teams working at the grids different levels of abstraction. Second, fault tolerance schemes today implemented on grids tolerate only crash failures. Since grids are prone to more complex failures, such those caused by heisenbugs, one needs to tolerate tougher failures. Our hope is that the very heterogeneity, that makes a grid a complex environment, can help in the creation of diverse software replicas, a strategy that can tolerate more complex failures.

Exploiting replication and data reuse to efficiently schedule data-intensive applications on grids

Data-intensive applications executing over a computational grid demand large data transfers. These are costly operations. Therefore, taking them into account is mandatory to achieve efficient scheduling of data-intensive applications on grids. Further, within a heterogeneous and ever changing environment such as a grid, better schedules are typically attained by heuristics that use dynamic information about the grid and the applications. However, this information is often difficult to be accurately obtained. On the other hand, although there are schedulers that attain good performance without requiring dynamic information, they were not designed to take data transfer into account. This paper presents Storage Affinity, a novel scheduling heuristic for bag-of-tasks data-intensive applications running on grid environments. Storage Affinity exploits a data reuse pattern, common on many data-intensive applications, that allows it to take data transfer delays into account and reduce the makespan of the application. Further, it uses a replication strategy that yields efficient schedules without relying upon dynamic information that is difficult to obtain. Our results show that Storage Affinity may attain better performance than the state-of-the-art knowledge-dependent schedulers. This is achieved at the expense of consuming more CPU cycles and network bandwidth.

Discouraging free riding in a peer-to-peer CPU-sharing grid

Grid computing has excited many with the promise of access to huge amounts of resources distributed across the globe. However, there are no largely adopted solutions for automatically assembling grids, and this limits the scale of todays grids. Some argue that this is due to the overwhelming complexity of the proposed economy-based solutions. Peer-to-peer grids Iwve emerged as a less complex alternative. We are currently deploying OurGrid, one such peer-to-peer grid. OurGrid is a CPU-sharing grid that targets bag-of-tasks applications (i.e. parallel applications whose tasks are independent). In order to ease system deployment, OurGrid is based on a very lightweight autonomous reputation scheme. Free riding is an important issue for any peer-to-peer system. The aim is to show that OurGrids reputation system successfully discourages free riding, making it in each peer s own interest to collaborate with the peer-to-peer community. We show this in two steps. First, we analyze the conditions under which a reputation scheme can discourage free riding in a CPU-sharing grid. Second, we show that OurGrids reputation scheme satisfies these conditions, even in the presence of malicious peers. Unlike other distributed mechanisms for discouraging free riding, OurGrids reputation scheme achieves this without requiring a shared cryptographic infrastructure or specialized storage.

Resource demand and supply in BitTorrent content-sharing communities

BitTorrent is a widely popular peer-to-peer content distribution protocol. Unveiling patterns of resource demand and supply in its usage is paramount to inform operators and designers of BitTorrent and of future content distribution systems. This study examines three BitTorrent content-sharing communities regarding resource demand and supply. The resulting characterization is significantly broader and deeper than previous BitTorrent investigations: it compares multiple BitTorrent communities and investigates aspects that have not been characterized before, such as aggregate user behavior and resource contention. The main findings are three-fold: (i) resource demand - a more accurate model for the peer arrival rate over time is introduced, contributing to workload synthesis and analysis; additionally, torrent popularity distributions are found to be non-heavy-tailed, what has implications on the design of BitTorrent caching mechanisms; (ii) resource supply - a small set of users contributes most of the resources in the communities, but the set of heavy contributors changes over time and is typically not responsible for most resources used in the distribution of an individual file; these results imply some level of robustness can be expected in BitTorrent communities and directs resource allocation efforts; (iii) relation between resource demand and supply - users that provide more resources are also those that demand more from it; also, the distribution of a file usually experiences resource contention, although the communities achieve high rates of served requests.

On the efficacy, efficiency and emergent behavior of task replication in large distributed systems

Large distributed systems challenge traditional schedulers, as it is often hard to determine a priori how long each task will take to complete on each resource, information that is input for such schedulers. Task replication has been applied in a variety of scenarios as a way to circumvent this problem. Task replication consists of dispatching multiple replicas of a task and using the result from the first replica to finish. Replication schedulers (i.e. schedulers that employ task replication) are able to achieve good performance even in the absence of information on tasks and resources. They are also of smaller complexity than traditional schedulers, making them better suitable for large distributed systems. On the other hand, replication schedulers waste cycles with the replicas that are not the first to finish. Moreover, this extra consumption of resources raises severe concerns about the system-wide performance of a distributed system with multiple, competing replication schedulers. This paper presents a comprehensive study of task replication, comparing replication schedulers against traditional information-based schedulers, and establishing their efficacy (the performance delivered to the application), efficiency (the amount of resources wasted), and emergent behavior (the system-wide behavior of a system with multiple replication schedulers). We also introduce a simple access control strategy that can be implemented locally by each resource and greatly improves overall performance of a system on which multiple replication schedulers compete for resources.

Automatic grid assembly by promoting collaboration in peer-to-peer grids

Currently, most computational grids (systems allowing transparent sharing of computing resources across organizational boundaries) are assembled using human negotiation. This procedure does not scale well, and is too inflexible to allow for large open grids. Peer-to-peer (P2P) grids present an alternative way to build grids with many sites. However, to actually assemble a large grid, peers must have an incentive to provide resources to the system. In this paper we present an incentive mechanism called the Network of Favors, which makes it in the interest of each participating peer to contribute its spare resources. We show through simulations with up to 10,000 peers and experiments with software implementing the mechanism in a deployed system that the Network of Favors promotes collaboration in a simple, robust and scalable fashion. We also discuss experiences of using OurGrid, a grid based on this mechanism.