Andre Merzky

Enabling Applications on the Grid: A Gridlab Overview

Grid technology is widely emerging. Still, there is an eminent shortage of real Grid users, mostly due to the lack of a “critical mass” of widely deployed and reliable higher-level Grid services, tailored to application needs. The GridLab project aims to provide fundamentally new capabilities for applications to exploit the power of Grid computing, thus bridging the gap between application needs and existing Grid middleware. We present an overview of GridLab, a large-scale, EU-funded Grid project spanning over a dozen groups in Europe and the US. We first outline our vision of Grid-empowered applications and then discuss GridLab’s general architecture and its Grid Application Toolkit (GAT). We illustrate how applications can be Grid-enabled with the GAT and discuss GridLab’s scheduler as an example of GAT services.

The Cactus Code: a problem solving environment for the grid

Cactus is an open source problem solving environment designed for scientists and engineers. Its modular structure facilitates parallel computation across different architectures and collaborative code development between different groups. The Cactus Code originated in the academic research community, where it has been developed and used over many years by a large international collaboration of physicists and computational scientists. We discuss how the intensive computing requirements of physics applications now using the Cactus Code encourage the use of distributed and metacomputing, describe the development and experiments which have already been performed with Cactus, and detail how its design makes it an ideal application test-bed for Grid computing.

GridLab: a grid application toolkit and testbed

In this paper we present the new project called GridLab which is funded by the European Commission under the Fifth Framework Programme. The GridLab project, made up of computer scientists, astrophysicists and other scientists from various application areas, will develop and implement the grid application toolkit (GAT) together with a set of services to enable easy and efficient use of Grid resources in a real and production grid environment. GAT will provide core, easy to use functionality through a carefully constructed set of generic higher level grid APIs through which an application will be able to call the grid services laying beneath in order to perform efficiently in the Grid environment using various, dramatically wild application scenarios.

Cactus Tools for Grid Applications

Cactus is an open source problem solving environment designed for scientists and engineers. Its modular structure facilitates parallel computation across different architectures and collaborative code development between different groups. The Cactus Code originated in the academic research community, where it has been developed and used over many years by a large international collaboration of physicists and computational scientists. We discuss here how the intensive computing requirements of physics applications now using the Cactus Code encourage the use of distributed and metacomputing, and detail how its design makes it an ideal application test-bed for Grid computing. We describe the development of tools, and the experiments which have already been performed in a Grid environment with Cactus, including distributed simulations, remote monitoring and steering, and data handling and visualization. Finally, we discuss how Grid portals, such as those already developed for Cactus, will open the door to global computing resources for scientific users.

Solving Einstein's equations on supercomputers

In 1916, Albert Einstein published his famous general theory of relativity, which contains the rules of gravity and provides the basis for modern theories of astrophysics and cosmology. For many years, physicists, astrophysicists and mathematicians have striven to develop techniques for unlocking the secrets contained in Einsteins theory of gravity; more recently, computational science research groups have added their expertise to the endeavor. Because the underlying scientific project provides such a demanding and rich system for computational science, techniques developed to solve Einsteins equations will apply immediately to a large family of scientific and engineering problems. The authors have developed a collaborative computational framework that allows remote monitoring and visualization of simulations, at the center of which lies a community code called Cactus. Many researchers in the general scientific computing community have already adopted Cactus, as have numerical relativists and astrophysicists. In June 1999, an international team of researchers at various sites ran some of the largest such simulations in numerical relativity yet undertaken, using a 256-processor SGI Origin 2000 supercomputer at the National Center for Supercomputing Applications (NCSA). Other globally distributed scientific teams are running visual simulations of Einsteins equations on the gravitational effects of colliding black holes.

Programming Abstractions for Data Intensive Computing on Clouds and Grids

MapReduce has emerged as an important data-parallel programming model for data-intensive computing – for Clouds and Grids. However most if not all implementations of MapReduce are coupled to a specific infrastructure. SAGA is a high-level programming interface which provides the ability to create distributed applications in an infrastructure independent way. In this paper, we show how MapReduce has been implemented using SAGA and demonstrate its interoperability across different distributed platforms – Grids, Cloud-like infrastructure and Clouds. We discuss the advantages of programmatically developing MapReduce using SAGA, by demonstrating that the SAGA-based implementation is infrastructure independent whilst still providing control over the deployment, distribution and runtime decomposition. The ability to control the distribution and placement of the computation units (workers) is critical in order to implement the ability to move computational work to the data. This is required to keep data network transfer low and in the case of commercial Clouds the monetary cost of computing the solution low. Using data-sets of size up to 10GB, and upto 10 workers, we provide detailed performance analysis of the SAGA-MapReduce implementation, and show how controllingthe distribution of computation and the payload per worker helps enhance performance.

P∗: A model of pilot-abstractions

Pilot-Jobs support effective distributed resource utilization, and are arguably one of the most widely-used distributed computing abstractions - as measured by the number and types of applications that use them, as well as the number of production distributed cyberinfrastructures that support them. In spite of broad uptake, there does not exist a well-defined, unifying conceptual model of Pilot-Jobs which can be used to define, compare and contrast different implementations. Often Pilot-Job implementations are strongly coupled to the distributed cyber-infrastructure they were originally designed for. These factors present a barrier to extensibility and interoperability. This paper is an attempt to (i) provide a minimal but complete model (P*) of Pilot-Jobs, (ii) establish the generality of the P* Model by mapping various existing and well known Pilot-Job frameworks such as Condor and DIANE to P*, (iii) derive an interoperable and extensible API for the P* Model (Pilot-API), (iv) validate the implementation of the Pilot-API by concurrently using multiple distinct Pilot-Job frameworks on distinct production distributed cyberinfrastructures, and (v) apply the P* Model to Pilot-Data.

GridLab: Enabling Applications on the Grid

Grid technology is widely emerging. Still, there is an eminent shortage of real Grid users, due to the absence of two important catalysts: First, a widely accepted vision on how applications can substantially benefit from Grids, and second a toolkit of higher-level Grid services, tailored to application needs. The GridLab project aims to provide fundamentally new capabilities for applications to exploit the power of Grid computing, thus bridging the gap between application needs and existing Grid middleware. We present an overview of GridLab, a largescale, EU-funded Grid project spanning over a dozen groups in Europe and the US. We first outline our vision of Grid-empowered applications and then discuss GridLabs general architecture.

Application Level Interoperability between Clouds and Grids

SAGA is a high-level programming interface which provides the ability to develop distributed applications in aninfrastructure independent way. In an earlier paper, we discussed how SAGA was used to develop a version of MapReduce which provided the user with the ability to control the relative placement of compute and data, whilst utilizing different distributed infrastructure. In this paper, we use the SAGA-based implementation of MapReduce, and demonstrate its interoperability across Clouds and Grids. We discuss how a range of cloud adaptors have beendeveloped for SAGA. The major contribution of this paper isthe demonstration – possibly the first ever, of interoperability between different Clouds and Grids, without any changes to the application. We analyse the performance of SAGA-MapReduce when using multiple, different, heterogeneous infrastructure concurrently for the same problem instance; However, we do not strive to provide a rigorous performance model, but to provide a proof-of-concept of application-level interoperabilityand illustrate its importance

Grid Interoperability at the Application Level Using SAGA

SAGA is a high-level programming abstraction, which significantly facilitates the development and deployment of Grid-aware applications. The primary aim of this paper is to discuss how each of the three main components of the SAGA landscape - interface specification, specific implementation and the different adaptors for middleware distribution - facilitate application-level interoperability. We discuss SAGA in relation to the ongoing GIN Community Group efforts and show the consistency of the SAGA approach with the GIN Group efforts. We demonstrate how interoperability can be enabled by the use of SAGA, by discussing two simple, yet meaningful applications: in the first, SAGA enables applications to utilize interoperability and in the second example SAGA adaptors provide the basis for interoperability.