Mihael Hategan

Swift: Fast, Reliable, Loosely Coupled Parallel Computation

We present Swift, a system that combines a novel scripting language called SwiftScript with a powerful runtime system based on CoG Karajan, Falkon, and Globus to allow for the concise specification, and reliable and efficient execution, of large loosely coupled computations. Swift adopts and adapts ideas first explored in the GriPhyN virtual data system, improving on that system in many regards. We describe the SwiftScript language and its use of XDTM to describe the logical structure of complex file system structures. We also present the Swift runtime system and its use of CoG Karajan, Falkon, and Globus services to dispatch and manage the execution of many tasks in parallel and grid environments. We describe application experiences and performance experiments that quantify the cost of Swift operations.

Swift: A language for distributed parallel scripting

Scientists, engineers, and statisticians must execute domain-specific application programs many times on large collections of file-based data. This activity requires complex orchestration and data management as data is passed to, from, and among application invocations. Distributed and parallel computing resources can accelerate such processing, but their use further increases programming complexity. The Swift parallel scripting language reduces these complexities by making file system structures accessible via language constructs and by allowing ordinary application programs to be composed into powerful parallel scripts that can efficiently utilize parallel and distributed resources. We present Swifts implicitly parallel and deterministic programming model, which applies external applications to file collections using a functional style that abstracts and simplifies distributed parallel execution.

GridAnt: a client-controllable grid workflow system

Process management is an extremely important concept in both business and scientific communities. Several workflow management tools have been proposed in recent years offering advanced functionality in various domains. In the business world, workflow vendors offer commercial and customized solutions targeting specific users. In the scientific world, several open-source workflow management tools are freely available. However they are directed toward service aggregation rather than distributed process management. Little consideration is given to the needs of the client in terms of mapping the process flow of the client. In the grid community it is essential that the grid users have such a tool available enabling them to orchestrate complex work-flows on the fly without substantial help from the service providers. At the same time it is important that the grid user not be burdened with the intricacies of the workflow system. With the perspective of the grid user in mind, an extensible client-side workflow management system, called GridAnt, has been developed. This paper discusses the design principles, functionality, and application of the proposed GridAnt workflow manager.

Analysis and Provision of QoS for Distributed Grid Applications

Abstract Grid computing provides the infrastructure necessary to access and use distributed resources as part of virtual organizations. When used in this way, Grid computing makes it possible for users to participate in collaborative and distributed applications such as tele-immersion, visualization, and computational simulation. Some of these applications operate in a collaborative mode, requiring data to be stored and delivered in a timely manner. This class of applications must adhere to stringent real-time constraints and Quality-of-Service (QoS) requirements. A QoS management approach is therefore required to orchestrate and guarantee the timely interaction between such applications and services. We discuss the design and a prototype implementation of a QoS system, and demonstrate how we enable Grid applications to become QoS compliant. We validate this approach through a case study of an image processing task derived from a nanoscale structures application.

Parallel Scripting for Applications at the Petascale and Beyond

Scripting accelerates and simplifies the composition of existing codes to form more powerful applications. Parallel scripting extends this technique to allow for the rapid development of highly parallel applications that can run efficiently on platforms ranging from multicore workstations to petascale supercomputers.

Workflow Concepts of the Java CoG Kit

Many scientific simulations and experiments require the coordination of numerous tasks posed by interdisciplinary research teams. Grids can provide access to the necessary high-end resources to conduct such tasks. The complex tasks and their interactions must be supported through convenient tools. To address this issue, we introduce a number of Grid abstractions that make the development of Grid middleware-independent tools possible and allow for the integration of a number of commodity tools. Our vision is implemented through an integrated approach based on a layered architecture that bridges the gap between Grid middleware and scientific applications. Our abstractions include specialized services, a Grid workflow engine and language, and Gridfaces – graphical abstractions that can be employed in science portals and standalone applications.

The Open Grid Computing Environments collaboration: Portlets and services for science gateways

We review the efforts of the Open Grid Computing Environments collaboration. By adopting a general three-tiered architecture based on common standards for portlets and Grid Web services, we can deliver numerous capabilities to science gateways from our diverse constituent efforts. In this paper, we discuss our support for standards-based Grid portlets using the Velocity development environment. Our Grid portlets are based on abstraction layers provided by the Java CoG kit, which hide the differences of different Grid toolkits. Sophisticated services are decoupled from the portal container using Web service strategies. We describe advance information, semantic data, collaboration, and science application services developed by our consortium. Copyright

Coasters: Uniform Resource Provisioning and Access for Clouds and Grids

In this paper we present the Coaster System. It is an automatically-deployed node provisioning (Pilot Job) system for grids, clouds, and ad-hoc desktop-computer networks supporting file staging, on-demand opportunistic multi-node allocation, remote logging, and remote monitoring. The Coaster System has been previously [32] shown to work at scales of thousands of cores. It has been used since 2009 for applications in fields that include biochemistry, earth systems science, energy modeling, and neuroscience. The system has been used successfully on the Open Science Grid, the TeraGrid [1], supercomputers (IBM Blue Gene/P [15], Cray XT and XE systems [5], and Sun Constellation [26]), a number of smaller clusters, and three cloud infrastructures (BioNimbus [2], Future Grid [20] and Amazon EC2 [16]).

Java CoG Kit Workflow

In order to satisfy the need for sophisticated experiment and simulation management solutions for the scientific user community, various frameworks must be provided. Such frameworks include APIs, services, templates, patterns, GUIs, command-line tools, and workflow systems that are specifically addressed towards the goal of assisting in the complex process of experiment and simulation management. Workflow by itself is just one of the ingredients for a successful experiment and simulation management tool.

Abstracting the Grid

The Grid approach allows collaborative pooling of distributed resources across multiple domains. However, the benefits of the Grid are limited to those offered by the commodity application development framework used. Several elegant and flexible application development frameworks support only specific Grid architectures, thereby not allowing the applications to exploit the full potential of the Grid. In order to initiate community interest to standardize a high-level abstraction layer for different Grid architectures, we introduce a collection of abstractions and data structures that collectively build a basis for an open Grid computing environment.