Jan Dünnweber

From Grid Middleware to Grid Applications: Bridging the Gap with Hocs

This paper deals with the problem of application programming for grid systems that combine heterogeneous data and computational resources via the Internet. We argue that grid programming is still too complex because of the big gap between the currently used and anticipated grid middleware, (e.g., Globus or WSRF) and the application level. We suggest that this gap needs to be closed in future-generation grids and propose a novel approach to bridging the gap by using Higher-Order Components (HOCs) — recurring patterns of parallel behaviour that are provided to the user as program building blocks with pre-packaged implementation and middleware setup. The presentation is illustrated with a simple case study of computing fractal images. Our experiments demonstrate that HOCs can simplify grid application programming significantly, without serious performance loss.

HOC-SA: a grid service architecture for higher-order components

The current efforts on programming grid applications often rely on service-oriented approaches like grid services. This work presents HOC-SA -a service architecture for higher-order components, which provides the programmer with reusable and composable patterns of parallelism and is interoperable with the latest Globus toolkit implementations. We describe our implementation of HOC-SA using OGSA-DAI, a framework for integrating grids with distributed databases. We present a simple example application and report first measurements on our grid testbed.

HOCS:Higher-Order Components for Grids

We present HOCs — Higher-Order Components — that provide the Grid application programmer with reusable and composable patterns of parallelism. HOCs can be viewed formally as higher-order functions, i.e. a generic implementation of a HOC on a remote machine can be customized with application-specific code parameters which are supplied by the user and shipped via the network. We take the well-known “Farm of Workers” pattern as our motivating example, present an experimental implementation of the Farm-HOC as a Grid Service using the Globus Toolkit, and report first measurements for a case study of computing fractal images using the Farm-HOC.

Optimization techniques for skeletons on grids

Skeletons are common patterns of parallelism, such as farm and pipeline, that can be abstracted and offered to the application programmer as programming primitives. We describe the use and implementation of skeletons on emerging computational grids, with the skeleton system Lithium, based on Java and RMI, as our reference programming syttem. Our main contribution is the exploration of optimization techniques for implementing skeletons on grids based on an optimized, future-based RMI mechanism, which we integrate into the macro-dataflow evaluation mechanism of Lithium. We discuss three optimizations: 1) a lookahead mechanism that allows to process multiple tasks concurrently at each grid server and thereby increases the overall degree of parallelism, 2) a lazy taskbinding technique that reduces interactions between grid servers and the task dispatcher, and 3) dynamic improvements that optimize the collecting of results and the work-load balancing. We report experimental results that demonstrate the improvements due to our optimizations on various testbeds, including a heterogeneous grid-like environment.

Towards Automatic Creation of Web Services for Grid Component Composition

While high-level software components simplify the programming of grid applications and Web services increase their interoperability, developing such components and configuring the interconnecting services is a demanding task. In this paper, we consider the combination of Higher-Order Components (HOCs) with the Fractal component model and the ProActive library.

Higher-Order Components for Grid Programming

A major challenge in grid computing remains the application software development for this new kind of infrastructure. Grid application programmers have to take into account several complicated aspects: distribution of data and computations, parallel computations on different sites and processors, heterogeneity of the involved computers, load balancing, etc. Grid programmers thus demand novel programming methodologies that abstract over such technical details while preserving the beneficial features of modern grid middleware. For this purpose, the authors introduce Higher-Order Components (HOCs). HOCs implement generic parallel/distributed processing patterns, together with the required middleware support, and they are offered to users via a high-level service interface. Users only have to provide the application-specific pieces of their programs as parameters, while low-level implementation details, such as the transfer of data across the grid, are handled by the HOCs. HOCs were developed within the CoreGRID European Network of Excellence and have become an optional extension of the popular Globus middleware. The book provides the reader with hands-on experience, describing a broad collection of example applications from various fields of science and engineering, including biology, physics, etc. The Java code for these examples is provided online, complementing the book. The expected application performance is studied and reported for extensive performance experiments on different testbeds, including grids with worldwide distribution. The book is targeted at graduate students, advanced professionals, and researchers in both academia and industry. Readers can raise their level of knowledge about methodologies for programming contemporary parallel and distributed systems, and, furthermore, they can gain practical experience in using distributed software. Practical examples show how the complementary online material can easily be adopted in various new projects.

Adaptable Parallel Components for Grid Programming

We suggest that parallel software components used for grid computing should be adaptable to application-specific requirements, instead of developing new components from scratch for each particular application. As an example, we take a parallel farm component which is “embarrassingly parallel”, i. e., free of dependencies, and adapt it to the wavefront processing pattern with dependencies that impact its behavior. We describe our approach in the context of Higher-Order Components (HOCs), with the Java-based system Lithium as our implementation framework. The adaptation process relies on HOCs’ mobile code parameters that are shipped over the network of the grid. We describe our implementation of the proposed component adaptation method and report first experimental results for a particular grid application — the alignment of DNA sequence pairs, a popular, time-critical problem in computational molecular biology.

Clayworks: Toward user-oriented software for collaborative modeling and simulation

We deal with the problem of developing a software system which integrates collaborative real-time modeling and distributed computing. The main challenge is user-orientation: we need a collaborative workspace for geographically dispersed users with a seamless access of every user to high-performance servers. Wedescribe a particular system, Clayworks, that allows modeling of virtual clay objects and running computation-intensive deformation simulations for objects crashing into each other. To integrate heterogeneous computational resources, we adopted modern Grid middleware and provided the users with an intuitive graphical interface. We parallelized the computation of simulations using a Higher-Order Component (HOC) which abstracts over the Globus Web service resource framework (WSRF) used to interconnect our worksuite to the computation server. Clayworks is a representative of a large class of demanding systems which combine collaborative, user-oriented modeling with performance-critical computations, e.g., crash-tests or simulations for biological population evolution.

LooPo-HOC: A Grid Component with Embedded Loop Parallelization

This work integrates two distinct research areas of parallel and distributed computing, (1) automatic loop parallelization, and (2) component-based Grid programming. The latter includes technologies developed within CoreGRID for simplifying Grid programming: the Grid Component Model (GCM) and HigherOrder Components (HOCs). Components support developing applications on the Grid without taking all the technical details of the particular platform type into account (network communication, heterogeneity, etc.). The GCM enables a hierarchical composition of program pieces and HOCs enable the reuse of component code in the development of new applications by specifying application-specific operations in a program via code parameters. When a programmer is provided, e. g. , with a compute farm HOC, only the independent worker tasks must be described. But, once an application exhibits data or control dependences, the trivial farm is no longer sufficient. Here, the power of loop parallelization tools, like LooPo, comes into play: by embedding LooPo into a HOC, we show that these two technologies in combination facilitate the automatic transformation of a sequential loop nest with complex dependences (supplied by the user as a HOC parameter) into an ordered task graph, which can be processed on the Grid in parallel. This technique can significantly simplify GCM-based systems which combine multiple HOCs and other components. We use an equation system solver based on the successive overrelaxation method (SOR) as our motivating application example and for performance experiments.

Clayworks: A System for Collaborative Real-Time Modeling and High-Performance Simulation

Clayworks is a software system which integrates collaborative real-time modeling and distributed computing. It addresses the challenge of developing a collaborative workspace with a seamless access to high-performance servers. Clayworks allows modeling of virtual clay objects and running computation-intensive deformation simulations for objects crashing into each other. To integrate heterogeneous computational resources, we adopted modern Grid middleware and provided the users with an intuitive graphical interface. We parallelized the computation of simulations using a Higher-Order Component (HOC) which abstracts over the Globus Web service resource framework (WSRF) used to interconnect our worksuite to the computation server. Clayworks is a representative of a large class of demanding systems which combine collaborative modeling with performance-critical computations, e.g., crash-tests or simulations for biological population evolution.