A. Stephen McGough

ICENI: optimisation of component applications within a Grid environment

Effective exploitation of Computational Grids can only be achieved when applications are fully integrated with the Grid middleware and the underlying computational resources. Fundamental to this exploitation is information. Information about the structure and behaviour of the application, the capability of the computational and networking resources, and the availability and access to these resources by an individual, a group or an organisation.In this paper we describe Imperial College e-Science Networked Infrastructure (ICENI), a Grid middleware framework developed within the London e-Science Centre. ICENI is a platform-independent framework that uses open and extensible XML derived protocols, within a framework built using Java and Jini, to explore effective application execution upon distributed federated resources. We match a high-level application specification, defined as a network of components, to an optimal combination of the currently available component implementations within our Grid environment, by using composite performance models. We demonstrate the effectiveness of this architecture through the high-level specification and solution of a set of linear equations by automatic and selection of optimal resources and implementations.

Capacity planning and scheduling in Grid computing environments

Grid computing infrastructures embody a cost-effective computing paradigm that virtualises heterogeneous system resources to meet the dynamic needs of critical business and scientific applications. These applications range from batch processes and long-running tasks to real-time and even transactional applications. Grid computing environments are inherently dynamic and unpredictable environments sharing services amongst many different users. Grid schedulers aim to make the most efficient use of Grid resources (high utilisation) while providing the best possible performance to the Grid applications (reducing makespan) and satisfying the associated performance and Quality of Service (QoS) constraints. Additionally, in commercial Grid settings where economic considerations are an increasingly important part of Grid scheduling, it is necessary to minimise the cost of application execution on the behalf of the Grid users while ensuring that the applications meet their QoS constraints. Furthermore, efficient resource allocation may allow a resource broker to maximise their profit by minimising the quantity of resource procurement. Scheduling in such a large-scale, dynamic and distributed environment is a complex undertaking. In this paper, we propose an approach to Grid scheduling which abstracts over the details of individual applications, focusing instead on the global cost optimisation problem while taking into account the entire workload, dynamically adjusting to the varying service demands. Our model places particular emphasis on the stochastic and unpredictable nature of the Grid, leading to a more accurate reflection of the state of the Grid and hence more efficient and accurate scheduling decisions.

Meaning and Behaviour in Grid Oriented Components

The ICENI middleware utilises information captured within a component based application in order to facilitate Grid-based scheduling. We describe a system of application related meta-data that features a separation of concerns between meaning, behaviour and implementation, which allows for both communication and implementation selection at run-time, while providing the user with a flow-based programming model. It is shown that this separation enables a flexible approach to scheduling, and eases the integration of components with disparate control flow patterns or data types, by means of converters and tees for collective communication. By explicitly recording application information and supporting multiple scheduling approaches, communication protocols and component applications, while retaining OGSA compatibility, the ICENI component model is ideally suited to Grid computing.

Optimisation of component-based applications within a grid environment

Effective exploitation of computational grids can only be achieved when applications are fully integrated with the grid middleware and the underlying computational resources. Fundamental to this exploitation is information. Information about the structure and behaviour of the application, the capability of the computational and networking resources, and the availability and access to these resources by an individual, a group or an organisation.This paper describes an integrated grid environment that is open, extensible and platform independent. We match a high-level application specification, defined as a network of components, to an optimal combination of the currently available component implementations within our grid environment. We demonstrate the effectiveness of this architecture through high-level specification and solution of a set of linear equations by automatic and optimal resource and implementation selection.

An End-to-end Workflow Pipeline for Large-scale Grid Computing

In this paper we describe a service-based, software architecture that enables end-to-end, high-level workflow processing in a Grid environment consisting of many heterogeneous resources. Our architecture is essentially a pipeline that extends from the abstract application specification phase to the deployment and execution stages through to returning the results to the user. We envision a large-scale Grid environment that contains heterogeneous resources. Our architecture caters for flexible deployment, performance, reliability and charging for resource usage. These are addressed at the specification level as well as at the realisation (brokering) and execution levels. The proposed architecture is derived from previous work in LeSC that has produced the ICENI pipeline, and our experience with e-Science projects, such as GENIE, e-Protein and RealityGrid from which we derive a set of key requirements.

RealityGrid : an integrated approach to middleware through ICENI

The advancement of modelling and simulation within complex scientific applications is currently constrained by the rate at which knowledge can be extracted from the data produced. As Grid computing evolves, new means of increasing the efficiency of data analysis are being explored. RealityGrid aims to enable more efficient use of scientific computing resources within the condensed matter, materials and biological science communities. The Imperial College e-Science Networked Infrastructure (ICENI) Grid middleware provides an end-to-end pipeline that simplifies the stages of computation, simulation and collaboration. The intention of this work is to allow all scientists to have access to these features without the need for heroic efforts that have been associated with this sort of work in the past. Scientists can utilise advanced scheduling mechanisms to ensure efficient planning of computations, visualize and interactively steer simulations and securely collaborate with colleagues via the Access Grid through a single integrated middleware application.

An Integrated Grid Environment for Component Applications

Computational grids present many obstacles to their effective exploitation by non-trivial applications. We present a grid middleware, implemented using Java and Jini, that eliminates these obstacles through the intelligent use of meta-data relating to the structure, behaviour and performance of an application. We demonstrate how different problem sizes and selection criteria (minimum execution time or minimum cost) utilise different implementations for the optimal solution of a set of linear equations.

A standards based approach to enabling legacy applications on the Grid

The Grid holds great potential for users, software developers and resource owners. Users of the Grid are abstracted away from its complexity, while software developers are being provided with a rich middleware in which to develop their applications without the need for a large investment in resources. Resource owners are able to expose their resources, potentially for financial reward, without the need to invest in software. However, as most of the applications which currently exist pre-date the concept of the Grid, they lack the appropriate functionality and/or modularization to exploit it. In this paper we propose the use of a standards based job submission and monitoring system which is capable of deploying legacy applications onto existing resources within the Grid, environment containers for mapping applications into Grid applications, along with the use of web-based portals to expose these applications to the end user. We further propose that applications which comprise a number of legacy tasks can be handled through the use of a workflow enactment service submitting each of these tasks through the standards based job submission system. We exemplify our approach through the e-Protein project by taking their existing proteome annotation software and exposing it as a Grid service. We show how the use of this technique over the Grid can significantly reduce the makespan required for the application.

A Component Framework for HPC Applications

We describe a general component software framework designed for demanding grid environments that provides optimal performance for the assembled component application. This is achieved by separating the high level abstract description of the composition from the low level implementations. These implementations are chosen at run time by performance analysis of the composed application on the currently available resources. We show through the solution of a simple linear algebra problem that the framework introduces minimal overheads while always selecting the most effective implementation.

Capacity Planning and Stochastic Scheduling in Large-Scale Grids

Grid Infrastructures are inherently dynamic and unpredictable environments where resource management and scheduling play an important part in ensuring that Grid applications execute while satisfying defined cost and performance constraints. Traditional Grid Scheduling approaches have focused on the scheduling and optimisation of single applications, typically without regard to the state of the other applications and the Grid in general. Advance Reservation-based approaches, in particular, have been quite popular and play a pivotal part in most Grid scheduling architectures. In this paper, we aim to demonstrate that advance reservation-based approaches show uncertainty and their performance is heavily defined by the workload characteristics and resource costs and that these approaches do not scale well as the size of the Grid increases. We demonstrate that an alternative scheduling approach, which borrows from Capacity Planning and Operations Research techniques, can improve upon the performance of the existing Grid schedulers.