Andrew Stephen McGough

Making the Grid Predictable through Reservations and Performance Modelling

Unpredictable job execution environments pose a significant barrier to the widespread adoption of the Grid paradigm, because of the innate risk of jobs failing to execute at the time specified by the user. We demonstrate that predictability can be enhanced with a supporting infrastructure consisting of three parts: Performance modelling and monitoring, scheduling which exploits application structure and an advanced reservation resource management service. We prove theoretically that execution times using advanced reservations display less variance than those without. We also show that the costs of advanced reservations can be reduced by providing the system with more accurate performance models. Following the theoretical discussion, we describe the implementation of a fully functional workflow enactment framework that supports advanced reservations and performance modelling thereby providing predictable execution behavior. We further provide experimental results confirming our theoretical models.

Flood modelling for cities using Cloud computing

Urban flood risk modelling is a highly topical example of intensive computational processing. Such processing is increasingly required by a range of organisations including local government, engineering consultancies and the insurance industry to fulfil statutory requirements and provide professional services. As the demands for this type of work become more common, then ownership of high-end computational resources is warranted but if use is more sporadic and with tight deadlines then the use of Cloud computing could provide a cost-effective alternative. However, uptake of the Cloud by such organisations is often thwarted by the perceived technical barriers to entry. In this paper we present an architecture that helps to simplify the process of performing parameter sweep work on an Infrastructure as a Service Cloud. A parameter sweep version of the urban flood modelling, analysis and visualisation software “CityCat” was developed and deployed to estimate spatial and temporal flood risk at a whole city scale – far larger than had previously been possible. Performing this work on the Cloud allowed us access to more computing power than we would have been able to purchase locally for such a short time-frame (∼21 months of processing in a single calendar month). We go further to illustrate the considerations, both functional and non-functional, which need to be addressed if such an endeavour is to be successfully achieved.

QoS-Constrained Stochastic Workflow Scheduling in Enterprise and Scientific Grids

Grid computing infrastructures are inherently dynamic and unpredictable environments shared by many users. Grid schedulers aim to make efficient use of Grid resources while providing the best possible performance to the Grid applications and satisfying the associated performance and policy constraints. Additionally, in commercial Grid settings, where the Grid resource brokering becomes an increasingly important part of Grid scheduling, it is necessary to minimise the cost of application execution on the behalf of the Grid users, while ensuring that the applications meet their QoS constraints. Efficient resource allocation could in turn also allow the resource broker to maximise its profit by minimising the number of resources procured. Scheduling in such a large-scale, dynamic and distributed environment is a complex undertaking. In this paper, we propose an approach to Grid scheduling which abstracts over the details of individual applications, focusing instead on the global cost optimisation problem and the scheduling of the entire Grid workload. Our model places particular emphasis on the stochastic and unpredictable nature of the Grid, leading to a more accurate reflection of the state of the Grid and hence more efficient and accurate scheduling decisions.

On Quality of Service Support for Grid Computing

Computing Grids are hardware and software infrastructures that support secure sharing and concurrent access to distributed services by a large number of competing users from different virtual organizations. Concurrency can easily lead to overload and resource shortcomings in large-scale Grid infrastructures, as today’s Grids do not offer differentiated services. We propose a framework for supporting quality of service guarantees via both reservation and discovery of best-effort services based on the matchmaking of application requirements and quality of service performance profiles of the candidate services. We illustrate the middleware components needed to support both strict and loose guarantees and the performance assessment techniques for the discovery of suitable services.

GRIDCC: real-time workflow system

The Grid is a concept which allows the sharing of resources between distributed communities, allowing each to progress towards potentially different goals. As adoption of the Grid increases so are the activities that people wish to conduct through it. The GRIDCC project is a European Union funded project addressing the issues of integrating instruments into the Grid. This increases the requirement of workflows and Quality of Service upon these workflows as many of these instruments have real-time requirements. In thispaper we present the workflow management service within the GRIDCC project which is tasked with optimising the workflows and ensuring that they meet the pre-defined QoS requirements specified upon them.

Adding instruments and workflow support to existing grid architectures

Many Grid architectures have been developed in recent years. These range from the large community Grids such as LHG and EGEE to single site deployments such as Condor. However, these Grid architectures have tended to focus on the single or batch submission of executable jobs. Application scientists are now seeking to manage and use physical instrumentation on the Grid, integrating these with the computational tasks they already perform. This will require the functionality of current Grid systems to be extended to allow the submission of entire workflows. Thus allowing the scientists to perform increasingly larger parts of their experiments within the Grid environment. We propose here a set of high level services which may be used on-top of these existing Grid architectures such that the benefits of these architectures may be exploited along with the new functionality of workflows.

QoS Support For Workflows In A Volatile Grid

The grid can be seen as a collection of services each of which performs some functionality. Users of the grid seek to use combinations of these services to perform the overall task they need to achieve. In general this can be seen as a set of services with a workflow document describing how these services should be combined. The user may also have certain constraints on the workflow operations, such as execution time or cost to the user, specified in the form of a quality of service (QoS) document. The users submit their workflow to a brokering service along with the QoS document. The brokering services task is to map any given workflow to a subset of the grid services taking the QoS and state of the grid into account - service availability and performance. We propose in this paper an approach for generating constraint equations describing the workflow, the QoS requirements and the state of the grid. This set of equations may be solved using mixed-integer linear programming (MILP), which is the traditional method. We further develop a novel 2-stage stochastic MILP which is capable of dealing with the volatile nature of the grid and adapting the selection of the services during the lifetime of the workflow. We present experimental results comparing our approaches, showing that the 2-stage stochastic programming approach performs consistently better than other traditional approaches

Analysis of power-saving techniques over a large multi-use cluster with variable workload.

Reduction of power consumption for any computer system is now an important issue, although this should be carried out in a manner that is not detrimental to the users of that computer system. We present a number of policies that can be applied to multi‐use clusters where computers are shared between interactive users and high‐throughput computing. We evaluate policies by trace‐driven simulations to determine the effects on power consumed by the high‐throughput workload and impact on high‐throughput users. We further evaluate these policies for higher workloads by synthetically generating workloads based around the profiled workload observed through our system. We demonstrate that these policies could save 55% of the currently used energy for our high‐throughput jobs over our current cluster policies without affecting the high‐throughput users’ experience. Copyright

Stochastic Workflow Scheduling with QoS Guarantees in Grid Computing Environments

Grid computing infrastructures embody a cost-effective computing paradigm that virtualises heterogeneous system resources to meet the dynamic needs of critical business and scientific applications. These applications range from batch processes and long-running tasks to more real-time and even transactional applications. Grid schedulers aim to make efficient use of grid resources in a cost-effective way, while satisfying the quality-of-service requirements of the applications. Scheduling in such a large-scale, dynamic and distributed environment is a complex undertaking. In this paper, we propose an approach to grid scheduling which abstracts over the details of individual applications and aims to provide a globally optimal schedule, while having the ability to dynamically adjust to varying workload demands using various capacity planning techniques. Our model places particular emphasis on the stochastic and unpredictable nature of the grid, leading to a more accurate reflection of the state of the grid and hence more efficient and accurate scheduling decisions

Real Time Monitor of Grid job executions

In this paper we describe the architecture and operation of the Real Time Monitor (RTM), developed by the Grid team in the HEP group at Imperial College London. This is arguably the most popular dissemination tool within the EGEE [1] Grid. Having been used, on many occasions including GridFest and LHC inauguration events held at CERN in October 2008. The RTM gathers information from EGEE sites hosting Logging and Bookkeeping (LB) services. Information is cached locally at a dedicated server at Imperial College London and made available for clients to use in near real time. The system consists of three main components: the RTM server, enquirer and an apache Web Server which is queried by clients. The RTM server queries the LB servers at fixed time intervals, collecting job related information and storing this in a local database. Job related data includes not only job state (i.e. Scheduled, Waiting, Running or Done) along with timing information but also other attributes such as Virtual Organization and Computing Element (CE) queue – if known. The job data stored in the RTM database is read by the enquirer every minute and converted to an XML format which is stored on a Web Server. This decouples the RTM server database from the client removing the bottleneck problem caused by many clients simultaneously accessing the database. This information can be visualized through either a 2D or 3D Java based client with live job data either being overlaid on to a 2 dimensional map of the world or rendered in 3 dimensions over a globe map using OpenGL.