Zdeněk Salvet

Practical approaches to Grid workload and resource management in the EGEE project

Resource management and scheduling of distributed, data-driven applications in a Grid environment are challenging problems. Although significant results were achieved in the past few years, the development and the proper deployment of generic, reliable, standard components present issues that still need to be completely solved. Interested domains include workload management, resource discovery, resource matchmaking and brokering, accounting, authorization policies, resource access, reliability and dependability. The evolution towards a service-oriented architecture, supported by emerging standards, is another activity that will demand attention. All these issues are being tackled within the EU-funded EGEE project (Enabling Grids for E-science in Europe), whose primary goals are the provision of robust middleware components and the creation of a reliable and dependable Grid infrastructure to support e-Science applications. In this paper we present the plans and the preliminary activities aiming at providing adequate workload and resource management components, suitable to be deployed in a production-quality Grid.

The DataGrid Workload Management System: Challenges and Results

The workload management task of the DataGrid project was mandated to define and implement a suitable architecture for distributed scheduling and resource management in a Grid environment. The result was the design and implementation of a Grid Workload Management System, a super-scheduler with the distinguishing property of being able to take data access requirements into account when scheduling jobs to the available Grid resources. Many novel issues in various fields were faced such as resource management, resource reservation and co-allocation, Grid accounting. In this paper, the architecture and the functionality provided by the DataGrid Workload Management System are presented.

An Active Network Architecture: Distributed Computer or Transport Medium

Future computer networks must be more flexible and faster then today. Active network paradigm is the way how to add flexibility to networks. During the last five years, several active network architecture models were presented. A new one, based on model of active nodes is presented here. The key features of this architecture are the separation of session or connection management functions from the bulk data packet processing functions and associated session management protocol that facilitates user control over active network processing. This architecture is designed to be sufficiently general to accommodate and build on top of any packet-based networking technology. The description of the model is followed by brief of prototype implementation using PC-class computers with NetBSD operating system.

gLite job provenance

The Job Provenance (JP) service is designed to automate keeping track of computations on large scale Grids, giving thus users a tool to correctly archive information about their jobs and to re-submit any job in a reconstructed environment. JP provides a permanent minimal record of job (and its environment) related information, to which free-form user annotations can be added. JP also offers the capability of configuring any number of indexed logical views on the large collections of raw data, allowing efficient processing of even complex user queries selecting on both system data and the annotations. The scalable architecture, capable to handle millions of jobs in a single JP installation, and integrated into the EGEE gLite middleware environment is presented.

Distributed Tracking, Storage, and Re-use of Job State Information on the Grid

The Logging and Bookkeeping service tracks jobs passing through the Grid. It collects important events generated by both the grid middleware components and applications, and processes them at a chosen LB server to provide the job state. The events are transported through secure and reliable channels. Job tracking is fully distributed and does not depend on a single information source, the robustness is achieved through speculative job state computation in case of reordered, delayed or lost events. The state computation is easily adaptable to modified job control flow.

Job Provenance --- Insight into Very Large Provenance Datasets

Following the job-centric monitoring concept, Job Provenance (JP) service organizes provenance records on the per-job basis. It is designed to manage very large number of records, as was required in the EGEE project where it was developed originally. The quantitative aspect is also a focus of the presented demonstration. We show JP capability to retrieve data items of interest from a large dataset of full records of more than 1 million of jobs, to perform non-trivial transformation on those data, and organize the results in such a way that repeated interactive queries are possible. The application area of the demo is derived from that of previous Provenance Challenges. Though the topic of the demo -- a computational experiment -- is arranged rather artificially, the demonstration still delivers its main message that JP supports non-trivial transformations and interactive queries on large data sets.

Supporting the Development Process of the DataGrid Workload Management System Software with GNU autotools, CVS and RPM

Supporting the development of the Workload Management System in the context of the European DataGrid was a challenging task as the team was characterized by a high geographic and administrative dispersion, with developers distributed in various institutions and countries. Further, software dependencies were complex as it was required to integrate and interface a significant number of external software packages. In this paper, we discuss how a combination of Concurrent Version System, GNU autotools and other tools and practices was organized to allow the development, build, test and distribution of the software. With the proposed solution, we managed to combine ease-of-use for distributed developers while preserving the central coordination needed by the project-wide steering.