Daniel Kouřil

Practical approaches to Grid workload and resource management in the EGEE project

Resource management and scheduling of distributed, data-driven applications in a Grid environment are challenging problems. Although significant results were achieved in the past few years, the development and the proper deployment of generic, reliable, standard components present issues that still need to be completely solved. Interested domains include workload management, resource discovery, resource matchmaking and brokering, accounting, authorization policies, resource access, reliability and dependability. The evolution towards a service-oriented architecture, supported by emerging standards, is another activity that will demand attention. All these issues are being tackled within the EU-funded EGEE project (Enabling Grids for E-science in Europe), whose primary goals are the provision of robust middleware components and the creation of a reliable and dependable Grid infrastructure to support e-Science applications. In this paper we present the plans and the preliminary activities aiming at providing adequate workload and resource management components, suitable to be deployed in a production-quality Grid.

The DataGrid Workload Management System: Challenges and Results

The workload management task of the DataGrid project was mandated to define and implement a suitable architecture for distributed scheduling and resource management in a Grid environment. The result was the design and implementation of a Grid Workload Management System, a super-scheduler with the distinguishing property of being able to take data access requirements into account when scheduling jobs to the available Grid resources. Many novel issues in various fields were faced such as resource management, resource reservation and co-allocation, Grid accounting. In this paper, the architecture and the functionality provided by the DataGrid Workload Management System are presented.

Authentication and Authorization Mechanisms for Multi-domain Grid Environments

This article discusses the authentication and the authorization aspects of security in grid environments spanning multiple administrative domains. Achievements in these areas are presented using the EU DataGrid project as an example implementation. It also gives an outlook on future directions of development.

gLite job provenance

The Job Provenance (JP) service is designed to automate keeping track of computations on large scale Grids, giving thus users a tool to correctly archive information about their jobs and to re-submit any job in a reconstructed environment. JP provides a permanent minimal record of job (and its environment) related information, to which free-form user annotations can be added. JP also offers the capability of configuring any number of indexed logical views on the large collections of raw data, allowing efficient processing of even complex user queries selecting on both system data and the annotations. The scalable architecture, capable to handle millions of jobs in a single JP installation, and integrated into the EGEE gLite middleware environment is presented.

Distributed Tracking, Storage, and Re-use of Job State Information on the Grid

The Logging and Bookkeeping service tracks jobs passing through the Grid. It collects important events generated by both the grid middleware components and applications, and processes them at a chosen LB server to provide the job state. The events are transported through secure and reliable channels. Job tracking is fully distributed and does not depend on a single information source, the robustness is achieved through speculative job state computation in case of reordered, delayed or lost events. The state computation is easily adaptable to modified job control flow.

3.5 Million Smartmeters in the Cloud

Currently the operator of the Czech national electric power distribution network runs a pilot project of testing deployment of approximately 35,000 smart meters – devices which measure various quantities related to power consumption at the end-user side. The measured data are gath- ered along the distribution to concentrator devices (each serving several hundreds of smart meters typically) which send it further to a cluster of central servers for data processing. The pilot will be followed by deployment of approximately 100x more such devices. However, such deployment is unprecedented with the software in use. Therefore a simulation experiment is required to discover and mitigate potential scaling issues in advance. The simulation is run in the cloud infrastructure of CERIT-SC (managed by the OpenNebula stack). The central servers are deployed on hardware of similar scale and in the same software configuration as intended for the production. The data sources – smart meters are simulated at the level of concentrators. The implementation allows to simulate varying patterns of measurements and patterns of unaccessibility. Unchanged communication protocols between the concentrator and the central server are used then. Because the target setup assumes up to 50,000 concentrators, up to thousands of concentrators must be handled by a single simulator instance. The target environment uses data links of varying qualities, starting even at GPRS links. Therefore the simulator also includes modules which emulate packet loss, delay, and jitter in the network communication in a controlled way. This setup allows testing sensitivity of the entire system on network properties. The cloud is an essential flexible infrastructure to run such an experiment. With the exception of the RDBMS server running on a large SMP machine all the servers (both the central cluster and the data source simulators) are run in virtual machines in the cloud. This allows multiplatform setup (Windows and two Linux flavours are used) and, in particular, servers can be added to various groups (simulators, front-ends, application logic, etc.) easily. On the other hand, resources which are not required in a certain stage of the experiment, can be easily released to other use, which makes even such a large experiment affordable

Reputation Based Trust Management System Supporting Collaboration in a Medical Application

In a small group of people it is quite easy to start a collaboration based on shared trust, because people quickly recognize quality of each other. But this is not true when we move to the highly distributed environment with hundreds of users, not only from one institution or town, but even from different countries. It becomes very complicated task to distinguish experienced an trusted people from malicious users.

Using PKI to Provide Credential Delegation in non Web-based Federations

Authentication is basic functionality required by most services that provide access to protected resources or personalized content. In order to authenticate to services users maintain sets of credentials that they use to prove their identity. Credential delegation allows users to seamlessly access multiple services across the network. The concept manifested their utility in the scope of single domain authentication mechanisms. Therefore, emerging identity federations are expected to provide similar functions, too. Recently, various non web-based federation models have emerged, unfortunately they do not cover properly delegation of credentials. In this paper we introduce a mechanism utilizing digital certificates and PKI, which provides support for credential delegation in non web-based federations. The viability of the concept is demonstrated on integration of the mechanism with the Moonshot federation framework. However, the solution forms an independent middleware layer that can be used by several federation models.

A Race for Security: Identifying Vulnerabilities on 50 000 Hosts Faster than Attackers

Unpatched security vulnerabilities are often misused by attackers to take over machines or cause other harm to computers and their legitimate users. Having proper and timely patch management is crucial to keep the system secure and resistant to common attacks targeting known weak spots. For this, a monitoring system that is able to provide a global view of the whole infrastructure is inevitable. In this paper we present service Pakiti that makes it possible to monitor patches across a number of machines and retain a fresh overview about the patching status. Using Pakiti a system administrator can detect machine where patching failed for whatever reason or was not initiated at all.

Catch-All Virtual Organizations - Solution for Heterogeneous and Disperse Grid Users Communities

The virtual organizations form a key concept for seamless utilization of all advanced features of available worldwide grid environments, the EGEE Grid in particular. However, apart from large, well organized user communities there is a substantial, non-trivial activation barrier in adopting grid concept and creating a new VO for small research groups. To minimize this barrier, a so-called “catch-all” virtual organization approach has been implemented. Here we present two successful use cases of catch-all virtual organizations (Virtual Organization for Central Europe and for EUAsia), demonstrating the strengths of this approach, discussing their setups, encountered challenges and suggesting corresponding solutions.