Francesco Giacomini

The gLite workload management system

The gLite Workload Management System (WMS) is a collection of components that provide the service responsible for distributing and managing tasks across computing and storage resources available on a Grid. The WMS basically receives requests of job execution from a client, finds the required appropriate resources, then dispatches and follows the jobs until completion, handling failure whenever possible. Other than single batch-like jobs, compound job types handled by the WMS are Directed Acyclic Graphs (a set of jobs where the input/output/execution of one of more jobs may depend on one or more other jobs), Parametric Jobs (multiple jobs with one parametrized description), and Collections (multiple jobs with a common description). Jobs are described via a flexible, high-level Job Definition Language (JDL). New functionality was recently added to the system (use of Service Discovery for obtaining new service endpoints to be contacted, automatic sandbox files archival/compression and sharing, support for bulk-submission and bulk-matchmaking). Intensive testing and troubleshooting allowed to dramatically increase both job submission rate and service stability. Future developments of the gLite WMS will be focused on reducing external software dependency, improving portability, robustness and usability.

Practical approaches to Grid workload and resource management in the EGEE project

Resource management and scheduling of distributed, data-driven applications in a Grid environment are challenging problems. Although significant results were achieved in the past few years, the development and the proper deployment of generic, reliable, standard components present issues that still need to be completely solved. Interested domains include workload management, resource discovery, resource matchmaking and brokering, accounting, authorization policies, resource access, reliability and dependability. The evolution towards a service-oriented architecture, supported by emerging standards, is another activity that will demand attention. All these issues are being tackled within the EU-funded EGEE project (Enabling Grids for E-science in Europe), whose primary goals are the provision of robust middleware components and the creation of a reliable and dependable Grid infrastructure to support e-Science applications. In this paper we present the plans and the preliminary activities aiming at providing adequate workload and resource management components, suitable to be deployed in a production-quality Grid.

On advance reservation of heterogeneous network paths

The availability of information about properties and status of resources is essential for Grid resource brokers. However, while abstractions of computing and storage resources already exist, the notion of Grid network resource is far from being understood today. As a result, the integration of advanced network services is still difficult when a Grid system spans large-scale heterogeneous network infrastructures. In this paper, we propose a single definition of a Grid network resource abstraction for multiple types of network connectivity. This abstraction was successfully implemented and tested in a network resource management prototype supporting a variety of network technologies.

Network monitoring for GRID performance optimization

Given the fundamental relevance of network performance for the optimization of distributed systems transactions over the network, the concept of a novel service called the Network-based Optimization Service is introduced and its architecture, based on the use of internal cost functions is presented. We show how network metrics can be combined to form complex compound metrics like the Closeness and the Proximity function. The Optimization Service can be used in a variety of different use cases; in this paper we show how Resource Brokers and Data Management can make use of network status for a considerable improvement of their decision-taking functions.

Distributed Tracking, Storage, and Re-use of Job State Information on the Grid

The Logging and Bookkeeping service tracks jobs passing through the Grid. It collects important events generated by both the grid middleware components and applications, and processes them at a chosen LB server to provide the job state. The events are transported through secure and reliable channels. Job tracking is fully distributed and does not depend on a single information source, the robustness is achieved through speculative job state computation in case of reordered, delayed or lost events. The state computation is easily adaptable to modified job control flow.

Explorations of the viability of ARM and Xeon Phi for physics processing

We report on our investigations into the viability of the ARM processor and the Intel Xeon Phi co-processor for scientic computing. We describe our experience porting software to these processors and running benchmarks using real physics applications to explore the potential of these processors for production physics processing.

A Practical Approach for a Workflow Management System

A variety of grid middlewares and workflow languages causes the existence of many workflow management systems (WfMS). Formalisms used to represent workflows vary from simple Directed Acyclic Graphs (DAG) to more complex (non deterministic) Petri Nets. Therefore a workflow description is strictly bound to a particular WfMS and to the computational resources that WfMS address, as far as no cooperation among WfMSs exists. This might be critical in scientific workflows where a large amount of resources is usually needed. In this paper we propose a WfMS that aims at language independence and Grid middleware abstraction dealing with interoperability as proposed in the reference model ofion dealing with interoperability as proposed in the reference model of the Workflow Management Coalition (WfMC). The main goal of such WfMS is to provide an effective solution to run complex scientific workflows (legacy or not) taking full advantage of the distributed and etherogeneous nature of the Grid. A Petri Net formalism has been chosen as internal representation due to its formal behavioral description and the existence of several analysis tools. Our proposed WfMS will be implemented on top of the gLite Grid middleware provided by the EGEE project because of its stability and large adoption.

First statistical analysis of Geant4 quality software metrics

Geant4 is a simulation system of particle transport through matter, widely used in several experimental areas from high energy physics and nuclear experiments to medical studies. Some of its applications may involve critical use cases; therefore they would benefit from an objective assessment of the software quality of Geant4. In this paper, we provide a first statistical evaluation of software metrics data related to a set of Geant4 physics packages. The analysis aims at identifying risks for Geant4 maintainability, which would benefit from being addressed at an early stage. The findings of this pilot study set the grounds for further extensions of the analysis to the whole of Geant4 and to other high energy physics software systems.

An integrated infrastructure in support of software development

This paper describes the design and the current state of implementation of an infrastructure made available to software developers within the Italian National Institute for Nuclear Physics (INFN) to support and facilitate their daily activity. The infrastructure integrates several tools, each providing a well-identified function: project management, version control system, continuous integration, dynamic provisioning of virtual machines, efficiency improvement, knowledge base. When applicable, access to the services is based on the INFN-wide Authentication and Authorization Infrastructure. The system is being installed and progressively made available to INFN users belonging to tens of sites and laboratories and will represent a solid foundation for the software development efforts of the many experiments and projects that see the involvement of the Institute. The infrastructure will be beneficial especially for small- and medium-size collaborations, which often cannot afford the resources, in particular in terms of know-how, needed to set up such services.

Large-scale DAQ tests for the LHCb upgrade

To increase the event yield, the LHCb experiment will undergo a major detector upgrade planned during the second long shutdown of the Large Hadron Collider (2019–2020). The new data acquisition has to process the whole 40-MHz input event rate, relying only on the large-scale computing farm implementation of the high-level trigger. Event fragments will be forwarded at 40 MHz from the detector front-end electronics to the event builder (EB), through optical links and peripheral component interconnect express cards. The EB farm, of about 500 computers, shall provide an aggregated throughput of 32 Tb/s. To reach the required EB performance, we are testing various interconnect technologies and network protocols on large-scale computing clusters. For this purpose, we have developed an EB software evaluator. We report here about the results of the measurements performed on high-performance computing facilities to test throughput and scalability.