V. Tsulaia

The ATLAS Event Service: A new approach to event processing

The ATLAS Event Service (ES) implements a new fine grained approach to HEP event processing, designed to be agile and efficient in exploiting transient, short-lived resources such as HPC hole-filling, spot market commercial clouds, and volunteer computing. Input and output control and data flows, bookkeeping, monitoring, and data storage are all managed at the event level in an implementation capable of supporting ATLAS-scale distributed processing throughputs (about 4M CPU-hours/day). Input data flows utilize remote data repositories with no data locality or pre-staging requirements, minimizing the use of costly storage in favor of strongly leveraging powerful networks. Object stores provide a highly scalable means of remotely storing the quasi-continuous, fine grained outputs that give ES based applications a very light data footprint on a processing resource, and ensure negligible losses should the resource suddenly vanish. We will describe the motivations for the ES system, its unique features and capabilities, its architecture and the highly scalable tools and technologies employed in its implementation, and its applications in ATLAS processing on HPCs, commercial cloud resources, volunteer computing, and grid resources.Notice: This manuscript has been authored by employees of Brookhaven Science Associates, LLC under Contract No. DE-AC02-98CH10886 with the U.S. Department of Energy. The publisher by accepting the manuscript for publication acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes.

Fine grained event processing on HPCs with the ATLAS Yoda system

High performance computing facilities present unique challenges and opportunities for HEP event processing. The massive scale of many HPC systems means that fractionally small utilization can yield large returns in processing throughput. Parallel applications which can dynamically and efficiently fill any scheduling opportunities the resource presents benefit both the facility (maximal utilization) and the (compute-limited) science. The ATLAS Yoda system provides this capability to HEP-like event processing applications by implementing event-level processing in an MPI-based master-client model that integrates seamlessly with the more broadly scoped ATLAS Event Service. Fine grained, event level work assignments are intelligently dispatched to parallel workers to sustain full utilization on all cores, with outputs streamed off to destination object stores in near real time with similarly fine granularity, such that processing can proceed until termination with full utilization. The system offers the efficiency and scheduling flexibility of preemption without requiring the application actually support or employ check-pointing. We will present the new Yoda system, its motivations, architecture, implementation, and applications in ATLAS data processing at several US HPC centers.

The description of the ATLAS detector.

The ATLAS Detector consists of several major subsystems: an inner detector composed of pixels, micro-strip detectors and a transition radiation tracker; electromagnetic and hadronic calorimetry, and a muon spectrometer. Over the last year, these systems have been described in terms of a set of geometrical primitives known as GeoModel. Software components for detector description interpret structured data from a relational database and build from that a complete description of the detector. This description is now used in the GEANT-4 based simulation program and also for reconstruction. Detector-specific services that are not handled in a generic way (e.g strip pitches and calorimetric tower boundaries) are added as an additional layer which is synched to the raw geometry. The ATLAS geometry system in the last year has undergone extensive visual debugging, and experience with the new system has been gained not only though the data challenge but also through the combined test beam. This paper gives an overview of the ATLAS detector description and discusses operational experience with the system in the data challenges and com-

THE GEANT4-BASED ATLAS FAST ELECTROMAGNETIC SHOWER SIMULATION

We present a three-pronged approach to fast electromagnetic shower simulation in ATLAS. Parameterisation is used for high-energy, shower libraries for medium-energy, and an averaged energy deposition for very low-energy particles. We present a comparison between the fast simulation and full simulation in an ATLAS Monte Carlo production.

Status and Future Evolution of the ATLAS Offline Software

These proceedings give a summary of the many software upgrade projects undertaken to prepare ATLAS for the challenges of Run-2 of the LHC. Those projects include a significant reduction of the CPU time required for reconstruction of real data with high average pile-up event rates compared to 2012. This is required to meet the challenges of the expected increase in pileup and the higher data taking rate of up to 1 kHz. By far the most ambitious project is the implementation of a completely new analysis model, based on a new ROOT readable reconstruction format, xAOD. The new model also includes a reduction framework based on a train model to centrally produce skimmed data samples and an analysis framework. These proceedings close with a brief overview of future software projects and plans that will lead up to the coming Long Shutdown 2 as the next major ATLAS software upgrade phase.

Running ATLAS workloads within massively parallel distributed applications using Athena Multi-Process framework (AthenaMP)

AthenaMP is a multi-process version of the ATLAS reconstruction, simulation and data analysis framework Athena. By leveraging Linux fork and copy-on-write mechanisms, it allows for sharing of memory pages between event processors running on the same compute node with little to no change in the application code. Originally targeted to optimize the memory footprint of reconstruction jobs, AthenaMP has demonstrated that it can reduce the memory usage of certain configurations of ATLAS production jobs by a factor of 2. AthenaMP has also evolved to become the parallel event-processing core of the recently developed ATLAS infrastructure for fine-grained event processing (Event Service) which allows the running of AthenaMP inside massively parallel distributed applications on hundreds of compute nodes simultaneously. We present the architecture of AthenaMP, various strategies implemented by AthenaMP for scheduling workload to worker processes (for example: Shared Event Queue and Shared Distributor of Event Tokens) and the usage of AthenaMP in the diversity of ATLAS event processing workloads on various computing resources: Grid, opportunistic resources and HPC.

The Virtual Point 1 event display for the ATLAS experiment

We present an event display for the ATLAS Experiment, called Virtual Point 1 (VP1), designed initially for deployment at point 1 of the LHC, the location of the ATLAS detector. The Qt/OpenGL based application provides truthful and interactive 3D representations of both event and non-event data, and now serves a general-purpose role within the experiment. Thus, VP1 is used both online (in the control room itself or remotely via a special live mode) and offline environments to provide fast debugging and understanding of events, detector status and software. In addition to a flexible plugin infrastructure and a high level of configurability, this multi-purpose role is mainly facilitated by embedding the application directly into the ATLAS offline software framework, enabling it to use the native Event Data Model directly, and thus run on any source of ATLAS data, or even directly from within processes such as reconstruction jobs. Finally, VP1 provides high-quality pictures and movies, useful for outreach purposes.

Scaling up ATLAS Event Service to production levels on opportunistic computing platforms

Continued growth in public cloud and HPC resources is on track to exceed the dedicated resources available for ATLAS on the WLCG. Examples of such platforms are Amazon AWS EC2 Spot Instances, Edison Cray XC30 supercomputer, backfill at Tier 2 and Tier 3 sites, opportunistic resources at the Open Science Grid (OSG), and ATLAS High Level Trigger farm between the data taking periods. Because of specific aspects of opportunistic resources such as preemptive job scheduling and data I/O, their efficient usage requires workflow innovations provided by the ATLAS Event Service. Thanks to the finer granularity of the Event Service data processing workflow, the opportunistic resources are used more efficiently. We report on our progress in scaling opportunistic resource usage to double-digit levels in ATLAS production.

I/O Strategies for Multicore Processing in ATLAS

A critical component of any multicore/manycore application architecture is the handling of input and output. Even in the simplest of models, design decisions interact both in obvious and in subtle ways with persistence strategies. When multiple workers handle I/O independently using distinct instances of a serial I/O framework, for example, it may happen that because of the way data from consecutive events are compressed together, there may be serious inefficiencies, with workers redundantly reading the same buffers, or multiple instances thereof. With shared reader strategies, caching and buffer management by the persistence infrastructure and by the control framework may have decisive performance implications for a variety of design choices. Providing the next event may seem straightforward when all event data are contiguously stored in a block, but there may be performance penalties to such strategies when only a subset of a given events data are needed; conversely, when event data are partitioned by type in persistent storage, providing the next event becomes more complicated, requiring marshalling of data from many I/O buffers. Output strategies pose similarly subtle problems, with complications that may lead to significant serialization and the possibility of serial bottlenecks, either during writing or in post-processing, e.g., during data stream merging. In this paper we describe the I/O components of AthenaMP, the multicore implementation of the ATLAS control framework, and the considerations that have led to the current design, with attention to how these I/O components interact with ATLAS persistent data organization and infrastructure.

ATLAS DETECTOR SIMULATION: STATUS AND OUTLOOK

The simulation program for the ATLAS experiment at CERN is currently in a full operational mode and integrated into the ATLASs common analysis framework, ATHENA. The OO approach, based on GEANT4, and in use during the DC2 data challenge has been interfaced within ATHENA and to GEANT4 using the LCG dictionaries and Python scripting. The robustness of the application was proved during the DC2 data challenge. The Python interface has added the flexibility, modularity and interactivity that the simulation tool needs to tackle, in a common way, different full ATLAS simulations setups, test beams and cosmic ray studies. Generation, simulation and digitization steps were exercised for performance and robustness tests. The comparison with real data has been possible in the context of the ATLAS combined test beam (2004) and ongoing cosmic ray studies