Julia Andreeva

Distributed computing grid experiences in CMS

The CMS experiment is currently developing a computing system capable of serving, processing and archiving the large number of events that will be generated when the CMS detector starts taking data. During 2004 CMS undertook a large scale data challenge to demonstrate the ability of the CMS computing system to cope with a sustained data-taking rate equivalent to 25% of startup rate. Its goals were: to run CMS event reconstruction at CERN for a sustained period at 25 Hz input rate; to distribute the data to several regional centers; and enable data access at those centers for analysis. Grid middleware was utilized to help complete all aspects of the challenge. To continue to provide scalable access from anywhere in the world to the data, CMS is developing a layer of software that uses Grid tools to gain access to data and resources, and that aims to provide physicists with a user friendly interface for submitting their analysis jobs. This paper describes the data challenge experience with Grid infrastructure and the current development of the CMS analysis system.

Experiment Dashboard: the monitoring system for the LHC experiments

In this paper we present the Experiment Dashboard monitoring system, which is currently in use by four Large Hadron Collider (LHC)[1] experiments. The goal of the Experiment Dashboard is to monitor the activities of the LHC experiments on the distributed infrastructure, providing monitoring data from the virtual organization/user perspectives. All LHC experiments are using various Grid infrastructures (LCG/EGEE, OSG, Nordugrid) with correspondingly various middleware flavors and job submission methods. Providing a uniform and complete view of the various activities like job processing, data movement and publishing, access to distributed databases regardless of the underlying Grid flavor is the challenging task. In this paper we will describe the solutions implemented in the Experiment Dashboard.

Experiment Dashboard - a generic, scalable solution for monitoring of the LHC computing activities, distributed sites and services

The Experiment Dashboard system provides common solutions for monitoring job processing, data transfers and site/service usability. Over the last seven years, it proved to play a crucial role in the monitoring of the LHC computing activities, distributed sites and services. It has been one of the key elements during the commissioning of the distributed computing systems of the LHC experiments. The first years of data taking represented a serious test for Experiment Dashboard in terms of functionality, scalability and performance. And given that the usage of the Experiment Dashboard applications has been steadily increasing over time, it can be asserted that all the objectives were fully accomplished.

CMS Dashboard Task Monitoring: A user-centric monitoring view

We are now in a phase change of the CMS experiment where people are turning more intensely to physics analysis and away from construction. This brings a lot of challenging issues with respect to monitoring of the user analysis. The physicists must be able to monitor the execution status, application and grid-level messages of their tasks that may run at any site within the CMS Virtual Organisation. The CMS Dashboard Task Monitoring project provides this information towards individual analysis users by collecting and exposing a user-centric set of information regarding submitted tasks including reason of failure, distribution by site and over time, consumed time and efficiency. The development was user-driven with physicists invited to test the prototype in order to assemble further requirements and identify weaknesses with the application.

Experiment Dashboard for Monitoring Computing Activities of the LHC Virtual Organizations

The Large Hadron Collider (LHC) is preparing for data taking at the end of 2009. The Worldwide LHC Computing Grid (WLCG) provides data storage and computational resources for the high energy physics community. Operating the heterogeneous WLCG infrastructure, which integrates 140 computing centers in 33 countries all over the world, is a complicated task. Reliable monitoring is one of the crucial components of the WLCG for providing the functionality and performance that is required by the LHC experiments. The Experiment Dashboard system provides monitoring of the WLCG infrastructure from the perspective of the LHC experiments and covers the complete range of their computing activities. This work describes the architecture of the Experiment Dashboard system and its main monitoring applications and summarizes current experiences by the LHC experiments, in particular during service challenges performed on the WLCG over the last years.

High-Energy Physics on the Grid: the ATLAS and CMS Experience

In this paper we present the experience of the ATLAS and CMS High-Energy Physics (HEP) experiments at the Large Hadron Collider (LHC) with the LCG/EGEE Grid infrastructure. The activity developed around the following two main lines: large-scale physics and detector simulations and end-user analysis. The LCG/EGEE Grid infrastructure offers a large amount of computing and storage resources and is growing very rapidly. It provides the natural environment for large-scale physics and detector simulations. Also, the analysis of these detector simulation data (and in the near future of the reconstructed data from physics collisions) requires efficient end-users access to Grid resources. In this paper, the main findings and lessons learned in terms of performance, robustness and scalability of the whole system are discussed in detail.

Designing and developing portable large-scale JavaScript web applications within the Experiment Dashboard framework

Improvements in web browser performance and web standards compliance, as well as the availability of comprehensive JavaScript libraries, provides an opportunity to develop functionally rich yet intuitive web applications that allow users to access, render and analyse data in novel ways. However, the development of such large-scale JavaScript web applications presents new challenges, in particular with regard to code sustainability and team-based work. We present an approach that meets the challenges of large-scale JavaScript web application design and development, including client-side model-view-controller architecture, design patterns, and JavaScript libraries. Furthermore, we show how the approach leads naturally to the encapsulation of the data source as a web API, allowing applications to be easily ported to new data sources. The Experiment Dashboard framework is used for the development of applications for monitoring the distributed computing activities of virtual organisations on the Worldwide LHC Computing Grid. We demonstrate the benefits of the approach for large-scale JavaScript web applications in this context by examining the design of several Experiment Dashboard applications for data processing, data transfer and site status monitoring, and by showing how they have been ported for different virtual organisations and technologies.

Common Accounting System for Monitoring the ATLAS Distributed Computing Resources

This paper covers in detail a variety of accounting tools used to monitor the utilisation of the available computational and storage resources within the ATLAS Distributed Computing during the first three years of Large Hadron Collider data taking. The Experiment Dashboard provides a set of common accounting tools that combine monitoring information originating from many different information sources; either generic or ATLAS specific. This set of tools provides quality and scalable solutions that are flexible enough to support the constantly evolving requirements of the ATLAS user community.

hBrowse - Generic framework for hierarchical data visualisation

CERNE-mail: laura.sargsyan@cern.chThe hBrowse framework is a client-side JavaScript application that can be adjusted and imple-mented according to each specific community’s needs. It utilises the latest web technologies (e.g.jQuery framework, Highcharts plotting library and DataTables jQuery plugin) and capabilitiesthat modern browsers expose to the user. It can be combined with any kind of server as long asit can send JSON formatted data via the HTTP protocol. Each part of this software (dynamictables overlay, user selection etc.) is in fact a separate plugin which can be used separately fromthe main application. The Experiment Dashboard framework utilises hBrowse to provide genericjob monitoring applications for the ATLAS and CMS Large Handron Collider (LHC) VirtualOrganisations (VOs). hBrowse is also used in mini-Dashboard which is part of the EGI Introduc-tory Package and it is used to monitor the status of jobs submitted through the Ganga or Dianesubmission systems.EGI Community Forum 2012 / EMI Second Technical Conference26-30 March, 2012Munich, Germany

Distributed analysis in CMS

The CMS experiment expects to manage several Pbytes of data each year during the LHC programme, distributing them over many computing sites around the world and enabling data access at those centers for analysis. CMS has identified the distributed sites as the primary location for physics analysis to support a wide community with thousands potential users. This represents an unprecedented experimental challenge in terms of the scale of distributed computing resources and number of user. An overview of the computing architecture, the software tools and the distributed infrastructure is reported. Summaries of the experience in establishing efficient and scalable operations to get prepared for CMS distributed analysis are presented, followed by the user experience in their current analysis activities.