Jakub T. Moscicki

DIANE - distributed analysis environment for GRID-enabled simulation and analysis of physics data

Distributed analysis environment (DIANE) is the result of R&D in CERN IT Division focused on interfacing semi-interactive parallel applications with distributed GRID technology. DIANE provides a master-worker workflow management layer above low-level GRID services. DIANE is application and language-neutral. Component-container architecture and component adapters provide flexibility necessary to fulfill the diverse requirements of distributed applications. Physical transport layer assures interoperability with existing middleware frameworks based on Web services. Several distributed simulations based on Geant 4 were deployed and tested in real-life scenarios with DIANE.

Dynamic workload balancing of parallel applications with user-level scheduling on the Grid

This paper suggests a hybrid resource management approach for efficient parallel distributed computing on the Grid. It operates on both application and system levels, combining user-level job scheduling with dynamic workload balancing algorithm that automatically adapts a parallel application to the heterogeneous resources, based on the actual resource parameters and estimated requirements of the application. The hybrid environment and the algorithm for automated load balancing are described, the influence of resource heterogeneity level is measured, and the speedup achieved with this technique is demonstrated for different types of applications and resources.

The User-Level Scheduling of Divisible Load Parallel Applications With Resource Selection and Adaptive Workload Balancing on the Grid

This paper presents a hybrid resource management environment, operating on both application and system levels developed for minimizing the execution time of parallel applications with divisible workload on heterogeneous grid resources. The system is based on the adaptive workload balancing algorithm (AWLB) incorporated into the distributed analysis environment (DIANE) user-level scheduling (ULS) environment. The AWLB ensures optimal workload distribution based on the discovered application requirements and measured resource parameters. The ULS maintains the user-level resource pool, enables resource selection and controls the execution. We present the results of performance comparison of default self-scheduling used in DIANE with AWLB-based scheduling, evaluate dynamic resource pool and resource selection mechanisms, and examine dependencies of application performance on aggregate characteristics of selected resources and application profile.

hBrowse - Generic framework for hierarchical data visualisation

CERNE-mail: laura.sargsyan@cern.chThe hBrowse framework is a client-side JavaScript application that can be adjusted and imple-mented according to each specific community’s needs. It utilises the latest web technologies (e.g.jQuery framework, Highcharts plotting library and DataTables jQuery plugin) and capabilitiesthat modern browsers expose to the user. It can be combined with any kind of server as long asit can send JSON formatted data via the HTTP protocol. Each part of this software (dynamictables overlay, user selection etc.) is in fact a separate plugin which can be used separately fromthe main application. The Experiment Dashboard framework utilises hBrowse to provide genericjob monitoring applications for the ATLAS and CMS Large Handron Collider (LHC) VirtualOrganisations (VOs). hBrowse is also used in mini-Dashboard which is part of the EGI Introduc-tory Package and it is used to monitor the status of jobs submitted through the Ganga or Dianesubmission systems.EGI Community Forum 2012 / EMI Second Technical Conference26-30 March, 2012Munich, Germany

Lattice QCD thermodynamics on the Grid

Abstract We describe how we have used simultaneously O ( 10 3 ) nodes of the EGEE Grid, accumulating ca. 300 CPU-years in 2–3 months, to determine an important property of Quantum Chromodynamics. We explain how Grid resources were exploited efficiently and with ease, using user-level overlay based on Ganga and DIANE tools above standard Grid software stack. Application-specific scheduling and resource selection based on simple but powerful heuristics allowed to improve efficiency of the processing to obtain desired scientific results by a specified deadline. This is also a demonstration of combined use of supercomputers, to calculate the initial state of the QCD system, and Grids, to perform the subsequent massively distributed simulations. The QCD simulation was performed on a 16 3 × 4 lattice. Keeping the strange quark mass at its physical value, we reduced the masses of the up and down quarks until, under an increase of temperature, the system underwent a second-order phase transition to a quark–gluon plasma. Then we measured the response of this system to an increase in the quark density. We find that the transition is smoothened rather than sharpened. If confirmed on a finer lattice, this finding makes it unlikely for ongoing experimental searches to find a QCD critical point at small chemical potential.

Biomedical applications on the GRID: efficient management of parallel jobs

Distributed computing based on the Master-Worker and PULL interaction model is applicable to a number of applications in high energy physics, medical physics and bio-informatics. We demonstrate a realistic medical physics use-case of a dosimetric system for brachytherapy using distributed GRID resources. We present the efficient techniques for running parallel jobs in a case of the BLAST, a gene sequencing application, as well as for the Monte Carlo simulation based on Geant4. We present a strategy for improving the runtime performance and robustness of the jobs as well as for the minimization of the development time needed to migrate the applications to a distributed environment.

Distributed analysis in ATLAS using GANGA

Distributed data analysis using Grid resources is one of the fundamental applications in high energy physics to be addressed and realized before the start of LHC data taking. The needs to manage the resources are very high. In every experiment up to a thousand physicists will be submitting analysis jobs to the Grid. Appropriate user interfaces and helper applications have to be made available to assure that all users can use the Grid without expertise in Grid technology. These tools enlarge the number of Grid users from a few production administrators to potentially all participating physicists. The GANGA job management system (http://cern.ch/ganga), developed as a common project between the ATLAS and LHCb experiments, provides and integrates these kind of tools. GANGA provides a simple and consistent way of preparing, organizing and executing analysis tasks within the experiment analysis framework, implemented through a plug-in system. It allows trivial switching between running test jobs on a local batch system and running large-scale analyzes on the Grid, hiding Grid technicalities. We will be reporting on the plug-ins and our experiences of distributed data analysis using GANGA within the ATLAS experiment. Support for all Grids presently used by ATLAS, namely the LCG/EGEE, NDGF/NorduGrid, and OSG/PanDA is provided. The integration and interaction with the ATLAS data management system DQ2 into GANGA is a key functionality. An intelligent job brokering is set up by using the job splitting mechanism together with data-set and file location knowledge. The brokering is aided by an automated system that regularly processes test analysis jobs at all ATLAS DQ2 supported sites. Large numbers of analysis jobs can be sent to the locations of data following the ATLAS computing model. GANGA supports amongst other things tasks of user analysis with reconstructed data and small scale production of Monte Carlo data.

From DICOM to GRID: a dosimetric system for brachytherapy born from HEP

In brachytherapy software defines the experimental configuration of radioactive seeds appropriate to achieve the desired dose distribution in the patient. We will show you how nowadays it is possible to develop a software for brachytherapy which achieves the goal of high accuracy and speed thanks to the combination of various software toolkits: Geant4 simulation toolkit, AIDA analysis toolkit, GRID and the Web. Geant4-based brachytherapy application calculates dose distribution in tissues with great accuracy, in a realistic experimental set-up derived from CT data. The AIDA analysis toolkit provides the elaboration of simulation results. It is possible to run Geant4-based brachytherapy application through a Web portal, sharing distributed computing resources thanks to the integration in the GRID, making possible even to modest size hospitals to profit of advanced treatment planning tools.

SWAN: a Service for Interactive Analysis in the Cloud

Abstract SWAN (Service for Web based ANalysis) is a platform to perform interactive data analysis in the cloud. SWAN allows users to write and run their data analyses with only a web browser, leveraging on the widely-adopted Jupyter notebook interface. The user code, executions and data live entirely in the cloud. SWAN makes it easier to produce and share results and scientific code, access scientific software, produce tutorials and demonstrations as well as preserve analyses. Furthermore, it is also a powerful tool for non-scientific data analytics. This paper describes how a pilot of the SWAN service was implemented and deployed at CERN. Its backend combines state-of-the-art software technologies with a set of existing IT services such as user authentication, virtual computing infrastructure, mass storage, file synchronisation and sharing, specialised clusters and batch systems. The added value of this combination of services is discussed, with special focus on the opportunities offered by the CERNBox service and its massive storage backend, EOS. In particular, it is described how a cloud-based analysis model benefits from synchronised storage and sharing capabilities.

Key developments of the Ganga task-management framework

Ganga is the main end-user distributed analysis tool for the ATLAS and LHCb experiments and provides the foundation layer for the HammerCloud system, used by the LHC experiments for validation and stress testing of their numerous distributed computing facilities. Here we illustrate recent developments and demonstrate how tools that were initially developed for a specific user community have been migrated into the Ganga core, and so can be exploited by a wider user-base. Similarly, examples will be given where Ganga components have been adapted for use by communities in their custom analysis packages.