Jim Kowalkowski

PDACS: a portal for data analysis services for cosmological simulations

Accessing and analyzing data from cosmological simulations is a major challenge due to the prohibitive size of cosmological datasets and the diversity of the associated large-scale analysis tasks. Analysis of the simulated models requires direct access to the datasets, considerable compute infrastructure, and storage capacity for the results. Resource limitations can become serious obstacles to performing research on the most advanced cosmological simulations. The Portal for Data Analysis services for Cosmological Simulations (PDACS) is a web-based workflow service and scientific gateway for cosmology. The PDACS platform provides access to shared repositories for datasets, analytical tools, cosmological workflows, and the infrastructure required to perform a wide variety of analyses. PDACS is a repurposed implementation of the Galaxy workflow engine and supports a rich collection of cosmology-specific datatypes and tools. The platform leverages high-performance computing infrastructure at the National Energy Research Scientific Computing Center (NERSC) and Argonne National Laboratory (ANL), enabling researchers to deploy computationally intensive workflows. In this paper we present PDACS and discuss the process and challenges of developing a research platform for cosmological research.

Layout-aware scientific computing: a case study using MILC

Nowadays, high performance computers have more cores and nodes than ever before. Computation is spread out among them, leading to more communication. For this reason, communication can easily become the bottleneck of a system and limit its scalability. The layout of an application on a computer is the key factor to preserve communication locality and reduce its cost. In this paper, we propose a simple model to optimize the layout for scientific applications by minimizing inter-node communication cost. The model takes into account the latency and bandwidth of the network and associates them with the dominant layout variables of the application. We take MILC as an example and analyze its communication patterns. According to our experimental results, the model developed for MILC achieved a satisfactory accuracy for predicting the performance, leading to up to 31% performance improvement.

RTES demo system2004

The RTES Demo System 2004 is a prototype for reliable, fault-adaptive infrastructure applicable to commodity-based dedicated application computer farms, such as the Level 2/3 trigger for the proposed BTeV high energy physics project. This paper describes the prototype, and its demonstration at the 11th IEEE Real Time and Embedded Technology Applications Symposium, RTAS 2005.

Exploring the Performance of Spark for a Scientific Use Case

We present an evaluation of the performance of a Spark implementation of a classification algorithm in the domain of High Energy Physics (HEP). Spark is a general engine for in-memory, large-scale data processing, and is designed for applications where similar repeated analysis is performed on the same large data sets. Classification problems are one of the most common and critical data processing tasks across many domains. Many of these data processing tasks are both computation-and data-intensive, involving complex numerical computations employing extremely large data sets. We evaluated the performance of the Spark implementation on Cori, a NERSC resource, and compared the results to an untuned MPI implementation of the same algorithm. While the Spark implementation scaled well, it is not competitive in speed to our MPI implementation, even when using significantly greater computational resources.

Online monitoring in the upcoming Fermilab Tevatron run II

We describe the online event monitoring systems using ROOT for the CDF and DO collaborations in the upcoming Fermilab Tevatron runII. The CDF and DO experiments consist of many detector subsystems and will run in a high rate large bandwidth data transfer environment. In the experiments, it is crucial to monitor the performance of each subsystem and the integrity of the data, in real time with minimal interruption. ROOT is used as the main analysis tool for the monitoring systems and its GUI is used to browse the results via socket, allowing multiple GUI client connections.

Big Data in HEP: A comprehensive use case study

Experimental Particle Physics has been at the forefront of analyzing the worlds largest datasets for decades. The HEP community was the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems collectively called Big Data technologies have emerged to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (filtering and transforming experiment-specific data formats), these new technologies use different approaches and promise a fresh look at analysis of very large datasets and could potentially reduce the time-to-physics with increased interactivity. In this talk, we present an active LHC Run 2 analysis, searching for dark matter with the CMS detector, as a testbed for Big Data technologies. We directly compare the traditional NTuple-based analysis with an equivalent analysis using Apache Spark on the Hadoop ecosystem and beyond. In both cases, we start the analysis with the official experiment data formats and produce publication physics plots. We will discuss advantages and disadvantages of each approach and give an outlook on further studies needed.

PDACS: A Portal for Data Analysis Services for Cosmological Simulations

PDACS (Portal for Data Analysis Services for Cosmological Simulations) is a Web-based analysis portal that provides access to large simulations and large-scale parallel analysis tools to the research community. It provides opportunities to access, transfer, manipulate, search, and record simulation data, as well as to contribute applications and carry out (possibly complex) computational analyses of the data. PDACS also enables wrapping of analysis tools written in a large number of languages within its workflow system, providing a powerful way to carry out multilevel/multistep analyses. The system allows for cross-layer provenance tracking, implementing a transparent method for sharing workflow specifications, as well as a convenient mechanism for checking reproducibility of results generated by the workflows. Users are able to submit their own tools to the system and to share tools with the rest of the community.

The RTES project - BTeV, and beyond

The real time embedded systems (RTES) project was created to study the design and implementation of high-performance, heterogeneous, and fault-adaptive real time embedded systems. The driving application for this research was the proposed BTeV high energy physics experiment, which called for large farms of embedded computational elements (DSPs), as well as a large farm of conventional high-performance processors to implement its Level 1 and Level 2/3 triggers. At the time of BTeVs termination early in 2005, the RTES project was within days of completing a prototype implementation for providing a reliable and fault-adaptive infrastructure to the L2/3 farm; a prototype for the L1 farm had been completed in 2003. This paper documents the conclusion of the RTES focus on BTeV, and provides an evaluation of the applicability of the RTES concepts to other systems

Spark and HPC for High Energy Physics Data Analyses

A full High Energy Physics (HEP) data analysis is divided into multiple data reduction phases. Processing within these phases is extremely time consuming, therefore intermediate results are stored in files held in mass storage systems and referenced as part of large datasets. This processing model limits what can be done with interactive data analytics. Growth in size and complexity of experimental datasets, along with emerging big data tools are beginning to cause changes to the traditional ways of doing data analyses. Use of big data tools for HEP analysis looks promising, mainly because extremely large HEP datasets can be represented and held in memory across a system, and accessed interactively by encoding an analysis using high- level programming abstractions. The mainstream tools, however, are not designed for scientific computing or for exploiting the available HPC platform features. We use an example from the Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider (LHC) in Geneva, Switzerland. The LHC is the highest energy particle collider in the world. Our use case focuses on searching for new types of elementary particles explaining Dark Matter in the universe. We use HDF5 as our input data format, and Spark to implement the use case. We show the benefits and limitations of using Spark with HDF5 on Edison at NERSC.

Layout-aware scientific computing: A case study using the MILC code

Abstract Nowadays, high performance computers have more cores and nodes than ever before. Computation is spread out among them, leading to more communication cost than before. For this reason, communication can easily become the bottleneck of a system and limit its scalability. The layout of an application on a computer is the key factor to preserve communication locality and reduce its cost. In this paper, we propose a straightforward model to optimize the layout for scientific applications by minimizing inter-node communication cost. The model takes into account the latency and bandwidth of the network and associates them with the dominant layout variables of the application. We take the MILC code as an example and analyze its communication patterns. According to our experimental results, the model developed for the MILC code achieved a satisfactory accuracy for predicting the performance, leading to up to 31% performance improvement.