Hal Finkel

HACC: Simulating sky surveys on state-of-the-art supercomputing architectures

Abstract Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the ‘Dark Universe’, dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers that enable these simulations. Here we report on HACC (Hardware/Hybrid Accelerated Cosmology Code), a recently developed and evolving cosmology N-body code framework, designed to run efficiently on diverse computing architectures and to scale to millions of cores and beyond. HACC can run on all current supercomputer architectures and supports a variety of programming models and algorithms. It has been demonstrated at scale on Cell- and GPU-accelerated systems, standard multi-core node clusters, and Blue Gene systems. HACC’s design allows for ease of portability, and at the same time, high levels of sustained performance on the fastest supercomputers available. We present a description of the design philosophy of HACC, the underlying algorithms and code structure, and outline implementation details for several specific architectures. We show selected accuracy and performance results from some of the largest high resolution cosmological simulations so far performed, including benchmarks evolving more than 3.6 trillion particles.

HACC: extreme scaling and performance across diverse architectures

Supercomputing is evolving towards hybrid and accelerator-based architectures with millions of cores. The HACC (Hardware/Hybrid Accelerated Cosmology Code) framework exploits this diverse landscape at the largest scales of problem size, obtaining high scalability and sustained performance. Developed to satisfy the science requirements of cosmological surveys, HACC melds particle and grid methods using a novel algorithmic structure that flexibly maps across architectures, including CPU/GPU, multi/many-core, and Blue Gene systems. We demonstrate the success of HACC on two very different machines, the CPU/GPU system Titan and the BG/Q systems Sequoia and Mira, attaining unprecedented levels of scalable performance. We demonstrate strong and weak scaling on Titan, obtaining up to 99.2% parallel efficiency, evolving 1.1 trillion particles. On Sequoia, we reach 13.94 PFlops (69.2% of peak) and 90% parallel efficiency on 1,572,864 cores, with 3.6 trillion particles, the largest cosmological benchmark yet performed. HACC design concepts are applicable to several other supercomputer applications.

Gravitational waves from oscillon preheating

A bstractOscillons are long-lived, localized excitations of nonlinear scalar fields which may be copiously produced during preheating after inflation, leading to a possible oscillon dominated phase in the early Universe. For example, this can happen after axion monodromy inflation, on which we run our simulations. We investigate the stochastic gravitational wave background associated with an oscillon-dominated phase. An isolated oscillon is spherically symmetric and does not radiate gravitational waves, and we show that the flux of gravitational radiation generated between oscillons is also small. However, a significant stochastic gravitational wave background may be generated during preheating itself (i.e, when oscillons are forming), and in this case the characteristic size of the oscillons is imprinted on the gravitational wave power spectrum, which has multiple, distinct peaks.

Large-scale structure formation with massive neutrinos and dynamical dark energy

Over the next decade, cosmological measurements of the large-scale structure of the Universe will be sensitive to the combined effects of dynamical dark energy and massive neutrinos. The matter power spectrum is a key repository of this information. We extend higher-order perturbative methods for computing the power spectrum to investigate these effects over quasilinear scales. Through comparison with N-body simulations, we establish the regime of validity of a time-renormalization group perturbative treatment that includes dynamical dark energy and massive neutrinos. We also quantify the accuracy of standard, renormalized and Lagrangian resummation (LPT) perturbation theories without massive neutrinos. We find that an approximation that neglects neutrino clustering as a source for nonlinear matter clustering predicts the baryon acoustic oscillation (BAO) peak position to 0.25% accuracy for redshifts

THE MIRA–TITAN UNIVERSE: PRECISION PREDICTIONS FOR DARK ENERGY SURVEYS

1\ensuremath{\le}z\ensuremath{\le}3

Cosmic Emulation: Fast Predictions for the Galaxy Power Spectrum

, justifying the use of LPT for BAO reconstruction in upcoming surveys. We release a modified version of the public Copter code which includes the additional physics discussed in the paper.

Meshing the Universe: Integrating Analysis in Cosmological Simulations

Ground and space-based sky surveys enable powerful cosmological probes based on measurements of galaxy properties and the distribution of galaxies in the Universe. These probes include weak lensing, baryon acoustic oscillations, abundance of galaxy clusters, and redshift space distortions; they are essential to improving our knowledge of the nature of dark energy. On the theory and modeling front, large-scale simulations of cosmic structure formation play an important role in interpreting the observations and in the challenging task of extracting cosmological physics at the needed precision. These simulations must cover a parameter range beyond the standard six cosmological parameters and need to be run at high mass and force resolution. One key simulation-based task is the generation of accurate theoretical predictions for observables, via the method of emulation. Using a new sampling technique, we explore an 8-dimensional parameter space including massive neutrinos and a variable dark energy equation of state. We construct trial emulators using two surrogate models (the linear power spectrum and an approximate halo mass function). The new sampling method allows us to build precision emulators from just 26 cosmological models and to increase the emulator accuracy by adding new sets of simulations in a prescribed way. This allows emulator fidelity to be systematically improved as new observational data becomes available and higher accuracy is required. Finally, using one LCDM cosmology as an example, we study the demands imposed on a simulation campaign to achieve the required statistics and accuracy when building emulators for dark energy investigations.

Large-scale compute-intensive analysis via a combined in-situ and co-scheduling workflow approach

The halo occupation distribution (HOD) approach has proven to be an effective method for modeling galaxy clustering and bias. In this approach, galaxies of a given type are probabilistically assigned to individual halos in N-body simulations. In this paper, we present a fast emulator for predicting the fully nonlinear galaxy–galaxy auto and galaxy–dark matter cross power spectrum and correlation function over a range of freely specifiable HOD modeling parameters. The emulator is constructed using results from 100 HOD models run on a large ΛCDM N-body simulation, with Gaussian Process interpolation applied to a PCA-based representation of the galaxy power spectrum. The total error is currently ∼1% in the auto correlations and ∼2% in the cross correlations from z = 1 to z = 0, over the considered parameter range. We use the emulator to investigate the accuracy of various analytic prescriptions for the galaxy power spectrum, parametric dependencies in the HOD model, and the behavior of galaxy bias as a function of HOD parameters. Additionally, we obtain fully nonlinear predictions for tangential shear correlations induced by galaxy–galaxy lensing from our galaxy–dark matter cross power spectrum emulator. All emulation products are publicly available at http://www.hep.anl.gov/cosmology/CosmicEmu/emu.html.

Multi-moment ADER-Taylor methods for systems of conservation laws with source terms in one dimension

Mesh tessellations are indispensable tools for analyzing point data because they transform sparse discrete samples into dense continuous functions. Meshing the output of petascale simulations, however, can be as data-intensive as the simulations themselves and often must be executed in parallel on the same supercomputers in order to fit in memory. To date, however, no general-purpose large-scale parallel tessellation tools exist. We present a prototype method for computing such a Voronoi tessellation in situ during cosmological simulations. In principle, similar methods can be applied to other computational geometry problems such as Delaunay tetrahedralizations and convex hulls in other science domains. We demonstrate the utility of our approach as part of an in situ cosmology tools framework that runs various analysis tools at selected time steps, saves results to parallel storage, and includes visualization and further analysis in a widely used visualization package. In the example highlighted in this paper, connected components of Voronoi cells are interrogated to detect and characterize cosmological voids.

The Mira-Titan Universe. II. Matter Power Spectrum Emulation

Large-scale simulations can produce hundreds of terabytes to petabytes of data, complicating and limiting the efficiency of workflows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in-situ, utilizing the same resources as the simulation, and/or off-loading subsets of the data to a compute-intensive analysis system. We introduce an analysis framework developed for HACC, a cosmological N-body code, that uses both in-situ and co-scheduling approaches for handling petabyte-scale outputs. We compare different analysis set-ups ranging from purely off-line, to purely in-situ to in-situ/co-scheduling. The analysis routines are implemented using the PISTON/VTK-m framework, allowing a single implementation of an algorithm that simultaneously targets a variety of GPU, multi-core, and many-core architectures.