Robert Harkness

ENZO: AN ADAPTIVE MESH REFINEMENT CODE FOR ASTROPHYSICS

This paper describes the open-source code Enzo, which uses block-structured adaptive mesh refinement to provide high spatial and temporal resolution for modeling astrophysical fluid flows. The code is Cartesian, can be run in one, two, and three dimensions, and supports a wide variety of physics including hydrodynamics, ideal and non-ideal magnetohydrodynamics, N-body dynamics (and, more broadly, self-gravity of fluids and particles), primordial gas chemistry, optically thin radiative cooling of primordial and metal-enriched plasmas (as well as some optically-thick cooling models), radiation transport, cosmological expansion, and models for star formation and feedback in a cosmological context. In addition to explaining the algorithms implemented, we present solutions for a wide range of test problems, demonstrate the codes parallel performance, and discuss the Enzo collaborations code development methodology.

Introducing Enzo, an AMR Cosmology Application

In this paper we introduce Enzo, a 3D MPI-parallel Eulerian block-structured adaptive mesh refinement cosmology code. Enzo is designed to simulate cosmological structure formation, but can also be used to simulate a wide range of astrophysical situations. Enzo solves dark matter N-body dynamics using the particle-mesh technique. The Poisson equation is solved using a combination of fast fourier transform (on a periodic root grid) and multigrid techniques (on non-periodic subgrids). Euler’s equations of hydrodynamics are solved using a modified version of the piecewise parabolic method. Several additional physics packages are implemented in the code, including several varieties of radiative cooling, a metagalactic ultraviolet background, and prescriptions for star formation and feedback. We also show results illustrating properties of the adaptive mesh portion of the code. Information on profiling and optimizing the performance of the code can be found in the contribution by James Bordner in this volume.

Fully coupled simulation of cosmic reionization. I. numerical methods and tests

The Astrophysical Journal Supplement Series, 216:16 (24pp), 2015 January C 2015. doi:10.1088/0067-0049/216/1/16 The American Astronomical Society. All rights reserved. FULLY COUPLED SIMULATION OF COSMIC REIONIZATION. I. NUMERICAL METHODS AND TESTS Michael L. Norman 1,2 , Daniel R. Reynolds 3 , Geoffrey C. So 1 , Robert P. Harkness 2,4 , and John H. Wise 5 CASS, University of California, San Diego, 9500 Gilman Drive La Jolla, CA 92093-0424, USA SDSC, University of California, San Diego, 9500 Gilman Drive La Jolla, CA 92093-0505, USA 3 Southern Methodist University, 6425 Boaz Lane, Dallas, TX 75205, USA 4 NICS, Oak Ridge National Laboratory, 1 Bethel Valley Road, Oak Ridge, TN 37831, USA 5 Center for Relativistic Astrophysics, Georgia Institute of Technology, 837 State Street, Atlanta, GA 30332, USA Received 2013 June 8; accepted 2014 November 23; published 2015 January 9 ABSTRACT We describe an extension of the Enzo code to enable fully coupled radiation hydrodynamical simulation of inhomogeneous reionization in large ∼(100 Mpc) 3 cosmological volumes with thousands to millions of point sources. We solve all dynamical, radiative transfer, thermal, and ionization processes self-consistently on the same mesh, as opposed to a postprocessing approach which coarse-grains the radiative transfer. We do, however, employ a simple subgrid model for star formation which we calibrate to observations. The numerical method presented is a modification of an earlier method presented in Reynolds et al. differing principally in the operator splitting algorithm we use to advance the system of equations. Radiation transport is done in the gray flux-limited diffusion (FLD) approximation, which is solved by implicit time integration split off from the gas energy and ionization equations, which are solved separately. This results in a faster and more robust scheme for cosmological applications compared to the earlier method. The FLD equation is solved using the hypre optimally scalable geometric multigrid solver from LLNL. By treating the ionizing radiation as a grid field as opposed to rays, our method is scalable with respect to the number of ionizing sources, limited only by the parallel scaling properties of the radiation solver. We test the speed and accuracy of our approach on a number of standard verification and validation tests. We show by direct comparison with Enzo’s adaptive ray tracing method Moray that the well-known inability of FLD to cast a shadow behind opaque clouds has a minor effect on the evolution of ionized volume and mass fractions in a reionization simulation validation test. We illustrate an application of our method to the problem of inhomogeneous reionization in a 80 Mpc comoving box resolved with 3200 3 Eulerian grid cells and dark matter particles. Key words: cosmology: theory – methods: numerical – radiative transfer redshift interval 6 z 10 using the Hubble Space Telescope support the galaxy reionizer hypothesis, with the caveat that the faint end of the luminosity function which contributes substantially to the number of ionizing photons has not yet been measured (Robertson et al. 2010; Bouwens et al. 2012). Given the paucity of observational information about the pro- cess of cosmic reionization, researchers have resorted to theory and numerical simulation to fill in the blanks. As reviewed by Trac & Gnedin (2011), progress in this area has been dramatic, driven by a synergistic interplay between semi-analytic ap- proaches and numerical simulations. The combination of these two approaches have converged on a qualitative picture of how H reionization proceeds assuming the primary ionizing sources are young, star-forming galaxies. The physics of the reioniza- tion process is determined by the physics of the sources and sinks of ionizing radiation in an expanding universe. Adopting the ΛCDM model of structure formation, galaxies form hierar- chically through the merger of dark matter halos. The structure and evolution of the dark matter density field is now well un- derstood through ultra-high-resolution numerical N-body sim- ulations (Springel et al. 2005; Klypin et al. 2011) and through analytic models based on these simulations (Cooray & Sheth 2002). By making certain assumptions about how ionizing light traces mass and the dynamics of H ii regions, a basic picture of the reionization process has emerged (Furlanetto et al. 2004, 2006; Iliev et al. 2006b; Zahn et al. 2007) that is confirmed by detailed numerical simulations; e.g., Zahn et al. (2011). The basic picture is that galaxies form in the peaks of the dark matter density field and drive expanding H ii regions into their surroundings by virtue of the UV radiation emitted from young, massive stars. These H ii regions are initially isolated, 1. INTRODUCTION The epoch of reionization (EoR) is a current frontier of cos- mological research both observationally and theoretically. Ob- servations constrain the transition from a largely neutral inter- galactic medium (IGM) of primordial gas to a largely ionized one (singly ionized H and He) to the redshift interval z ∼ 11–6, which is a span of roughly 500 Myr. The completion of H reion- ization by z ≈ 6 is firmly established through quasar absorption line studies to luminous, high-redshift quasars which exhibit Lyα Gunn–Peterson absorption troughs (Fan et al. 2006). The precise onset of H reionization (presumably tied to the formation of the first luminous ionizing sources) is presently unknown ob- servationally, however, cosmic microwave background (CMB) measurements of the Thomson optical depth to the surface of last scattering by the Wilkinson Microwave Anisotropy Probe (WMAP) and Planck satellites indicates that the IGM was sub- stantially ionized by z ∼ 10 (Spergel et al. 2003; Komatsu et al. 2009; Jarosik et al. 2011; Planck Collaboration et al. 2014). Since the optical depth measurement is redshift-integrated and averaged over the sky, the CMB observations provide no infor- mation about how reionization proceeded or the nature of the radiation sources that caused it. It is generally believed that reionization begins with the formation of Population III stars at z ∼ 20–30 (Abel et al. 2002; Yoshida et al. 2003; Bromm & Larson 2004; Sokasian et al. 2004), but that soon the ionizing photon budget becomes dominated by young, star forming galaxies (see e.g., Wise et al. 2012; Xu et al. 2013), and to a lesser extent by the first quasars (Madau et al. 1999; Bolton & Haehnelt 2007; Haardt & Madau 2012; Becker & Bolton 2013). Observations of galaxies in the

Exploring the hyper-grid idea with grand challenge applications: the DEISA-TeraGrid interoperability demonstration

A supercomputing hyper-grid spanning two continents was created to move a step towards interoperability of leading grids. A dedicated network connection was established between DEISA, the leading European supercomputing grid, and TeraGrid, the leading American supercomputing grid. Both grids have adopted the approach of establishing a common, high performance global file system, the wide-area version of IBMs GPFS. Teragrids approach is based on a single site server solution under Linux, hosted by San Diego Supercomputer Centre, DEISAs approach is a multi-site server solution, with currently servers in France, Germany and Italy. These two grid-internal global file systems were interconnected over a dedicated, trusted network connection. During the Supercomputing Conference 2005, grand challenge applications were carried out both within DEISA and within TeraGrid, and results were written transparently to the combined global file system with physically distributed locations of the involved disk systems. Simulations were carried out in Europe and in America, and results were directly written to the respective remote continent, accessible for all participating scientists in both continents, and were then directly further processed for visualization in a third location, the SC05 exhibition hall in Seattle. Grand challenge applications used for the demo included a protein structure prediction and a cosmological simulation carried out at San Diego Supercomputer Center (SDSC), US (www.sdsc.edu) and a Gyrokinetic turbulence simulation and also a cosmological simulation carried out at Garching Computing Centre of the Max Planck Society (RZG), Germany (www.rzg.mpg.de)

Baryon acoustic oscillations in the Lyman alpha forest

We use hydrodynamic cosmological simulations in a (600 Mpc)3 volume to study the observability of baryon acoustic oscillations (BAO) in the intergalactic medium as probed by Lyman alpha forest (LAF) absorption. The large scale separation between the wavelength of the BAO mode (~150 Mpc) and the size of LAF absorbers (~100 kpc) makes this a numerically challenging problem. We report on several 20483 simulations of the LAF using the ENZO code. We adopt WMAP5 concordance cosmological parameters and power spectrum including BAO perturbations. 5000 synthetic HI absorption line spectra are generated randomly piercing the box face. We calculate the cross-correlation function between widely separated pairs. We detect the BAO signal at z=3 where theory predicts to moderate statistical significance.

Direct Numerical Simulations of Cosmological Reionization: Field Comparison: Density

The light from early galaxies had a dramatic impact on the gasses filling the universe. This video highlights the spatial structure of the lights effect, by comparing two simulations: one with a self-consistent radiation field (radiative), and one without (non-radiative), each with a very high dynamic range. Looking at the simulations side-by-side its hard to see any difference. However, because the simulations have the same initial conditions, we can directly compare them, by looking at the relative difference of the density. The coral-like blobs are regions where light has radiated out, heating the gas, and raising the pressure. The red regions show where the density is much higher in the radiative simulation, while the yellow regions are where the non-radiative has more density, showing where gravity was able to pull the filaments into tighter cylinders, without having to work against pressure from stellar heating. This is the first known visualization of this process, known as Jeans smoothing.

Direct Numerical Simulations of Cosmological Reionization: Field Comparison: Ionization Fraction

The light from early galaxies had a dramatic impact on the gasses filling the universe. This video highlights the spatial structure of the lights effect, by comparing two simulations: one with a self-consistent radiation field (radiative), and one without (non-radiative), each with a very high dynamic range. Ionization fraction is the amount of the gas that has been ionized. Looking at this quantity from the simulations side-by-side one can clearly see differences but it can be difficult to decipher how the regions of concentration in the two simulations relate to one another. However, because the simulations have the same initial conditions, we can directly compare them, by looking at the relative difference of the ionization fraction in a single view. The yellow and red regions show where the gas has been ionized in the radiative simulation, while at the center of these blobs are small blue regions where the ionized gas from the non-radiative simulation is concentrated. The purple illustrates the boundary at the advancing edge of the ionization from the radiative simulation, where the two simulations are the same.