Michael Zingale

FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes

We report on the completion of the first version of a new-generation simulation code, FLASH. The FLASH code solves the fully compressible, reactive hydrodynamic equations and allows for the use of adaptive mesh refinement. It also contains state-of-the-art modules for the equations of state and thermonuclear reaction networks. The FLASH code was developed to study the problems of nuclear flashes on the surfaces of neutron stars and white dwarfs, as well as in the interior of white dwarfs. We expect, however, that the FLASH code will be useful for solving a wide variety of other problems. This first version of the code has been subjected to a large variety of test cases and is currently being used for production simulations of X-ray bursts, Rayleigh-Taylor and Richtmyer-Meshkov instabilities, and thermonuclear flame fronts. The FLASH code is portable and already runs on a wide variety of massively parallel machines, including some of the largest machines now extant.

A comparative study of the turbulent Rayleigh–Taylor instability using high-resolution three-dimensional numerical simulations: The Alpha-Group collaboration

The turbulent Rayleigh–Taylor instability is investigated in the limit of strong mode-coupling using a variety of high-resolution, multimode, three dimensional numerical simulations (NS). The perturbations are initialized with only short wavelength modes so that the self-similar evolution (i.e., bubble diameter Db∝amplitude hb) occurs solely by the nonlinear coupling (merger) of saturated modes. After an initial transient, it is found that hb∼αbAgt2, where A=Atwood number, g=acceleration, and t=time. The NS yield Db∼hb/3 in agreement with experiment but the simulation value αb∼0.025±0.003 is smaller than the experimental value αb∼0.057±0.008. By analyzing the dominant bubbles, it is found that the small value of αb can be attributed to a density dilution due to fine-scale mixing in our NS without interface reconstruction (IR) or an equivalent entrainment in our NS with IR. This may be characteristic of the mode coupling limit studied here and the associated αb may represent a lower bound that is insensiti...

On validating an astrophysical simulation code

We present a case study of validating an astrophysical simulation code. Our study focuses on validating FLASH, a parallel, adaptive-mesh hydrodynamics code for studying the compressible, reactive flows found in many astrophysical environments. We describe the astrophysics problems of interest and the challenges associated with simulating these problems. We describe methodology and discuss solutions to difficulties encountered in verification and validation. We describe verification tests regularly administered to the code, present the results of new verification tests, and outline a method for testing general equations of state. We present the results of two validation tests in which we compared simulations to experimental data. The first is of a laser-driven shock propagating through a multilayer target, a configuration subject to both Rayleigh-Taylor and Richtmyer-Meshkov instabilities. The second test is a classic Rayleigh-Taylor instability, where a heavy fluid is supported against the force of gravity by a light fluid. Our simulations of the multilayer target experiments showed good agreement with the experimental results, but our simulations of the Rayleigh-Taylor instability did not agree well with the experimental results. We discuss our findings and present results of additional simulations undertaken to further investigate the Rayleigh-Taylor instability.

CASTRO: A NEW COMPRESSIBLE ASTROPHYSICAL SOLVER. I. HYDRODYNAMICS AND SELF-GRAVITY

We present a new code, CASTRO, that solves the multicomponent compressible hydrodynamic equations for astrophysical flows including self-gravity, nuclear reactions, and radiation. CASTRO uses an Eulerian grid and incorporates adaptive mesh refinement (AMR). Our approach to AMR uses a nested hierarchy of logically rectangular grids with simultaneous refinement in both space and time. The radiation component of CASTRO will be described in detail in the next paper, Part II, of this series.

LOW MACH NUMBER MODELING OF TYPE Ia SUPERNOVAE. I. HYDRODYNAMICS

We introduce a low Mach number equation set for the large-scale numerical simulation of carbon-oxygen white dwarfs experiencing a thermonuclear deflagration. Since most of the interesting physics in a Type Ia supernova transpires at Mach numbers from 0.01 to 0.1, such an approach enables both a considerable increase in accuracy and a savings in computer time compared with frequently used compressible codes. Our equation set is derived from the fully compressible equations using low Mach number asymptotics, but without any restriction on the size of perturbations in density or temperature. Comparisons with simulations that use the fully compressible equations validate the low Mach number model in regimes where both are applicable. Comparisons to simulations based on the more traditional anelastic approximation also demonstrate the agreement of these models in the regime for which the anelastic approximation is valid. For low Mach number flows with potentially finite amplitude variations in density and temperature, the low Mach number model overcomes the limitations of each of the more traditional models and can serve as the basis for an accurate and efficient simulation tool.

Three-dimensional Numerical Simulations of Rayleigh-Taylor Unstable Flames in Type Ia Supernovae

Flame instabilities play a dominant role in accelerating the burning front to a large fraction of the speed of sound in a Type Ia supernova. We present a three-dimensional numerical simulation of a Rayleigh-Taylor unstable carbon flame, following its evolution through the transition to turbulence. A low-Mach number hydrodynamics method is used, freeing us from the harsh time step restrictions imposed by sound waves. We fully resolve the thermal structure of the flame and its reaction zone, eliminating the need for a flame model. A single density is considered, 1.5 × 107 g cm-3, and half-carbon, half-oxygen fuel: conditions under which the flame propagated in the flamelet regime in our related two-dimensional study. We compare to a corresponding two-dimensional simulation and show that while fire polishing keeps the small features suppressed in two dimensions, turbulence wrinkles the flame on far smaller scales in the three-dimensional case, suggesting that the transition to the distributed burning regime occurs at higher densities in three dimensions. Detailed turbulence diagnostics are provided. We show that the turbulence follows a Kolmogorov spectrum and is highly anisotropic on the large scales, with a much larger integral scale in the direction of gravity. Furthermore, we demonstrate that it becomes more isotropic as it cascades down to small scales. On the basis of the turbulent statistics and the flame properties of our simulation, we compute the Gibson scale. We show the progress of the turbulent flame through a classic combustion regime diagram, indicating that the flame just enters the distributed burning regime near the end of our simulation.

Morphology of Rising Hydrodynamic and Magnetohydrodynamic Bubbles from Numerical Simulations

Recent Chandra and XMM-Newton observations of galaxy cluster cooling flows have revealed X-ray emission voids of up to 30 kpc in size that have been identified with buoyant, magnetized bubbles. Motivated by these observations, we have investigated the behavior of rising bubbles in stratified atmospheres using the FLASH adaptive-mesh simulation code. We present results from two-dimensional simulations with and without the effects of magnetic fields and with varying bubble sizes and background stratifications. We find purely hydrodynamic bubbles to be unstable; a dynamically important magnetic field is required to maintain a bubbles integrity. This suggests that, even absent thermal conduction, for bubbles to be persistent enough to be regularly observed, they must be supported in large part by magnetic fields. Thermal conduction unmitigated by magnetic fields can dissipate the bubbles even faster. We also observe that the bubbles leave a tail as they rise; the structure of these tails can indicate the history of the dynamics of the rising bubble.

A Case Study in Application I/O on Linux Clusters

A critical but often ignored component of system performance is the I/O system. Today’s applications demand a great deal from underlying storage systems and software, and both high-performance distributed storage and high level interfaces have been developed to fill these needs. In this paper we discuss the I/O performance of a parallel scientific application on a Linux cluster, the FLASH astrophysics code. This application relies on three I/O software components to provide high-performance parallel I/O on Linux clusters: the Parallel Virtual File System, the ROMIO MPI-IO implementation, and the Hierarchical Data Format library. Through instrumentation of both the application and underlying system software code we discover the location of major software bottlenecks. We work around the most inhibiting of these bottlenecks, showing substantial performance improvement. We point out similarities between the inefficiencies found here and those found in message passing systems, indicating that research in the message passing field could be leveraged to solve similar problems in high-level I/O interfaces.

MAESTRO: AN ADAPTIVE LOW MACH NUMBER HYDRODYNAMICS ALGORITHM FOR STELLAR FLOWS

Many astrophysical phenomena are highly subsonic, requiring specialized numerical methods suitable for long-time integration. In a series of earlier papers we described the development of MAESTRO, a low Mach number stellar hydrodynamics code that can be used to simulate long-time, low-speed flows that would be prohibitively expensive to model using traditional compressible codes. MAESTRO is based on an equation set derived using low Mach number asymptotics; this equation set does not explicitly track acoustic waves and thus allows a significant increase in the time step. MAESTRO is suitable for two- and three-dimensional local atmospheric flows as well as three-dimensional full-star flows. Here, we continue the development of MAESTRO by incorporating adaptive mesh refinement (AMR). The primary difference between MAESTRO and other structured grid AMR approaches for incompressible and low Mach number flows is the presence of the time-dependent base state, whose evolution is coupled to the evolution of the full solution. We also describe how to incorporate the expansion of the base state for full-star flows, which involves a novel mapping technique between the one-dimensional base state and the Cartesian grid, as well as a number of overall improvements to the algorithm. We examine the efficiency and accuracy of our adaptive code, and demonstrate that it is suitable for further study of our initial scientific application, the convective phase of Type Ia supernovae.

High-Performance Reactive Fluid Flow Simulations Using Adaptive Mesh Refinement on Thousands of Processors

We present simulations and performance results of nuclear burning fronts in super- novae on the largest domain and at the finest spatial resolution studied to date. These simulations were performed on the Intel ASCI-Red machine at Sandia National Laboratories using FLASH, a code developed at the Center for Astrophysical Thermonuclear Flashes at the University of Chicago. FLASH is a modular, adaptive mesh, parallel simulation code capable of handling compressible, reactive fluid flows in astrophysical environments. FLASH is written primarily in Fortran 90, uses the Message-Passing Interface library for inter-processor communication and portability, and employs the PARAMESH package to manage a block-structured adaptive mesh that places blocks only where resolution is required and tracks rapidly changing flow features, such as detonation fronts, with ease. We describe the key algorithms and their implementation as well as the optimizations required to achieve sustained performance of 238 GFLOPS on 6420 processors of ASCI-Red in 64 bit arithmetic.