Henry M. Tufo

FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes

We report on the completion of the first version of a new-generation simulation code, FLASH. The FLASH code solves the fully compressible, reactive hydrodynamic equations and allows for the use of adaptive mesh refinement. It also contains state-of-the-art modules for the equations of state and thermonuclear reaction networks. The FLASH code was developed to study the problems of nuclear flashes on the surfaces of neutron stars and white dwarfs, as well as in the interior of white dwarfs. We expect, however, that the FLASH code will be useful for solving a wide variety of other problems. This first version of the code has been subjected to a large variety of test cases and is currently being used for production simulations of X-ray bursts, Rayleigh-Taylor and Richtmyer-Meshkov instabilities, and thermonuclear flame fronts. The FLASH code is portable and already runs on a wide variety of massively parallel machines, including some of the largest machines now extant.

A comparative study of the turbulent Rayleigh–Taylor instability using high-resolution three-dimensional numerical simulations: The Alpha-Group collaboration

The turbulent Rayleigh–Taylor instability is investigated in the limit of strong mode-coupling using a variety of high-resolution, multimode, three dimensional numerical simulations (NS). The perturbations are initialized with only short wavelength modes so that the self-similar evolution (i.e., bubble diameter Db∝amplitude hb) occurs solely by the nonlinear coupling (merger) of saturated modes. After an initial transient, it is found that hb∼αbAgt2, where A=Atwood number, g=acceleration, and t=time. The NS yield Db∼hb/3 in agreement with experiment but the simulation value αb∼0.025±0.003 is smaller than the experimental value αb∼0.057±0.008. By analyzing the dominant bubbles, it is found that the small value of αb can be attributed to a density dilution due to fine-scale mixing in our NS without interface reconstruction (IR) or an equivalent entrainment in our NS with IR. This may be characteristic of the mode coupling limit studied here and the associated αb may represent a lower bound that is insensiti...

On validating an astrophysical simulation code

We present a case study of validating an astrophysical simulation code. Our study focuses on validating FLASH, a parallel, adaptive-mesh hydrodynamics code for studying the compressible, reactive flows found in many astrophysical environments. We describe the astrophysics problems of interest and the challenges associated with simulating these problems. We describe methodology and discuss solutions to difficulties encountered in verification and validation. We describe verification tests regularly administered to the code, present the results of new verification tests, and outline a method for testing general equations of state. We present the results of two validation tests in which we compared simulations to experimental data. The first is of a laser-driven shock propagating through a multilayer target, a configuration subject to both Rayleigh-Taylor and Richtmyer-Meshkov instabilities. The second test is a classic Rayleigh-Taylor instability, where a heavy fluid is supported against the force of gravity by a light fluid. Our simulations of the multilayer target experiments showed good agreement with the experimental results, but our simulations of the Rayleigh-Taylor instability did not agree well with the experimental results. We discuss our findings and present results of additional simulations undertaken to further investigate the Rayleigh-Taylor instability.

Phylogenetic stratigraphy in the Guerrero Negro hypersaline microbial mat

The microbial mats of Guerrero Negro (GN), Baja California Sur, Mexico historically were considered a simple environment, dominated by cyanobacteria and sulfate-reducing bacteria. Culture-independent rRNA community profiling instead revealed these microbial mats as among the most phylogenetically diverse environments known. A preliminary molecular survey of the GN mat based on only ∼1500 small subunit rRNA gene sequences discovered several new phylum-level groups in the bacterial phylogenetic domain and many previously undetected lower-level taxa. We determined an additional ∼119 000 nearly full-length sequences and 28 000 >200 nucleotide 454 reads from a 10-layer depth profile of the GN mat. With this unprecedented coverage of long sequences from one environment, we confirm the mat is phylogenetically stratified, presumably corresponding to light and geochemical gradients throughout the depth of the mat. Previous shotgun metagenomic data from the same depth profile show the same stratified pattern and suggest that metagenome properties may be predictable from rRNA gene sequences. We verify previously identified novel lineages and identify new phylogenetic diversity at lower taxonomic levels, for example, thousands of operational taxonomic units at the family-genus levels differ considerably from known sequences. The new sequences populate parts of the bacterial phylogenetic tree that previously were poorly described, but indicate that any comprehensive survey of GN diversity has only begun. Finally, we show that taxonomic conclusions are generally congruent between Sanger and 454 sequencing technologies, with the taxonomic resolution achieved dependent on the abundance of reference sequences in the relevant region of the rRNA tree of life.

On the miscible Rayleigh{Taylor instability: two and three dimensions

We investigate the miscible Rayleigh{Taylor (RT) instability in both two and three dimensions using direct numerical simulations, where the working fluid is assumed incompressible under the Boussinesq approximation. We rst consider the case of randomly perturbed interfaces. With a variety of diagnostics, we develop a physical picture for the detailed temporal development of the mixed layer: we identify three distinct evolutionary phases in this development, which can be related to detailed variations in the growth of the mixing zone. Our analysis provides an explanation for the observed dierences between two- and three-dimensional RT instability; the analysis also leads us to concentrate on the RT models which (i) work equally well for both laminar and turbulent flows, and (ii) do not depend on turbulent scaling within the mixing layer between fluids. These candidate RT models are based on point sources within bubbles (or plumes) and their interaction with each other (or the background flow). With this motivation, we examine the evolution of single plumes, and relate our numerical results (for single plumes) to a simple analytical model for plume evolution.

A Comparison of Two Shallow-Water Models with Nonconforming Adaptive Grids

Abstract In an effort to study the applicability of adaptive mesh refinement (AMR) techniques to atmospheric models, an interpolation-based spectral element shallow-water model on a cubed-sphere grid is compared to a block-structured finite-volume method in latitude–longitude geometry. Both models utilize a nonconforming adaptation approach that doubles the resolution at fine–coarse mesh interfaces. The underlying AMR libraries are quad-tree based and ensure that neighboring regions can only differ by one refinement level. The models are compared via selected test cases from a standard test suite for the shallow-water equations, and via a barotropic instability test. These tests comprise the passive advection of a cosine bell and slotted cylinder, a steady-state geostrophic flow, a flow over an idealized mountain, a Rossby–Haurwitz wave, and the evolution of a growing barotropic wave. Both static and dynamics adaptations are evaluated, which reveal the strengths and weaknesses of the AMR techniques. Overa...

Fast Parallel Direct Solvers for Coarse Grid Problems

We have developed a fast direct solver for parallel solution of coarse grid problems, Ax=b, such as arise when domain decomposition or multigrid methods are applied to elliptic partial differential equations in d space dimensions. The approach is based on a (quasi-) sparse factorization of the inverse of A. If A is n×n and the number of processors is P, the algorithm requires O(n?logP) time for communication and O(n1+?/P) time for computation, where ??d?1d. The method is particularly suited to leading-edge multicomputer systems having thousands of processors. It achieves minimal message startup costs and substantially reduced message volume and arithmetic complexity compared with competing methods, which require O(nlogP) time for communication and O(n1+?) or O(n2/P) time for computation. Timings on the Intel Paragon and ASCI-Red machines reflect these complexity estimates.

High-Resolution Mesh Convergence Properties and Parallel Efficiency of a Spectral Element Atmospheric Dynamical Core

We first demonstrate the parallel performance of the dynamical core of a spectral element atmospheric model. The model uses continuous Galerkin spectral elements to discretize the surface of the Earth, coupled with finite differences in the radial direction. Results are presented from two distributed memory, mesh interconnect supercomputers (ASCI Red and BlueGene/L), using a two-dimensional space filling curve domain decomposition. Better than 80% parallel efficiency is obtained for fixed grids on up to 8938 processors. These runs represent the largest processor counts ever achieved for a geophysical application. They show that the upcoming Red Storm and BlueGene/L super-computers are well suited for performing global atmospheric simulations with a 10 km average grid spacing. We then demonstrate the accuracy of the method by performing a full three-dimensional mesh refinement convergence study, using the primitive equations to model breaking Rossby waves on the polar vortex. Due to the excellent parallel performance, the model is run at several resolutions up to 36 km with 200 levels using only modest computing resources. Isosurfaces of scaled potential vorticity exhibit complex dynamical features, e.g. a primary potential vorticity tongue, and a secondary instability causing roll-up into a ring of five smaller subvortices. As the resolution is increased, these features are shown to converge while potential vorticity gradients steepen.

An overlapping Schwarz method for spectral element simulation of three-dimensional incompressible flows.

As the sound speed is infinite for incompressible flows, computation of the pressure constitutes the stiffest component in the time advancement of unsteady simulations. For complex geometries, efficient solution is dependent upon the availability of fast solvers for sparse linear systems. In this paper we develop a Schwarz preconditioner for the spectral element method using overlapping subdomains for the pressure. These local subdomain problems are derived from tensor products of one-dimensional finite element discretizations and admit use of fast diagonalization methods based upon matrix-matrix products. In addition, we use a coarse grid projection operator whose solution is computed via a fast parallel direct solver. The combination of overlapping Schwarz preconditioning and fast coarse grid solver provides as much as a fourfold reduction in simulation time over previously employed methods based upon deflation for parallel solution of multi-million grid point flow problems.

High-Performance Reactive Fluid Flow Simulations Using Adaptive Mesh Refinement on Thousands of Processors

We present simulations and performance results of nuclear burning fronts in super- novae on the largest domain and at the finest spatial resolution studied to date. These simulations were performed on the Intel ASCI-Red machine at Sandia National Laboratories using FLASH, a code developed at the Center for Astrophysical Thermonuclear Flashes at the University of Chicago. FLASH is a modular, adaptive mesh, parallel simulation code capable of handling compressible, reactive fluid flows in astrophysical environments. FLASH is written primarily in Fortran 90, uses the Message-Passing Interface library for inter-processor communication and portability, and employs the PARAMESH package to manage a block-structured adaptive mesh that places blocks only where resolution is required and tracks rapidly changing flow features, such as detonation fronts, with ease. We describe the key algorithms and their implementation as well as the optimizations required to achieve sustained performance of 238 GFLOPS on 6420 processors of ASCI-Red in 64 bit arithmetic.