Charles A. Rendleman

Parallelization of structured, hierarchical adaptive mesh refinement algorithms

Abstract.We describe an approach to parallelization of structured adaptive mesh refinement algorithms. This type of adaptive methodology is based on the use of local grids superimposed on a coarse grid to achieve sufficient resolution in the solution. The key elements of the approach to parallelization are a dynamic load-balancing technique to distribute work to processors and a software methodology for managing data distribution and communications. The methodology is based on a message-passing model that exploits the coarse-grained parallelism inherent in the algorithms. The approach is illustrated for an adaptive algorithm for hyperbolic systems of conservation laws in three space dimensions. A numerical example computing the interaction of a shock with a helium bubble is presented. We give timings to illustrate the performance of the method.

LOW MACH NUMBER MODELING OF TYPE Ia SUPERNOVAE. I. HYDRODYNAMICS

We introduce a low Mach number equation set for the large-scale numerical simulation of carbon-oxygen white dwarfs experiencing a thermonuclear deflagration. Since most of the interesting physics in a Type Ia supernova transpires at Mach numbers from 0.01 to 0.1, such an approach enables both a considerable increase in accuracy and a savings in computer time compared with frequently used compressible codes. Our equation set is derived from the fully compressible equations using low Mach number asymptotics, but without any restriction on the size of perturbations in density or temperature. Comparisons with simulations that use the fully compressible equations validate the low Mach number model in regimes where both are applicable. Comparisons to simulations based on the more traditional anelastic approximation also demonstrate the agreement of these models in the regime for which the anelastic approximation is valid. For low Mach number flows with potentially finite amplitude variations in density and temperature, the low Mach number model overcomes the limitations of each of the more traditional models and can serve as the basis for an accurate and efficient simulation tool.

Three-dimensional Numerical Simulations of Rayleigh-Taylor Unstable Flames in Type Ia Supernovae

Flame instabilities play a dominant role in accelerating the burning front to a large fraction of the speed of sound in a Type Ia supernova. We present a three-dimensional numerical simulation of a Rayleigh-Taylor unstable carbon flame, following its evolution through the transition to turbulence. A low-Mach number hydrodynamics method is used, freeing us from the harsh time step restrictions imposed by sound waves. We fully resolve the thermal structure of the flame and its reaction zone, eliminating the need for a flame model. A single density is considered, 1.5 × 107 g cm-3, and half-carbon, half-oxygen fuel: conditions under which the flame propagated in the flamelet regime in our related two-dimensional study. We compare to a corresponding two-dimensional simulation and show that while fire polishing keeps the small features suppressed in two dimensions, turbulence wrinkles the flame on far smaller scales in the three-dimensional case, suggesting that the transition to the distributed burning regime occurs at higher densities in three dimensions. Detailed turbulence diagnostics are provided. We show that the turbulence follows a Kolmogorov spectrum and is highly anisotropic on the large scales, with a much larger integral scale in the direction of gravity. Furthermore, we demonstrate that it becomes more isotropic as it cascades down to small scales. On the basis of the turbulent statistics and the flame properties of our simulation, we compute the Gibson scale. We show the progress of the turbulent flame through a classic combustion regime diagram, indicating that the flame just enters the distributed burning regime near the end of our simulation.

Direct numerical simulations of type Ia supernovae flames II: The Rayleigh-Taylor instability

A Type Ia supernova explosion likely begins as a nuclear runaway near the center of a carbon-oxygen white dwarf. The outward propagating flame is unstable to the Landau-Darrieus, Rayleigh-Taylor, and Kelvin-Helmholtz instabilities, which serve to accelerate it to a large fraction of the speed of sound. We investigate the Rayleigh-Taylor unstable flame at the transition from the flamelet regime to the distributed-burning regime, around densities of 10e7 gm/cc, through detailed, fully resolved simulations. A low Mach number, adaptive mesh hydrodynamics code is used to achieve the necessary resolution and long time scales. As the density is varied, we see a fundamental change in the character of the burning--at the low end of the density range the Rayleigh-Taylor instability dominates the burning, whereas at the high end the burning suppresses the instability. In all cases, significant acceleration of the flame is observed, limited only by the size of the domain we are able to study. We discuss the implications of these results on the potential for a deflagration to detonation transition.

Direct numerical simulations of type Ia supernovae flames I: The landau-darrieus instability

Planar flames are intrinsically unstable in open domains due to the thermal expansion across the burning front--the Landau-Darrieus instability. This instability leads to wrinkling and growth of the flame surface, and corresponding acceleration of the flame, until it is stabilized by cusp formation. We look at the Landau-Darrieus in stability for C/O thermonuclear flames at conditions relevant to the late stages of a Type Ia supernova explosion. Two-dimensional direct numerical simulations of both single-mode and multi-mode perturbations using a low Mach number hydrodynamics code are presented. We show the effect of the instability on the flame speed as a function of both the density and domain size, demonstrate the existence of the small scale cutoff to the growth of the instability, and look for the proposed breakdown of the non-linear stabilization at low densities. The effects of curvature on the flame as quantified through measurements of the growth rate and computation of the corresponding Markstein number. While accelerations of a few percent are observed, they are too small to have any direct outcome on the supernova explosion.

Embedded Boundary Algorithms for Solving the Poisson Equation on Complex Domains

Author(s): Day, Marcus S.; Colella, Phillip; Lijewski, Michael J.; Rendleman, Charles A.; Marcus, Daniel L.

Small scale processes and entrainment in a stratocumulus marine boundary layer

Numerical studies of boundary layer meteorology are increasingly reliant on large eddy simulation (LES) models, but few detailed validation studies of these types of models have been done. In this paper the authors investigate the behavior of an LES model for simulation of a marine boundary layer. Specifically, the authors focus on the mechanisms that control numerical predictions of entrainment into the tops of marine stratus in a moist generalization of the 1995 Global Energy and Water Cycle Experiment Cloud System Studies model intercomparison. For the computational study the authors present a sequence of simulations of varying resolution, from a typical resolution (50 m horizontal and 25 m vertical mesh size) to a fine resolution (8 m horizontal and 4 m vertical mesh size). The authors also explore variations in the model such as different subgrid models and modifications of the advection scheme. It was found that the thickness of the inversion, the depth of entraining eddies, and the shape of vertical velocity spectra were determined mainly by the mesh spacing used. However, the entrainment rate was found to have a distinct dependence on the amount of combined numerical and subgrid-scale mixing. This indicates that the use of large eddy simulation to study mixing in stratocumulus boundary layers needs to account for both sources of mixing.

The physics of flames in Type Ia supernovae

We extend a low Mach number hydrodynamics method developed for terrestrial combustion, to the study of thermonuclear flames in Type Ia supernovae. We discuss the differences between 2-D and 3-D Rayleigh-Taylor unstable flame simulations, and give detailed diagnostics on the turbulence, showing that the kinetic energy power spectrum obeys Bolgiano-Obukhov statistics in 2-D, but Kolmogorov statistics in 3-D. Preliminary results from 3-D reacting bubble calculations are shown, and their implications for ignition are discussed.

BlueGene/L applications: Parallelism On a Massive Scale

BlueGene/L (BG/L), developed through a partnership between IBM and Lawrence Livermore National Laboratory (LLNL), is currently the worlds largest system both in terms of scale, with 131,072 processors, and absolute performance, with a peak rate of 367 Tflop/s. BG/L has led the last four Top500 lists with a Linpack rate of 280.6 Tflop/s for the full machine installed at LLNL and is expected to remain the fastest computer in the next few editions. However, the real value of a machine such as BG/L derives from the scientific breakthroughs that real applications can produce by successfully using its unprecedented scale and computational power. In this paper, we describe our experiences with eight large scale applications on BG/ L from several application domains, ranging from molecular dynamics to dislocation dynamics and turbulence simulations to searches in semantic graphs. We also discuss the challenges we faced when scaling these codes and present several successful optimization techniques. All applications show excellent scaling behavior, even at very large processor counts, with one code even achieving a sustained performance of more than 100 Tflop/s, clearly demonstrating the real success of the BG/L design.

Scaling physics and material science applications on a massively parallel Blue Gene/L system

Blue Gene/L represents a new way to build supercomputers, using a large number of low power processors, together with multiple integrated interconnection networks. Whether real applications can scale to tens of thousands of processors (on a machine like Blue Gene/L) has been an open question. In this paper, we describe early experience with several physics and material science applications on a 32,768 node Blue Gene/L system, which was installed recently at the Lawrence Livermore National Laboratory. Our study shows some problems in the applications and in the current software implementation, but overall, excellent scaling of these applications to 32K nodes on the current Blue Gene/L system. While there is clearly room for improvement, these results represent the first proof point that MPI applications can effectively scale to over ten thousand processors. They also validate the scalability of the hardware and software architecture of Blue Gene/L.