Vincent E. Beckner

Parallelization of structured, hierarchical adaptive mesh refinement algorithms

Abstract.We describe an approach to parallelization of structured adaptive mesh refinement algorithms. This type of adaptive methodology is based on the use of local grids superimposed on a coarse grid to achieve sufficient resolution in the solution. The key elements of the approach to parallelization are a dynamic load-balancing technique to distribute work to processors and a software methodology for managing data distribution and communications. The methodology is based on a message-passing model that exploits the coarse-grained parallelism inherent in the algorithms. The approach is illustrated for an adaptive algorithm for hyperbolic systems of conservation laws in three space dimensions. A numerical example computing the interaction of a shock with a helium bubble is presented. We give timings to illustrate the performance of the method.

An adaptive multifluid interface-capturing method for compressible flow in complex geometries

We present a numerical method for solving the multifluid equations of gas dynamics using an operator-split second-order Godunov method for flow in complex geometries in two and three dimensions. The multifluid system treats the fluid components as thermodynamically distinct entities and correctly models fluids with different compressibilities. This treatment allows a general equation-of-state (EOS) specification and the method is implemented so that the EOS references are minimized. The current method is complementary to volume-of-fluid (VOF) methods in the sense that a VOF representation is used, but no interface reconstruction is performed. The Godunov integrator captures the interface during the solution process. The basic multifluid integrator is coupled to a Cartesian grid algorithm that also uses a VOF representation of the fluid-body interface. This representation of the fluid-body interface allows the algorithm to easily accommodate arbitrarily complex geometries. The resulting single grid multifluid-Cartesian grid integration scheme is coupled to a local adaptive mesh refinement algorithm that dynamically refines selected regions of the computational grid to achieve a desired level of accuracy. The overall method is fully conservative with respect to the total mixture. The method will be used for a simple nozzle problem in two-dimensional axisymmetric coordinates.

Interaction of turblence and chemistry in a low-swirl burner

New combustion systems based on ultra-lean premixed combustion have the potential for dramatically reducing pollutant emissions in transportation systems, heat, and stationary power generation. However, lean premixed flames are highly susceptible to fluid-dynamical combustion instabilities, making robust and reliable systems difficult to design. Low-swirl burners are emerging as an important technology for meeting design requirements in terms of both reliability and emissions for next-generation combustion devices. In this paper, we present simlations of a laboratory-scale low-swirl burner using detailed chemistry and transport without incorporating explicit models for turbulence or turbulence/chemistry interaction. We consider two fuels, methane and hydrogen, each at two turbulent intensities. Here we examine some of the basic properties of the flow field and the flame structure. We focus on the differences in flame behavior for the two fuels, particularly on the hydrogen flame, which burns with a cellular structures.

Topology-aware performance optimization and modeling of adaptive mesh refinement codes for exascale

We introduce a topology-aware performance optimization and modeling workflow for AMR simulation that includes two new modeling tools, ProgrAMR and Mota Mapper, which interface with the BoxLib AMR framework and the SSTmacro network simulator. ProgrAMR allows us to generate and model the execution of task dependency graphs from high-level specifications of AMR-based applications, which we demonstrate by analyzing two example AMR-based multigrid solvers with varying degrees of asynchrony. Mota Mapper generates multiobjective, network topology-aware box mappings, which we apply to optimize the data layout for the example multigrid solvers. While the sensitivity of these solvers to layout and execution strategy appears to be modest for balanced scenarios, the impact of better mapping algorithms can be significant when performance is highly constrained by network hop latency. Furthermore, we show that network latency in the multigrid bottom solve is the main contributing factor preventing good scaling on exascale-class machines.

In situ and in-transit analysis of cosmological simulations

Modern cosmological simulations have reached the trillion-element scale, rendering data storage and subsequent analysis formidable tasks. To address this circumstance, we present a new MPI-parallel approach for analysis of simulation data while the simulation runs, as an alternative to the traditional workflow consisting of periodically saving large data sets to disk for subsequent ‘offline’ analysis. We demonstrate this approach in the compressible gasdynamics/N-body code Nyx, a hybrid MPI+OpenMP

AMR Code Simulations of Turbulent Combustion in Confined and Unconfined SDF Explosions

\mbox{MPI}+\mbox{OpenMP}

A communications simulation methodology for AMR codes using task dependency analysis

code based on the BoxLib framework, used for large-scale cosmological simulations. We have enabled on-the-fly workflows in two different ways: one is a straightforward approach consisting of all MPI processes periodically halting the main simulation and analyzing each component of data that they own (‘in situ’). The other consists of partitioning processes into disjoint MPI groups, with one performing the simulation and periodically sending data to the other ‘sidecar’ group, which post-processes it while the simulation continues (‘in-transit’). The two groups execute their tasks asynchronously, stopping only to synchronize when a new set of simulation data needs to be analyzed. For both the in situ and in-transit approaches, we experiment with two different analysis suites with distinct performance behavior: one which finds dark matter halos in the simulation using merge trees to calculate the mass contained within iso-density contours, and another which calculates probability distribution functions and power spectra of various fields in the simulation. Both are common analysis tasks for cosmology, and both result in summary statistics significantly smaller than the original data set. We study the behavior of each type of analysis in each workflow in order to determine the optimal configuration for the different data analysis algorithms.

Parallelization of an Adaptive Mesh Refinement Method for Low Mach Number Combustion

A heterogeneous continuum model is proposed to describe the dispersion and combustion of an aluminum particle cloud in an explosion. It combines the gasdynamic conservation laws for the gas phase with a continuum model for the dispersed phase, as formulated by Nigmatulin. Inter-phase mass, momentum and energy exchange are prescribed by phenomenological models. It incorporates a combustion model based on the mass conservation laws for fuel, air and products; source/sink terms are treated in the fast-chemistry limit appropriate for such gasdynamic fields, along with a model for mass transfer from the particle phase to the gas. The model takes into account both the afterburning of the detonation products of the booster with air, and the combustion of the Al particles with air. The model equations were integrated by high-order Godunov schemes for both the gas and particle phases. Numerical simulations of the explosion fields from 1.5-g Shock-Dispersed-Fuel (SDF) charge in a 6.6 liter calorimeter were used to validate the combustion model. Then the model was applied to 10-kg Al-SDF explosions in a vented two-room structure and in an unconfined height-of-burst explosion. Computed pressure histories are in reasonable (but not perfect) agreement with measured waveforms. Differences are caused by physical-chemical kinetic effects of particle combustion which induce ignition delays in the initial reactive blast wave and quenching of reactions at late times. Current simulations give initial insights into such modeling issues.

Turbulent combustion in aluminum-air clouds for different scale explosion fields

The ability to predict the performance of irregular, asynchronous applications on future hardware is essential to the exascale co-design process. Adaptive Mesh Refinement (AMR) applications are inherently irregular and dynamic in their computation and communication patterns, resulting in complex hardware/software interactions. We have developed a methodology to use architectural simulators to assess the performance of different AMR data placement strategies on a selection of potential hardware interconnect topologies for exascale-class supercomputers. We use our framework to study the CASTRO AMR compressible astrophysics code for the simulation of supernovae. The results show a performance improvement of up to 18 percent may be obtained through the use of locality-aware data distributions for some network topologies on an exascale-class supercomputer.

Numerical Simulation of Shock-Dispersed Fuel Charges

We describe the parallelization of a computer program for the adaptive mesh refinement simulation of variable density, viscous, incompressible fluid flows for low Mach number combustion. The adaptive methodology is based on the use of local grids superimposed on a coarse grid to achieve sufficient resolution in the solution. The key elements of the approach to parallelization are a dynamic load-balancing technique to distribute work to processors and a software methodology for managing data distribution and communications. The methodology is based on a message-passing model that exploits the coarse-grained parallelism inherent in the algorithms. A method is presented for parallelizing weakly sequential loops--loops with sparse dependencies among iterations.