Andrew R. Siegel

Multiphysics simulations: Challenges and opportunities

We consider multiphysics applications from algorithmic and architectural perspectives, where “algorithmic” includes both mathematical analysis and computational complexity, and “architectural” includes both software and hardware environments. Many diverse multiphysics applications can be reduced, en route to their computational simulation, to a common algebraic coupling paradigm. Mathematical analysis of multiphysics coupling in this form is not always practical for realistic applications, but model problems representative of applications discussed herein can provide insight. A variety of software frameworks for multiphysics applications have been constructed and refined within disciplinary communities and executed on leading-edge computer systems. We examine several of these, expose some commonalities among them, and attempt to extrapolate best practices to future systems. From our study, we summarize challenges and forecast opportunities.

Extensible component-based architecture for FLASH, a massively parallel, multiphysics simulation code

FLASH is a publicly available high performance application code which has evolved into a modular, extensible software system from a collection of unconnected legacy codes. FLASH has been successful because its capabilities have been driven by the needs of scientific applications, without compromising maintainability, performance, and usability. In its newest incarnation, FLASH3 consists of inter-operable modules that can be combined to generate different applications. The FLASH architecture allows arbitrarily many alternative implementations of its components to co-exist and interchange with each other, resulting in greater flexibility. Further, a simple and elegant mechanism exists for customization of code functionality without the need to modify the core implementation of the source. A built-in unit test framework providing verifiability, combined with a rigorous software maintenance process, allow the code to operate simultaneously in the dual mode of production and development. In this paper we describe the FLASH3 architecture, with emphasis on solutions to the more challenging conflicts arising from solver complexity, portable performance requirements, and legacy codes. We also include results from user surveys conducted in 2005 and 2007, which highlight the success of the code.

Morphology of Rising Hydrodynamic and Magnetohydrodynamic Bubbles from Numerical Simulations

Recent Chandra and XMM-Newton observations of galaxy cluster cooling flows have revealed X-ray emission voids of up to 30 kpc in size that have been identified with buoyant, magnetized bubbles. Motivated by these observations, we have investigated the behavior of rising bubbles in stratified atmospheres using the FLASH adaptive-mesh simulation code. We present results from two-dimensional simulations with and without the effects of magnetic fields and with varying bubble sizes and background stratifications. We find purely hydrodynamic bubbles to be unstable; a dynamically important magnetic field is required to maintain a bubbles integrity. This suggests that, even absent thermal conduction, for bubbles to be persistent enough to be regularly observed, they must be supported in large part by magnetic fields. Thermal conduction unmitigated by magnetic fields can dissipate the bubbles even faster. We also observe that the bubbles leave a tail as they rise; the structure of these tails can indicate the history of the dynamics of the rising bubble.

Petascale algorithms for reactor hydrodynamics

We describe recent algorithmic developments that have enabled large eddy simulations of reactor flows on up to P = 65, 000 processors on the IBM BG/P at the Argonne Leadership Computing Facility.

Crushing of interstellar gas clouds in supernova remnants - I. The role of thermal conduction and radiative losses

We model the hydrodynamic interaction of a shock wave of an evolved supernova remnant with a small interstellar gas cloud like the ones observed in the Cygnus loop and in the Vela SNR. We investigate the interplay between radiative cooling and thermal conduction during cloud evolution and their effect on the mass and energy exchange between the cloud and the surrounding medium. Through the study of two cases characterized by different Mach numbers of the primary shock (M = 30 and 50, corresponding to a post-shock temperature T 1.7 x 10 6 K and 4.7 x 10 6 K, respectively), we explore two very different physical regimes: for M = 30. the radiative losses dominate the evolution of the shocked cloud which fragments into cold, dense, and compact filaments surrounded by a hot corona which is ablated by the thermal conduction; instead, for M = 50, the thermal conduction dominates the evolution of the shocked cloud. which evaporates in a few dynamical time-scales. In both cases we find that the thermal conduction is very effective in suppressing the hydrodynamic instabilities that would develop at the cloud boundaries.

Mapping Initial Hydrostatic Models in Godunov Codes

We look in detail at the process of mapping an astrophysical initial model from a stellar evolution code onto the computational grid of an explicit, Godunov-type code while maintaining hydrostatic equilibrium. This mapping process is common in astrophysical simulations, when it is necessary to follow short-timescale dynamics after a period of long-timescale buildup. We look at the effects of spatial resolution, boundary conditions, the treatment of the gravitational source terms in the hydrodynamics solver, and the initialization process itself. We conclude with a summary detailing the mapping process that yields the lowest ambient velocities in the mapped model.

Data decomposition of Monte Carlo particle transport simulations via tally servers

United States. Dept. of Energy. Naval Reactors Division. Rickover Fellowship Program in Nuclear Engineering

The Response of Model and Astrophysical Thermonuclear Flames to Curvature and Stretch

Critically understanding the standard candle-like behavior of Type Ia supernovae requires understanding their explosion mechanism. One family of models for Type Ia supernovae begins with a deflagration in a carbon-oxygen white dwarf that greatly accelerates through wrinkling and flame instabilities. While the planar speed and behavior of astrophysically relevant flames is increasingly well understood, more complex behavior, such as the flames response to stretch and curvature, has not been extensively explored in the astrophysical literature; this behavior can greatly enhance or suppress instabilities and local flame-wrinkling, which in turn can increase or decrease the bulk burning rate. In this paper, we explore the effects of curvature on both nuclear flames and simpler model flames to understand the effect of curvature on the flame structure and speed.

Analysis of communication costs for domain decomposed Monte Carlo methods in nuclear reactor analysis

A domain decomposed Monte Carlo communication kernel is used to carry out performance tests to establish the feasibility of using Monte Carlo techniques for practical Light Water Reactor (LWR) core analyses. The results of the prototype code are interpreted in the context of simplified performance models which elucidate key scaling regimes of the parallel algorithm.

Madre: the Memory-Aware Data Redistribution Engine

We report on the development of a new computational framework for efficiently carrying out parallel data redistribution in a limited memory environment. This new library, MADRE (Memory-Aware Data Redistribution Engine), is an open-source, C/MPI-based toolkit designed for quick and easy integration into application codes that have demanding data migration needs. At the same time, MADRE exposes a lower-level application programming interface that greatly facilitates the development and incorporation of new algorithms into the MADRE framework, thus serving as a potential organizing entity for continued research in this area. Finally, we develop, describe, and test in detail several new parallel redistribution algorithms that are incorporated into the MADRE distribution.