Glenn E. Hammond

Multiphysics simulations: Challenges and opportunities

We consider multiphysics applications from algorithmic and architectural perspectives, where “algorithmic” includes both mathematical analysis and computational complexity, and “architectural” includes both software and hardware environments. Many diverse multiphysics applications can be reduced, en route to their computational simulation, to a common algebraic coupling paradigm. Mathematical analysis of multiphysics coupling in this form is not always practical for realistic applications, but model problems representative of applications discussed herein can provide insight. A variety of software frameworks for multiphysics applications have been constructed and refined within disciplinary communities and executed on leading-edge computer systems. We examine several of these, expose some commonalities among them, and attempt to extrapolate best practices to future systems. From our study, we summarize challenges and forecast opportunities.

Simulating Subsurface Flow and Transport on Ultrascale Computers using PFLOTRAN

We describe PFLOTRAN, a recently developed code for modeling multi-phase, multi-component subsurface flow and reactive transport using massively parallel computers. PFLOTRAN is built on top of PETSc, the Portable, Extensible Toolkit for Scientific Computation. Leveraging PETSc has allowed us to develop—with a relatively modest investment in development effort—a code that exhibits excellent performance on the largest-scale supercomputers. Very significant enhancements to the code are planned during our SciDAC-2 project. Here we describe the current state of the code, present an example of its use on Jaguar, the Cray XT3/4 system at Oak Ridge National Laboratory consisting of 11706 dual-core Opteron processor nodes, and briefly outline our future plans for the code.

Stochastic simulation of uranium migration at the Hanford 300 Area.

This work focuses on the quantification of groundwater flow and subsequent U(VI) transport uncertainty due to heterogeneity in the sediment permeability at the Hanford 300 Area. U(VI) migration at the site is simulated with multiple realizations of stochastically-generated high resolution permeability fields and comparisons are made of cumulative water and U(VI) flux to the Columbia River. The massively parallel reactive flow and transport code PFLOTRAN is employed utilizing 40,960 processor cores on DOEs petascale Jaguar supercomputer to simultaneously execute 10 transient, variably-saturated groundwater flow and U(VI) transport simulations within 3D heterogeneous permeability fields using the codes multi-realization simulation capability. Simulation results demonstrate that the cumulative U(VI) flux to the Columbia River is less responsive to fine scale heterogeneity in permeability and more sensitive to the distribution of permeability within the river hyporheic zone and mean permeability of larger-scale geologic structures at the site.

Subsurface multiphase flow and multicomponent reactive transport modeling using high-performance computing

Numerical modeling is a critical tool to the U.S. Department of Energy for evaluating the environmental impact of remediation strategies for subsurface legacy waste sites. Unfortunately, the physical and chemical complexity of many sites overwhelms the capabilities of even most state of the art groundwater models. Of particular concern is the representation of highly-heterogeneous stratified rock/soil layers in the subsurface and the biological and geochemical interactions of chemical species within multiple fluid phases. There is clearly a need for higher-resolution modeling (i.e. increased spatial and temporal resolution) and increasingly mechanistic descriptions of subsurface physicochemical processes (i.e. increased chemical degrees of freedom). We present SciDAC-funded research being performed in furthering the development of PFLOTRAN, a parallel multiphase flow and multicomponent reactive transport model. Written in Fortran90, PFLOTRAN is founded upon PETSc data structures and solvers. We are employing PFLOTRAN to simulate uranium transport at the Hanford 300 Area, a contaminated site of major concern to the Department of Energy, the State of Washington, and other government agencies. By leveraging the billions of degrees of freedom available through high-performance computation using tens of thousands of processors, we can better characterize the release of uranium into groundwater and its subsequent transport to the Columbia River, and thereby better understand and evaluate the effectiveness of various proposed remediation strategies.

Toward petascale computing in geosciences: application to the Hanford 300 area

Modeling uranium transport at the Hanford 300 Area presents new challenges for high performance computing. A field-scale three-dimensional domain with an hourly fluctuating Columbia river stage coupled to flow in highly permeable sediments results in fast groundwater flow rates requiring small time steps. In this work, high-performance computing has been applied to simulate variably saturated groundwater flow and tracer transport at the 300 Area using PFLOTRAN. Simulation results are presented for discretizations up to 10.8 million degrees of freedom, while PFLOTRAN performance was assessed on up to one billion degrees of freedom and 12,000 processor cores on Jaguar, the Cray XT4 supercomputer at ORNL.

Modeling subsurface reactive flows using leadership-class computing

We describe our experiences running PFLOTRAN–a code for simulation of coupled hydro-thermal-chemical processes in variably saturated, non-isothermal, porous media– on leadership-class supercomputers, including initial experiences running on the petaflop incarnation of Jaguar, the Cray XT5 at the National Center for Computational Sciences at Oak Ridge National Laboratory. PFLOTRAN utilizes fully implicit time-stepping and is built on top of the Portable, Extensible Toolkit for Scientific Computation (PETSc). We discuss some of the hurdles to at scale performance with PFLOTRAN and the progress we have made in overcoming them on leadership-class computer architectures.

Application of ensemble‐based data assimilation techniques for aquifer characterization using tracer data at Hanford 300 area

Subsurface aquifer characterization often involves high parameter dimensionality and requires tremendous computational resources if employing a full Bayesian approach. Ensemble-based data assimilation techniques, including filtering and smoothing, are computationally efficient alternatives. Despite the increasing number of applications of ensemble-based methods in assimilating flow and transport related data for subsurface aquifer charaterization, most are limited to either synthetic studies or two-dimensional problems. In this study, we applied ensemble-based techniques for assimilating field tracer experimental data obtained from the Integrated Field Research Challenge (IFRC) site at the Hanford 300 Area. The forward problem was simulated using the massively-parallel three-dimensional flow and transport code PFLOTRAN to effectively deal with the highly transient flow boundary conditions at the site and to meet the computational demands of ensemble-based methods. This study demonstrates the effectiveness of ensemble-based methods for characterizing a heterogeneous aquifer by sequentially assimilating multiple types of data. The necessity of employing high performance computing is shown to enable increasingly mechanistic non-linear forward simulations to be performed within the data assimilation framework for a complex system with reasonable turnaround time.

High performance simulation of environmental tracers in heterogeneous domains.

In this study, we use PFLOTRAN, a highly scalable, parallel, flow, and reactive transport code to simulate the concentrations of 3H, 3He, CFC-11, CFC-12, CFC-113, SF6, 39Ar, and the mean groundwater age in heterogeneous fields on grids with an excess of 10 million nodes. We utilize this computational platform to simulate the concentration of multiple tracers in high-resolution, heterogeneous 2D and 3D domains, and calculate tracer-derived ages. Tracer-derived ages show systematic biases toward younger ages when the groundwater age distribution contains water older than the maximum tracer age. The deviation of the tracer-derived age distribution from the true groundwater age distribution increases with increasing heterogeneity of the system. However, the effect of heterogeneity is diminished as the mean travel time gets closer to the tracer age limit. Age distributions in 3D domains differ significantly from 2D domains. 3D simulations show decreased mean age, and less variance in age distribution for identical heterogeneity statistics. High-performance computing allows for investigation of tracer and groundwater age systematics in high-resolution domains, providing a platform for understanding and utilizing environmental tracer and groundwater age information in heterogeneous 3D systems.

Modeling multicomponent reactive transport on parallel computers using Jacobian-Free Newton Krylov with operator-split preconditioning

Modern multicomponent reactive transport models necessitate the use of parallel computation due to extreme memory and processing demands. Innovative parallel algorithms are needed to fully exploit the advanced computational power provided by todays supercomputers within these models. In order to take advantage of available high-performance computing, we have developed the parallel reactive transport model PARTRAN. This three-dimensional, finite volume model is built upon the versatile PETSc library developed at Argonne National Laboratory. PARTRAN is capable of using either conventional techniques (Jacobian-based) or the Jacobianfree, Newton-Krylov method to solve the fully coupled, nonlinear system of equations representing complex geochemical transport processes in the subsurface. The use of Newton-Krylov eliminates the necessity of computing and storing a Jacobian, resulting in considerably less computation and memory use. While one disadvantage of Jacobian-free Newton-Krylov is the need for efficient matrix-free preconditioning, we develop an innovative operator-split preconditioning method to address this shortfall. In this paper, we provide a brief overview of PARTRAN and its implementation of the Jacobianfree, Newton Krylov technique. We evaluate the parallel performance of the model in detail, comparing the performance of the Newton-Krylov method with conventional Jacobian-based and operator-split methods. The Jacobian-free, Newton-Krylov method demonstrates excellent parallel performance based on runtime, scalability and efficiency, in comparison to the conventional methods.

A Hybrid Multiscale Framework for Subsurface Flow and Transport Simulations

Abstract Extensive research is aimed at improving predictive ability of biogeochemical earth and environmental system simulators, with applications ranging from contaminant transport and remediation to impacts of carbon and nitrogen cycling on local ecosystems and climate. Most process-based numerical models are designed for a single characteristic length and time scale. For application-relevant scales, it is necessary to introduce approximations and empirical parameterizations to describe complex systems because of limitations on process understanding, system characterization and computation. Using emerging understanding of biological and environmental processes at fundamental scales to advance predictions of the larger system behavior requires the development of multiscale simulators, and there is strong interest in coupling microscale and macroscale models together in a hybrid multiscale simulation. A limited number of hybrid multiscale simulations have been developed for biogeochemical systems, mostly using application-specific approaches for model coupling. We are developing a generalized approach to hierarchical model coupling designed for high-performance computational systems, based on the Swift computing workflow framework. In this presentation we will describe the generalized approach and provide two use cases: 1) simulation of a mixing-controlled biogeochemical reaction coupling pore- and continuum-scale models, and 2) simulation of biogeochemical impacts of groundwater–river water interactions coupling fine- and coarse-grid model representations. This generalized framework can be customized for use with any pair of linked models (microscale and macroscale) with minimal intrusiveness to the at-scale simulators. It combines a set of python scripts with the Swift workflow environment to execute a complex multiscale simulation utilizing an approach similar to the well-known Heterogeneous Multiscale Method. User customization is facilitated through user-provided input and output file templates and processing function scripts, and execution within a high-performance computing environment is handled by Swift, such that minimal to no user modification of at-scale codes is required.