James C. Sutherland

Improved boundary conditions for viscous, reacting, compressible flows

Previous studies on physical boundary conditions for flame-boundary interactions of an ideal, multicomponent, compressible gas have neglected reactive source terms in their boundary condition treatments. By combining analyses of incompletely parabolic systems with those based on the hyperbolic Euler equations, a rational set of boundary conditions is determined to address this shortcoming. Accompanying these conditions is a procedure for implementation into a maltidimensional code. In the limits of zero reaction rate or one species, the boundary conditions reduce in a predictable way to cases found in the literature. Application is made to premixed and nonpremixed flames in one and two dimensions to establish efficacy. Inclusion of source terms in boundary conditions derived from characteristic analysis is essential to avoid unphysical generation of pressure and velocity gradients as well as flow reversals. Minor deficiencies in the boundary conditions are attributed primarily to the diffusive terms. Imposing vanishing diffusive boundary-normal flux gradients works better than imposing vanishing fluxes but neither is entirely satisfactory.

Graph-Based Software Design for Managing Complexity and Enabling Concurrency in Multiphysics PDE Software

Multiphysics simulation software is plagued by complexity stemming from nonlinearly coupled systems of Partial Differential Equations (PDEs). Such software typically supports many models, which may require different transport equations, constitutive laws, and equations of state. Strong coupling and a multiplicity of models leads to complex algorithms (i.e., the properly ordered sequence of steps to assemble a discretized set of coupled PDEs) and rigid software. This work presents a design strategy that shifts focus away from high-level algorithmic concerns to low-level data dependencies. Mathematical expressions are represented as software objects that directly expose data dependencies. The entire system of expressions forms a directed acyclic graph and the high-level assembly algorithm is generated automatically through standard graph algorithms. This approach makes problems with complex dependencies entirely tractable, and removes virtually all logic from the algorithm itself. Changes are highly localized, allowing developers to implement models without detailed understanding of any algorithms (i.e., the overall assembly process). Furthermore, this approach complements existing MPI-based frameworks and can be implemented within them easily. Finally, this approach enables algorithmic parallelization via threads. By exposing dependencies in the algorithm explicitly, thread-based parallelism is implemented through algorithm decomposition, providing a basis for exploiting parallelism independent from domain decomposition approaches.

Large Scale Parallel Solution of Incompressible Flow Problems Using Uintah and Hypre

The Uintah Software framework was developed to provide an environment for solving fluid-structure interaction problems on structured adaptive grids on large-scale, long-running, data-intensive problems. Uintah uses a combination of fluid-flow solvers and particle-based methods for solids together with a novel asynchronous task-based approach with fully automated load balancing. As Uintah is often used to solve incompressible flow problems in combustion applications it is important to have a scalable linear solver. While there are many such solvers available, the scalability of those codes varies greatly. The hypre software offers a range of solvers and pre-conditioners for different types of grids. The weak scalability of Uintah and hypre is addressed for particular examples of both packages when applied to a number of incompressible flow problems. After careful software engineering to reduce startup costs, much better than expected weak scalability is seen for up to 100K cores on NSFs Kraken architecture and up to260K cpu cores, on DOEs new Titan machine. The scalability is found to depend in a crtitical way on the choice of algorithm used by hypre for a realistic application problem.

DAG-Based software frameworks for PDEs

The task-based approach to software and parallelism is well-known and has been proposed as a potential candidate, named the silver model, for exascale software. This approach is not yet widely used in the large-scale multi-core parallel computing of complex systems of partial differential equations. After surveying task-based approaches we investigate how well the Uintah software and an extension named Wasatch fit in the task-based paradigm and how well they perform on large scale parallel computers. The conclusion is that these approaches show great promise for petascale but that considerable algorithmic challenges remain.

The One-Dimensional-Turbulence Model

The one-dimensional turbulence (ODT) model represents an efficient and novel multiscale approach to couple the processes of reaction, diffusion and turbulent transport. The principal ingredients of the model include a coupled deterministic solution for reaction and molecular transport and a stochastic prescription for turbulent transport. The model may be implemented as stand-alone for simple turbulent flows and admits various forms for the description of spatially developing and temporally developing flows. It also may be implemented within the context of a coupled multiscale solution using the ODTLES approach. This chapter outlines the model formulation, and applications of ODT using stand-alone solutions and ODTLES.

A quantitative method for a priori evaluation of combustion reaction models

In recent years, direct numerical simulations have been used increasingly to evaluate the validity and performance of combustion reaction models. This study presents a new, quantitative method to determine the ideal model performance attainable by a given parameterization of the state variables. Data from direct numerical simulation (DNS) of unsteady CO/H2–air jet flames is analysed to determine how well various parameterizations represent the data, and how well specific models based on those parameterizations perform. Results show that the equilibrium model performs poorly relative to an ideal model parameterized by the mixture fraction. The steady laminar flamelet model performs quite well relative to an ideal model parameterized by mixture fraction and dissipation rate in some cases. However, at low dissipation rates or at dissipation rates exceeding the steady extinction limit, the steady flamelet model performs poorly. Interestingly, even in many cases where the steady flamelet model fails (particularly at low dissipation rate), the DNS data suggests that the state may be parameterized well by the mixture fraction and dissipation rate. A progress variable based on the CO2 mass fraction is proposed, together with a new model based on the CO2 progress variable. This model performs nearly ideally, and demonstrates the ability to capture extinction with remarkable accuracy for the CO/H2 flames considered.

Nebo: An efficient, parallel, and portable domain-specific language for numerically solving partial differential equations

Abstract This paper presents Nebo, a declarative domain-specific language embedded in C++ for discretizing partial differential equations for transport phenomena on multiple architectures. Application programmers use Nebo to write code that appears sequential but can be run in parallel, without editing the code. Currently Nebo supports single-thread execution, multi-thread execution, and many-core (GPU-based) execution. With single-thread execution, Nebo performs on par with code written by domain experts. With multi-thread execution, Nebo can linearly scale (with roughly 90% efficiency) up to 12 cores, compared to its single-thread execution. Moreover, Nebo’s many-core execution can be over 140x faster than its single-thread execution.

Dual timestepping methods for detailed combustion chemistry

In this work, we develop and study several dual time integration methods for the solution of stiff, explosive differential equations governing combustion chemistry. Dual time integration is an implicit method wherein the sub-iteration process of each timestep is performed as a steady-state integration process, rather than the commonly used Newton–Raphson method. This allows stabilisation when nonlinear ignition events are contained within a timestep, providing considerable freedom in the choice of resolved phenomena. Timesteps may be chosen so as to resolve relatively long process timescales accurately rather than fast chemical timescales, something not possible with the common Newtons method. We illustrate this method using several backward difference formula methods and demonstrate the efficacy of our method in resolving low-frequency solutions of continuous flow stirred-tank reactors with periodic ignition–extinction events. We are able to step over ignition–extinction events with our stable, adaptive dual time method, and we study numerical convergence and error scaling on process timescales.

Wasatch: An architecture-proof multiphysics development environment using a Domain Specific Language and graph theory

Abstract To address the coding and software challenges of modern hybrid architectures, we propose an approach to multiphysics code development for high-performance computing. This approach is based on using a Domain Specific Language (DSL) in tandem with a directed acyclic graph (DAG) representation of the problem to be solved that allows runtime algorithm generation. When coupled with a large-scale parallel framework, the result is a portable development framework capable of executing on hybrid platforms and handling the challenges of multiphysics applications. We share our experience developing a code in such an environment – an effort that spans an interdisciplinary team of engineers and computer scientists.

Reducing overhead in the Uintah framework to support short-lived tasks on GPU-heterogeneous architectures

The Uintah computational framework is used for the parallel solution of partial differential equations on adaptive mesh refinement grids using modern supercomputers. Uintah is structured with an application layer and a separate runtime system. The Uintah runtime system is based on a distributed directed acyclic graph (DAG) of computational tasks, with a task scheduler that efficiently schedules and execute these tasks on both CPU cores and on-node accelerators. The runtime system identifies task dependencies, creates a taskgraph prior to an iteration based on these dependencies, prepares data for tasks, automatically generates MPI message tags, and manages data after task computation. Managing tasks for accelerators pose significant challenges over their CPU task counterparts due to supporting more memory regions, API call latency, memory bandwidth concerns, and the added complexity of development. These challenges are greatest when tasks compute within a few milliseconds, especially those that have stencil based computations that involve halo data, have little reuse of data, and/or require many computational variables. Current and emerging heterogeneous architectures necessitate addressing these challenges within Uintah. This work is not designed to improve performance of existing tasks, but rather reduce runtime overhead to allow developers writing short-lived computational tasks to utilize Uintah in a heterogeneous environment. This work analyzes an initial approach for managing accelerator tasks alongside existing CPU tasks within Uintah. The principal contribution of this work is to identify and address inefficiencies that arise when mapping tasks onto the GPU, to implement new schemes to reduce runtime system overhead, to introduce new features that allow for more tasks to leverage on-node accelerators, and to show overhead reduction results from these improvements.