Tony Saad

Large Scale Parallel Solution of Incompressible Flow Problems Using Uintah and Hypre

The Uintah Software framework was developed to provide an environment for solving fluid-structure interaction problems on structured adaptive grids on large-scale, long-running, data-intensive problems. Uintah uses a combination of fluid-flow solvers and particle-based methods for solids together with a novel asynchronous task-based approach with fully automated load balancing. As Uintah is often used to solve incompressible flow problems in combustion applications it is important to have a scalable linear solver. While there are many such solvers available, the scalability of those codes varies greatly. The hypre software offers a range of solvers and pre-conditioners for different types of grids. The weak scalability of Uintah and hypre is addressed for particular examples of both packages when applied to a number of incompressible flow problems. After careful software engineering to reduce startup costs, much better than expected weak scalability is seen for up to 100K cores on NSFs Kraken architecture and up to260K cpu cores, on DOEs new Titan machine. The scalability is found to depend in a crtitical way on the choice of algorithm used by hypre for a realistic application problem.

Wasatch: An architecture-proof multiphysics development environment using a Domain Specific Language and graph theory

Abstract To address the coding and software challenges of modern hybrid architectures, we propose an approach to multiphysics code development for high-performance computing. This approach is based on using a Domain Specific Language (DSL) in tandem with a directed acyclic graph (DAG) representation of the problem to be solved that allows runtime algorithm generation. When coupled with a large-scale parallel framework, the result is a portable development framework capable of executing on hybrid platforms and handling the challenges of multiphysics applications. We share our experience developing a code in such an environment – an effort that spans an interdisciplinary team of engineers and computer scientists.

Reducing overhead in the Uintah framework to support short-lived tasks on GPU-heterogeneous architectures

The Uintah computational framework is used for the parallel solution of partial differential equations on adaptive mesh refinement grids using modern supercomputers. Uintah is structured with an application layer and a separate runtime system. The Uintah runtime system is based on a distributed directed acyclic graph (DAG) of computational tasks, with a task scheduler that efficiently schedules and execute these tasks on both CPU cores and on-node accelerators. The runtime system identifies task dependencies, creates a taskgraph prior to an iteration based on these dependencies, prepares data for tasks, automatically generates MPI message tags, and manages data after task computation. Managing tasks for accelerators pose significant challenges over their CPU task counterparts due to supporting more memory regions, API call latency, memory bandwidth concerns, and the added complexity of development. These challenges are greatest when tasks compute within a few milliseconds, especially those that have stencil based computations that involve halo data, have little reuse of data, and/or require many computational variables. Current and emerging heterogeneous architectures necessitate addressing these challenges within Uintah. This work is not designed to improve performance of existing tasks, but rather reduce runtime overhead to allow developers writing short-lived computational tasks to utilize Uintah in a heterogeneous environment. This work analyzes an initial approach for managing accelerator tasks alongside existing CPU tasks within Uintah. The principal contribution of this work is to identify and address inefficiencies that arise when mapping tasks onto the GPU, to implement new schemes to reduce runtime system overhead, to introduce new features that allow for more tasks to leverage on-node accelerators, and to show overhead reduction results from these improvements.

The Discrete Operator Approach to the Numerical Solution of Partial Differential Equations

The design of robust computational physics codes has always been a challenge to application programmers. One of the key di culties in writing multiphysics codes stems from the ine cient handling of spatial discretization and field operations. For example, in the context of the finite volume (FV) method, one often deals with structured and unstructured meshes that are either staggered or collocated. To handle this array of options, a complex data structure is required to represent arbitrary meshes. For structured grids, this approach imposes an unnecessary computational overhead that increases significantly with problem size. Instead, the programmer must write specific components to handle structured meshes thus defeating the purpose of code portability. Furthermore, dedicated discretization schemes are required for di erent types of meshes, thus increasing software complexity. Although modern computational codes rely extensively on a variety of libraries for linear algebra operations, they lack a framework that provides proper application-independent discretization tools. In this manuscript, we present a novel computational paradigm that separates the spatial discretization and field operations from the physics. This paradigm is based on the abstraction of the mathematical operators describing physical processes. An operator corresponds to a precise mathematical object that performs a certain calculation on a field. A field corresponds to any scalar or vector variable required in the solution process. In our model, an operator is represented discretely by a sparse matrix while a field is represented by a vector. At the outset, the discretization process corresponds to a sparse-matrix-vector multiplication. This approach completely decouples the physics from the spatial and field operations thus providing an avenue for improved code design and usability.

Comment on “Diffusion by a random velocity field” [Phys. Fluids 13, 22 (1970)]

This comment aims at addressing a mass conservation issue in a paper published in the physics of fluids. The paper [R. H. Kraichnan, “Diffusion by a random velocity field,” Phys. Fluids 13(1), 22 (1970)] introduces a novel method to generate synthetic isotropic turbulence for computational purposes. The method has been used in the literature to generate inlet boundary conditions and to model aeroacoustic noise as well as for validation and verification purposes. However, the technique uses a continuous formulation to derive the mass conservation constraint. In this comment, we argue that the continuous constraint is invalid on a discrete grid and provide an alternative derivation using the discrete divergence. In addition, we present an analysis to quantify the impact of a pressure projection on the kinetic energy of a non-solenoidal velocity field.

Some thoughts on the pressure integration requirements of the Navier–Stokes equations

The Navier–Stokes formulation represents a uniquely challenging system of partial differential equations that continues to influence modern applied science and engineering. In its simplest form, the system can be used to prescribe the motion of a viscous incompressible fluid with constant properties. It consists of four equations in three-dimensional space that account for both the kinematic and dynamic conditions that a fluid element senses. In this work, we investigate the pressure integration rules and restrictions that affect the resolution of the scalar pressure field. We begin our analysis by exploring the integration properties of Eulers equations in two dimensions while making use of Clairauts theorem on the commutativity of mixed partial derivatives. We then extend our findings to three-dimensional space. This process gives rise to a theorem and four corollaries that help to clarify the conditions needed to obtain exact or asymptotic solutions for the pressure distribution. Consequently, we identify the fundamental conditions under which the Navier–Stokes equations can be properly integrated to arrive at an analytic expression for the pressure field, namely, one that is continuous and twice differentiable. In closing, several configurations are used to test the theorem and showcase its connection with the pressure formulation. These include potential flows for which the pressure can be obtained unconditionally, and inviscid rotational motions of the Taylor–Culick type with and without headwall injection.