Rahul S. Sampath

Petascale Direct Numerical Simulation of Blood Flow on 200K Cores and Heterogeneous Architectures

We present a fast, petaflop-scalable algorithm for Stokesian particulate flows. Our goal is the direct simulation of blood, which we model as a mixture of a Stokesian fluid (plasma) and red blood cells (RBCs). Directly simulating blood is a challenging multiscale, multiphysics problem. We report simulations with up to 200 million deformable RBCs. The largest simulation amounts to 90 billion unknowns in space. In terms of the number of cells, we improve the state-of-the art by several orders of magnitude: the previous largest simulation, at the same physical fidelity as ours, resolved the flow of O(1,000-10,000) RBCs. Our approach has three distinct characteristics: (1) we faithfully represent the physics of RBCs by using nonlinear solid mechanics to capture the deformations of each cell; (2) we accurately resolve the long-range, N-body, hydrodynamic interactions between RBCs (which are caused by the surrounding plasma); and (3) we allow for the highly non-uniform distribution of RBCs in space. The new method has been implemented in the software library MOBO (for “Moving Boundaries”). We designed MOBO to support parallelism at all levels, including inter-node distributed memory parallelism, intra-node shared memory parallelism, data parallelism (vectorization), and fine-grained multithreading for GPUs. We have implemented and optimized the majority of the computation kernels on both Intel/AMD x86 and NVidias Tesla/Fermi platforms for single and double floating point precision. Overall, the code has scaled on 256 CPU-GPUs on the Teragrids Lincoln cluster and on 200,000 AMD cores of the Oak Ridge national Laboratorys Jaguar PF system. In our largest simulation, we have achieved 0.7 Petaflops/s of sustained performance on Jaguar.

Bottom-Up Construction and 2:1 Balance Refinement of Linear Octrees in Parallel

In this article, we propose new parallel algorithms for the construction and 2:1 balance refinement of large linear octrees on distributed memory machines. Such octrees are used in many problems in computational science and engineering, e.g., object representation, image analysis, unstructured meshing, finite elements, adaptive mesh refinement, and N-body simulations. Fixed-size scalability and isogranular analysis of the algorithms using an MPI-based parallel implementation was performed on a variety of input data and demonstrated good scalability for different processor counts (1 to 1024 processors) on the Pittsburgh Supercomputing Centers TCS-1 AlphaServer. The results are consistent for different data distributions. Octrees with over a billion octants were constructed and balanced in less than a minute on 1024 processors. Like other existing algorithms for constructing and balancing octrees, our algorithms have

A Parallel Geometric Multigrid Method for Finite Elements on Octree Meshes

\mathcal{O}(N\log N)

Low-constant parallel algorithms for finite element simulations using linear octrees

work and

Dendro: parallel algorithms for multigrid and AMR methods on 2:1 balanced octrees

\mathcal{O}(N)

Parallel Fast Gauss Transform

storage complexity. Under reasonable assumptions on the distribution of octants and the work per octant, the parallel time complexity is

Optimizing blocking and nonblocking reduction operations for multicore systems: Hierarchical design and implementation

\mathcal{O}(\frac{N}{n_p}\log(\frac{N}{n_p})+n_p\log n_p)

A parallel multi-domain solution methodology applied to nonlinear thermal transport problems in nuclear fuel pins

, where

A Validation Study of Pin Heat Transfer for MOX Fuel Based on the IFA-597 Experiments

N

Fiscal Year 2011 Infrastructure Refactorizations in AMP

is the size of the final linear octree and