Matthew Tyler Bettencourt

TOWARDS EXTREME-SCALE SIMULATIONS FOR LOW MACH FLUIDS WITH SECOND-GENERATION TRILINOS

Trilinos is an object-oriented software framework for the solution of large-scale, complex multi-physics engineering and scientific problems. While Trilinos was originally designed for scalable solutions of large problems, the fidelity needed by many simulations is significantly greater than what one could have envisioned two decades ago. When problem sizes exceed a billion elements even scalable applications and solver stacks require a complete revision. The second-generation Trilinos employs C++ templates in order to solve arbitrarily large problems. We present a case study of the integration of Trilinos with a low Mach fluids engineering application (SIERRA low Mach module/Nalu). Through the use of improved algorithms and better software engineering practices, we demonstrate good weak scaling for up to a nine billion element large eddy simulation (LES) problem on unstructured meshes with a 27 billion row matrix on 524,288 cores of an IBM Blue Gene/Q platform.

ASC ATDM Level 2 Milestone #5325: Asynchronous Many-Task Runtime System Analysis and Assessment for Next Generation Platforms.

This report provides in-depth information and analysis to help create a technical road map for developing nextgeneration programming models and runtime systems that support Advanced Simulation and Computing (ASC) workload requirements. The focus herein is on asynchronous many-task (AMT) model and runtime systems, which are of great interest in the context of “exascale” computing, as they hold the promise to address key issues associated with future extreme-scale computer architectures. This report includes a thorough qualitative and quantitative examination of three best-of-class AMT runtime systems—Charm++, Legion, and Uintah, all of which are in use as part of the ASC Predictive Science Academic Alliance Program II (PSAAP-II) Centers. The studies focus on each of the runtimes’ programmability, performance, and mutability. Through the experiments and analysis presented, several overarching findings emerge. From a performance perspective, AMT runtimes show tremendous potential for addressing extremescale challenges. Empirical studies show an AMT runtime can mitigate performance heterogeneity inherent to the machine itself and that Message Passing Interface (MPI) and AMT runtimes perform comparably under balanced conditions. From a programmability and mutability perspective however, none of the runtimes in this study are currently ready for use in developing production-ready Sandia ASC applications. The report concludes by recommending a codesign path forward, wherein application, programming model, and runtime system developers work together to define requirements and solutions. Such a requirements-driven co-design approach benefits the high-performance computing (HPC) community as a whole, with widespread community engagement mitigating risk for both application developers and runtime system developers.

Towards Extreme-Scale Simulations with Next-Generation Trilinos: A Low Mach Fluid Application Case Study

Trilinos is an object-oriented software framework for the solution of large-scale, complex multi-physics engineering and scientific problems. While the original version of Trilinos was designed for highly scalable solutions for large problems, the need for increasingly higher fidelity simulations has pushed the problem sizes beyond what could have been envisioned two decades ago. When problem sizes exceed a billion elements even highly scalable applications and solver stacks require a complete revision. The next-generation Trilinos employs C++ templates in order to solve arbitrarily large problems and enable extreme-scale simulations. We present a case study that involves integration of Trilinos with an engineering application (Sierra low Mach module/Nalu), involving the simulation of low Mach fluid flow for problems of size up to nine billion elements. Through the use of improved algorithms and better software engineering practices, we demonstrate good weak scaling for the matrix assembly and solve for the engineering application for up to a nine billion element fluid flow large eddy simulation (LES) problem on unstructured meshes with a 27 billion row matrix on 131,072 cores of a Cray XE6 platform.

Controlling Self-Force for Unstructured Particle-in-Cell (PIC) Codes

A new algorithm was developed, which reduces the self-force in particle-in-cell codes on unstructured meshes in a predictable and controllable way. This is accomplished by computing a charge density weighting function for a particle, which reproduces the Greens function solution to Poissons equation at nodes when using a standard finite element method methodology. This provides a superior local potential and allows for particle-particle particle-mesh techniques to be used to subtract off local force contributions, including fictitious self-forces resulting in accurate long-range forces on a particle and improved local Coulomb collisions. Local physical forces are then computed using the Greens function on local particle pairs and added to the long-range forces. Results were shown with up to five orders reduction in self-force and superior intraparticle forces for two test cases.

Weighting schemes for charges and fields to control self-force in unstructured finite element Particle-in-Cell codes

Summary form only given. Particle-in-Cell (PIC) is a powerful technique for simulating physical processes where discrete objects exert forces on each other. In the context of this talk we are examining electrostatic plasmas where the particles represent electrons or ions which are free to move around the system and the electric fields are defined on the mesh. While this talk focuses on electrostatics, the techniques discussed are generalizable to other domains. In traditional structured codes symmetry between the weighting operators which map the particles charge to the mesh and the electric field from the mesh back to the particle results in no self-force. In the unstructured finite element world the traditional approaches result in a self-force which causes a particle to push itself and to violate Newtons laws of motion. This talk focuses on a novel approach in controlling the selfforce of particles in PIC codes. This approach chooses a weighting scheme which closely reproduces the exact potential at grid nodes both near a charged particle and for the long range effects. This potential can then be differenced to compute an electric field at particle locations allowing for a exact cancellation up to the tolerance which the exact solution is reproduced at grid locations. This algorithm can be combined with a particle-particle--particle-mesh (P3M) approach to cancel all local effects and compute the N2 terms directly for a local patch resulting in reduced self-force and superior spatial resolution. This talk will present the details of this algorithm, lower self-force on a single particle, better Coulomb collisions for a few particles, and accurate results for highly under-refined meshes while maintaining the geometric flexibility of unstructured grids.

Performance portable sparse approximate inverse preconditioner for EFIE equations

A block base sparse approximate inverse preconditioner for the electric field integral equations is documented and tested. It utilized the Kokkos library for performance portability and shows superior performance when compared to a direct method, 36x faster for a 112K DOF problem. Furthermore, due to the abstractions available in the Kokkos library it allows one to migrate from CPU to GPU in a trivial way.

Mini-PIC — A Particle-In-Cell (PIC) code on unstructured grids for next generation platforms

This talk outlines the infrastructure and capabilities of Mini-PIC, an open-source electrostatic/electromagnetic Particle-In-Cell (PIC) code, which has been developed to simulate low density plasmas in complex domains on unstructured meshes. This application is hybrid MPI+X and has been developed using the Kokkos (part of the Trilinos solver library) abstraction layer for the X. Kokkos enables execution on traditional clusters, Intel Phi, and GPUs using threads, OpenMP or Cuda all with the same application code. The application uses a FEM approach for fields and the linear systems are stored and solved with Tpetra, the next-generation solvers stack as part of Trilinos. Performance results will be given for Sandybridge CPUs, Nights Corner Intel Phi, and Nvidia K40m cards showing performant, scalable algorithms across the various platforms. Mini-PIC is available on the Mantevo website http://mantevo.org and is distributed under the BSD license.

PPPS-2013: Accommodating large temporal, spatial, and particle weighting demands for simulating vacuum ARC discharge

Summary form only given. We have developed novel modeling approaches for simulation breakdown of vacuum arcs. Initiating an arc in vacuum spans many orders of magnitude in temporal and spatial scales, and number densities. We have developed specific approaches for each of these challenges-some more tested than others. These approaches are implemented in Aleph, a massively parallel 3D unstructured mesh electrostatic PIC-DSMC code. Aleph includes dynamic load balancing, volume chemistry (elastic collisions, charge exchange, ionization, etc.), and a variety of surface mechanisms. Our tool chain allows us to use conformal meshes to CAD geometry, a requirement for production use.

3D vacuum ARC breakdown simulation: Many challenges and some solutions

Summary form only given. We present our current capabilities and plans targeting the simulation of 3D vacuum arc discharge in realistic geometries. Vacuum arc discharge is an incredibly challenging problem due to the enormous dynamic changes in plasma growth, collisional processes, and time scales. Our simulation model targets a co-planar Cu-Cu vacuum breakdown experiment. We will estimate the computational requirements for this physically relevant breakdown system assuming a fully kinetic description. A fully kinetic description is required to accurately capture the initial breakdown. Progress on unstructured mesh collisional PIC methodology, dynamic particle weighting, managing multiple temporal and spatial scales, electrode models, and efficient parallel scaling will be addressed.