William F. Spotz

High-Resolution Mesh Convergence Properties and Parallel Efficiency of a Spectral Element Atmospheric Dynamical Core

We first demonstrate the parallel performance of the dynamical core of a spectral element atmospheric model. The model uses continuous Galerkin spectral elements to discretize the surface of the Earth, coupled with finite differences in the radial direction. Results are presented from two distributed memory, mesh interconnect supercomputers (ASCI Red and BlueGene/L), using a two-dimensional space filling curve domain decomposition. Better than 80% parallel efficiency is obtained for fixed grids on up to 8938 processors. These runs represent the largest processor counts ever achieved for a geophysical application. They show that the upcoming Red Storm and BlueGene/L super-computers are well suited for performing global atmospheric simulations with a 10 km average grid spacing. We then demonstrate the accuracy of the method by performing a full three-dimensional mesh refinement convergence study, using the primitive equations to model breaking Rossby waves on the polar vortex. Due to the excellent parallel performance, the model is run at several resolutions up to 36 km with 200 levels using only modest computing resources. Isosurfaces of scaled potential vorticity exhibit complex dynamical features, e.g. a primary potential vorticity tongue, and a secondary instability causing roll-up into a ring of five smaller subvortices. As the resolution is increased, these features are shown to converge while potential vorticity gradients steepen.

PyTrilinos: High-performance distributed-memory solvers for Python

PyTrilinos is a collection of Python modules that are useful for serial and parallel scientific computing. This collection contains modules that cover serial and parallel dense linear algebra, serial and parallel sparse linear algebra, direct and iterative linear solution techniques, domain decomposition and multilevel preconditioners, nonlinear solvers, and continuation algorithms. Also included are a variety of related utility functions and classes, including distributed I/O, coloring algorithms, and matrix generation. PyTrilinos vector objects are integrated with the popular NumPy Python module, gathering together a variety of high-level distributed computing operations with serial vector operations. PyTrilinos is a set of interfaces to existing, compiled libraries. This hybrid framework uses Python as front-end, and efficient precompiled libraries for all computationally expensive tasks. Thus, we take advantage of both the flexibility and ease of use of Python, and the efficiency of the underlying C++, C, and FORTRAN numerical kernels. Out numerical results show that, for many important problem classes, the overhead required by the Python interpreter is negligible. To run in parallel, PyTrilinos simply requires a standard Python interpreter. The fundamental MPI calls are encapsulated under an abstract layer that manages all interprocessor communications. This makes serial and parallel scripts using PyTrilinos virtually identical.

A semi-Lagrangian double Fourier method for the shallow water equations on the sphere

We describe a numerical method, based on the semi-Lagrangian semi-implicit approach, for solving the shallow water equations (SWEs) in spherical coordinates. The most popular spatial discretization method used in global atmospheric models is currently the spectral transform method, which generates high-order numerical solutions and provides an elegant solution to the pole problems induced by a spherical coordinate system. However, the standard spherical harmonic spectral transform method requires associated Legendre transforms, which for problems with resolutions of current interest, have a computational complexity of O(N3), where N is the number of spatial subintervals in one dimension. Thus, the double Fourier spectral method, which uses trigonometric series, may be a viable alternative. The advantage of the double Fourier method is that fast Fourier transforms, which have a computational complexity of O(N2 log N), can be used in both the longitudinal and latitudinal directions. In this implementation, the SWEs are discretized in time by means of the three-time-level semi-Lagrangian semi-implicit method, which integrates along fluid trajectories and allows large time steps while maintaining stability. Numerical results for the standard SWEs test suite are presented to demonstrate the stability and accuracy of the method.

Spherical harmonic projectors

The harmonic projection (HP), which is implicit in the numerous harmonic transforms between physical and spectral spaces, is responsible for the reliability of the spectral method for modeling geophysical phenomena. As currently configured, the HP consists of a forward transform from physical to spectral space (harmonic analysis) immediately followed by a harmonic synthesis back to physical space. Unlike its Fourier counterpart in Cartesian coordinates, the HP does not identically reconstruct the original function on the surface of the sphere but rather replaces it with a weighted least-squares approximation. The importance of the HP is that it uniformly resolves waves on the surface of the sphere and therefore eliminates high frequencies that are artificially induced by the clustering of grid points in the neighborhood of the poles. The HP also maintains spectral accuracy when combined with the double Fourier method. Originally the HP required O(N 3 ) storage where N is the number of latitudinal points. However, this was recently reduced to O(N 2 ) using an algorithm that also provided a savings of up to 50 percent in compute time. The HP was also generalized to an arbitrary latitudinal distribution of points. However, the HP as a composite of analysis and synthesis can be subject to considerable error depending on the point distribution. Here we define a variant of the traditional HP that is well conditioned, with condition number 1, for any point distribution. In addition, storage requirements are further reduced because the projections corresponding to all longitudinal wave numbers m are expressed in terms of a single orthogonal matrix.

Aeras: A next generation global atmosphere model

Sandia National Laboratories is developing a new global atmosphere model named Aeras that is performance portable and supports the quantification of uncertainties. These next-generation capabilities are enabled by building Aeras on top of Albany, a code base that supports the rapid development of scientific application codes while leveraging Sandias foundational mathematics and computer science packages in Trilinos and Dakota. Embedded uncertainty quantification (UQ) is an original design capability of Albany, and performance portability is a recent upgrade. Other required features, such as shell-type elements, spectral elements, efficient explicit and semi-implicit time-stepping, transient sensitivity analysis, and concurrent ensembles, were not components of Albany as the project began, and have been (or are being) added by the Aeras team. We present early UQ and performance portability results for the shallow water equations.

PyTrilinos: Recent advances in the Python interface to Trilinos

PyTrilinos is a set of Python interfaces to compiled Trilinos packages. This collection supports serial and parallel dense linear algebra, serial and parallel sparse linear algebra, direct and iterative linear solution techniques, algebraic and multilevel preconditioners, nonlinear solvers and continuation algorithms, eigensolvers and partitioning algorithms. Also included are a variety of related utility functions and classes, including distributed I/O, coloring algorithms and matrix generation. PyTrilinos vector objects are compatible with the popular NumPy Python package. As a Python front end to compiled libraries, PyTrilinos takes advantage of the flexibility and ease of use of Python, and the efficiency of the underlying C++, C and Fortran numerical kernels. This paper covers recent, previously unpublished advances in the PyTrilinos package.

Toward performance portability of the Albany finite element analysis code using the Kokkos library

Performance portability on heterogeneous high-performance computing (HPC) systems is a major challenge faced today by code developers: parallel code needs to be executed correctly as well as with high performance on machines with different architectures, operating systems, and software libraries. The finite element method (FEM) is a popular and flexible method for discretizing partial differential equations arising in a wide variety of scientific, engineering, and industrial applications that require HPC. This article presents some preliminary results pertaining to our development of a performance portable implementation of the FEM-based Albany code. Performance portability is achieved using the Kokkos library. We present performance results for the Aeras global atmosphere dynamical core module in Albany. Numerical experiments show that our single code implementation gives reasonable performance across three multicore/many-core architectures: NVIDIA General Processing Units (GPU’s), Intel Xeon Phis, and multicore CPUs.

Next-Generation Multiphase Flow Solver for Fluidized Bed Applications

A framework is developed to integrate MFiX (Multiphase Flow with Interphase eXchanges) with advanced linear solvers in Trilinos. MFiX is a widely used open source general purpose multiphase solver developed by National Energy Technology Laboratories and written in Fortran. Trilinos is an objectedoriented open source software development platform from Sandia National Laboratories for solving large scale multiphysics problems. The framework handles the different data structures in Fortran and C++ and exchanges the information from MFiX to Trilinos and vice versa. The integrated solver, called MFiXTrilinos hereafter, provides next-generation computational capabilities including scalable linear solvers for distributed memory massively parallel computers. In this paper, the solution from the standard linear solvers in MFiX-Trilinos is validated against the same from MFiX for 2D and 3D fluidized bed problems. The standard iterative solvers considered in this work are BiConjugate Gradient Stabilized (BiCGStab) and Generalized minimal residual methods (GMRES) as the matrix is non-symmetric in nature. The stopping criterion set for the iterative solvers is same. It is observed that the solution from the integrated solver and MFiX is in good agreement. NOMENCLATURE ~g Acceleration due to gravity

A framework to integrate MFiX with Trilinos for high fidelity fluidized bed computations

A framework is developed to integrate MFiX, an open source multiphase flow solver, with state-of-the-art pre-conditioners and linear solver packages in Trilinos via MFIX, Fortran, C and CPP wrappers. The computations are carried out to simulate flow in a fluidized bed problem with MFiX as well as the integrated solver, MFiX-Trilinos. BiConjugate gradient stabilized method as well as GMRES is used to solve the linear system of equations. The linear system of equations for the flow variable are solved using the built-in solvers in MFiX. On the other hand, MFiX-Trilinos uses the solvers from AztecOO package in Trilinos. The performance of the integrated solver is tested on various computer architectures for variety of problem sizes. The flow from the solver with the integrated framework and MFiX are in good agreement. However, the solver in MFiX-Trilinos is, approximately 30% faster compared to the same solver in MFiX.

A Python HPC Framework: PyTrilinos, ODIN, and Seamless

We present three Python software projects: PyTrilinos, for calling Trilinos distributed memory HPC solvers from Python; Optimized Distributed NumPy (ODIN), for distributed array computing; and Seamless, for automatic, Just-in-time compilation of Python source code. We argue that these three projects in combination provide a framework for high-performance computing in Python. They provide this framework by supplying necessary features (in the case of ODIN and Seamless) and algorithms (in the case of ODIN and PyTrilinos) for a user to develop HPC applications. Together they address the principal limitations (real or imagined) ascribed to Python when applied to high-performance computing. A high-level overview of each project is given, including brief explanations as to how these projects work in conjunction to the benefit of end users.