Damian W. I. Rouson

OpenCoarrays: Open-source Transport Layers Supporting Coarray Fortran Compilers

Coarray Fortran is a set of features of the Fortran 2008 standard that make Fortran a PGAS parallel programming language. Two commercial compilers currently support coarrays: Cray and Intel. Here we present two coarray transport layers provided by the new OpenCoarrays project: one library based on MPI and the other on GASNet. We link the GNU Fortran (GFortran) compiler to either of the two OpenCoarrays implementations and present performance comparisons between executables produced by GFortran and the Cray and Intel compilers. The comparison includes synthetic benchmarks, application prototypes, and an application kernel. In our tests, Intel outperforms GFortran only on intra-node small transfers (in particular, scalars). GFortran outperforms Intel on intra-node array transfers and in all settings that require inter-node transfers. The Cray comparisons are mixed, with either GFortran or Cray being faster depending on the chosen hardware platform, network, and transport layer.

A Scalable and Adaptable Solution Framework within Components of the Community Climate System Model

A framework for a fully implicit solution method is implemented into (1) the High Order Methods Modeling Environment (HOMME), which is a spectral element dynamical core option in the Community Atmosphere Model (CAM), and (2) the Parallel Ocean Program (POP) model of the global ocean. Both of these models are components of the Community Climate System Model (CCSM). HOMME is a development version of CAM and provides a scalable alternative when run with an explicit time integrator. However, it suffers the typical time step size limit to maintain stability. POP uses a time-split semi-implicit time integrator that allows larger time steps but less accuracy when used with scale interacting physics. A fully implicit solution framework allows larger time step sizes and additional climate analysis capability such as model steady state and spin-up efficiency gains without a loss in scalability. This framework is implemented into HOMME and POP using a new Fortran interface to the Trilinos solver library, ForTrilinos, which leverages several new capabilities in the current Fortran standard to maximize robustness and speed. The ForTrilinos solution template was also designed for interchangeability; other solution methods and capability improvements can be more easily implemented into the models as they are developed without severely interacting with the code structure. The utility of this approach is illustrated with a test case for each of the climate component models.

Design patterns for multiphysics modeling in Fortran 2003 and C

We present three new object-oriented software design patterns in Fortran 2003 and C++. These patterns integrate coupled differential equations, facilitating the flexible swapping of physical and numerical software abstractions at compile-time and runtime. The Semi-Discrete pattern supports the time advancement of a dynamical system encapsulated in a single abstract data type (ADT). The Puppeteer pattern combines ADTs into a multiphysics package, mediates interabstraction communications, and enables implicit marching even when nonlinear terms couple separate ADTs with private data. The Surrogate pattern emulates C++ forward references in Fortran 2003. After code demonstrations using the Lorenz equations, we provide architectural descriptions of our use of the new patterns in extending the Rouson et al. [2008a] Navier-Stokes solver to simulate multiphysics phenomena. We also describe the relationships between the new patterns and two previously developed architectural elements: the Strategy pattern of Gamma et al. [1995] and the template emulation technique of Akin [2003]. This report demonstrates how these patterns manage complexity by providing logical separation between individual physics models and the control logic that bridges between them. Additionally, it shows how language features such as operator overloading and automated memory management enable a clear mathematical notation for model bridging and system evolution.

A case study: agile development in the community laser-induced incandescence modeling environment (CLiiME)

The multidisciplinary requirements of current computational modeling problems preclude the development of scientific software that is maintained and used by selected scientists. The multidisciplinary nature of these efforts requires the development of large scale software projects established with a wide developer and user base in mind. This article describes some of the software-engineering practices adopted in a scientific-software application for a laser-induced incandescence community model. The project uses an Agile and Test-Driven Development approach to implement the infrastructure for the development of a collaborative model that is to be extended, modified, and used by different researchers. We discuss some of the software-engineering practices that can be exploited through the life of a project, starting with its inception when only a hand full of developers are contributing to the project and the mechanism we have put in place in order to allow the natural expansion of the model.

A grid-free abstraction of the Navier-Stokes equations in Fortran 95/2003

Computational complexity theory inspires a grid-free abstraction of the Navier-Stokes equations in Fortran 95/2003. A novel complexity analysis estimates that structured programming time grows at least quadratically with the number of program lines. Further analysis demonstrates how an object-oriented strategy focused on mathematical objects renders the quadratic estimate scale-invariant, so the time required for the limiting factor in program development (debugging) no longer grows as the code grows. Compared to the coordinate-free C++ programming of Grant et al. [2000], grid-free Fortran programming eliminates a layer of procedure calls, eliminates a related need for the C++ template construct, and offers a shorter migration path for Fortran programmers. The grid-free strategy is demonstrated by constructing a physical-space driver for a Fourier-space Navier-Stokes solver. Separating the expression of the continuous mathematical model from the discrete numerics clarifies issues that are otherwise easily conflated. A run-time profile suggests that grid-free design substantially reduces the fraction of the procedures that significantly impact runtime, freeing more code to be structured in ways that reduce development time. Applying Amdahls law to the total solution time (development time plus run time) leads to a strategy that negligibly impacts development time but achieves 58% of the maximum possible speedup.

Dynamic Memory De-allocation in Fortran 95/2003 derived type calculus

Abstract data types developed for computational science and engineering are frequently modeled after physical objects whose state variables must satisfy governing differential equations. Generalizing the associated algebraic and differential operators to operate on the abstract data types facilitates high-level program constructs that mimic standard mathematical notation. For non-trivial expressions, multiple object instantiations must occur to hold intermediate results during the expressions evaluation. When the dimension of each objects state space is not specified at compile-time, the programmer becomes responsible for dynamically allocating and de-allocating memory for each instantiation. With the advent of allocatable components in Fortran 2003 derived types, the potential exists for these intermediate results to occupy a substantial fraction of a programs footprint in memory. This issue becomes particularly acute at the highest levels of abstraction where coarse-grained data structures predominate. This paper proposes a set of rules for de-allocating memory that has been dynamically allocated for intermediate results in derived type calculus, while distinguishing that memory from more persistent objects. The new rules are applied to the design of a polymorphic time integrator for integrating evolution equations governing dynamical systems. Associated issues of efficiency and design robustness are discussed.

Design patterns for scientific computations on sparse matrices

We discuss object-oriented software design patterns in the context of scientific computations on sparse matrices. Design patterns arise when multiple independent development efforts produce very similar designs, yielding an evolutionary convergence onto a good solution: a flexible, maintainable, high-performance design. We demonstrate how to engender these traits by implementing an interface for sparse matrix computations on NVIDIA GPUs starting from an existing sparse matrix library. We also present initial performance results.

Dispersed-phase structural anisotropy in homogeneous magnetohydrodynamic turbulence at low magnetic Reynolds number

A new tensor statistic, the dispersed-phase structure dimensionality Dp, is defined to describe the preferred orientation of clusters of discrete bodies. The evolution of Dp is calculated via direct numerical simulations of passive, Stokesian particles driven by initially isotropic, decaying magnetohydrodynamic turbulence. Results are presented for five magnetic field strengths as characterized by magnetic interaction parameters, N, in the range 0–50. Four field strengths are studied at a grid resolution of 1283. The strongest field strength is also studied at 2563 resolution. In each case, the externally applied magnetic field was spatially uniform and followed a step function in time. Particles with initially uniform distributions were tracked through hydrodynamic turbulence for up to 2800 particle response times before the step change in the magnetic field. In the lower resolution simulation, the particle response time, τp, matched the Kolmogorov time scale at the magnetic field application time t0. Th...

Design patterns for sparse-matrix computations on hybrid CPU/GPU platforms

We apply object-oriented software design patterns to develop code for scientific software involving sparse matrices. Design patterns arise when multiple independent developments produce similar designs which converge onto a generic solution. We demonstrate how to use design patterns to implement an interface for sparse matrix computations on NVIDIA GPUs starting from PSBLAS, an existing sparse matrix library, and from existing sets of GPU kernels for sparse matrices. We also compare the throughput of the PSBLAS sparse matrix--vector multiplication on two platforms exploiting the GPU with that obtained by a CPU-only PSBLAS implementation. Our experiments exhibit encouraging results regarding the comparison between CPU and GPU executions in double precision, obtaining a speedup of up to 35.35 on NVIDIA GTX 285 with respect to AMD Athlon 7750, and up to 10.15 on NVIDIA Tesla C2050 with respect to Intel Xeon X5650.

Formal constraints on memory management for composite overloaded operations

The memory management rules for abstract data type calculus presented by Rouson, Morris & Xu [15] are recast as formal statements in the Object Constraint Language (OCL) and applied to the design of a thermal energy equation solver. One set of constraints eliminates memory leaks observed in composite overloaded expressions with three current Fortran 95/2003 compilers. A second set of constraints ensures economical memory recycling. The constraints are preconditions, postconditions and invariants on overloaded operators and the objects they receive and return. It is demonstrated that systematic run-time assertion checking inspired by the formal constraints facilitated the pinpointing of an exceptionally hard-to-reproduce compiler bug. It is further demonstrated that the interplay between OCLs modeling capabilities and Fortrans programming capabilities led to a conceptual breakthrough that greatly improved the readability of our code by facilitating operator overloading. The advantages and disadvantages of our memory management rules are discussed in light of other published solutions [11,19]. Finally, it is demonstrated that the run-time assertion checking has a negligible impact on performance.