Anshu Dubey

A survey of high level frameworks in block-structured adaptive mesh refinement packages

Over the last decade block-structured adaptive mesh refinement (SAMR) has found increasing use in large, publicly available codes and frameworks. SAMR frameworks have evolved along different paths. Some have stayed focused on specific domain areas, others have pursued a more general functionality, providing the building blocks for a larger variety of applications. In this survey paper we examine a representative set of SAMR packages and SAMR-based codes that have been in existence for half a decade or more, have a reasonably sized and active user base outside of their home institutions, and are publicly available. The set consists of a mix of SAMR packages and application codes that cover a broad range of scientific domains. We look at their high-level frameworks, their design trade-offs and their approach to dealing with the advent of radical changes in hardware architecture. The codes included in this survey are BoxLib, Cactus, Chombo, Enzo, FLASH, and Uintah. A survey of mature openly available state-of-the-art structured AMR libraries and codes.Discussion of their frameworks, challenges and design trade-offs.Directions being pursued by the codes to prepare for the future many-core and heterogeneous platforms.

Evolution of FLASH, a multi-physics scientific simulation code for high-performance computing

The FLASH code has evolved into a modular and extensible scientific simulation software system over the decade of its existence. During this time it has been cumulatively used by over a thousand researchers to investigate problems in astrophysics, cosmology, and in some areas of basic physics, such as turbulence. Recently, many new capabilities have been added to the code to enable it to simulate problems in high-energy density physics. Enhancements to these capabilities continue, along with enhancements enabling simulations of problems in fluid-structure interactions. The code started its life as an amalgamation of already existing software packages and sections of codes developed independently by various participating members of the team for other purposes. The code has evolved through a mixture of incremental and deep infrastructural changes. In the process, it has undergone four major revisions, three of which involved a significant architectural advancement. Along the way, a software process evolved that addresses the issues of code verification, maintainability, and support for the expanding user base. The software process also resolves the conflicts arising out of being in development and production simultaneously with multiple research projects, and between performance and portability. This paper describes the process of code evolution with emphasis on the design decisions and software management policies that have been instrumental in the success of the code. The paper also makes the case for a symbiotic relationship between scientific research and good software engineering of the simulation software.

Software Abstractions and Methodologies for HPC Simulation Codes on Future Architectures

Simulations with multi-physics modeling have become crucial to many science and engineering fields, and multi-physics capable scientific software is as important to these fields as instruments and facilities are to experimental sciences. The current generation of mature multi-physics codes would have sustainably served their target communities with modest amount of ongoing investment for enhancing capabilities. However, the revolution occurring in the hardware architecture has made it necessary to tackle the parallelism and performance management in these codes at multiple levels. The requirements of various levels are often at cross-purposes with one another, and therefore hugely complicate the software design. All of these considerations make it essential to approach this challenge cooperatively as a community. We conducted a series of workshops under an NSF-SI2 conceptualization grant to get input from various stakeholders, and to identify broad approaches that might lead to a solution. In this position paper we detail the major concerns articulated by the application code developers, and emerging trends in utilization of programming abstractions that we found through these workshops.

Trends in Data Locality Abstractions for HPC Systems

The cost of data movement has always been an important concern in high performance computing (HPC) systems. It has now become the dominant factor in terms of both energy consumption and performance. Support for expression of data locality has been explored in the past, but those efforts have had only modest success in being adopted in HPC applications for various reasons. them However, with the increasing complexity of the memory hierarchy and higher parallelism in emerging HPC systems, locality management has acquired a new urgency. Developers can no longer limit themselves to low-level solutions and ignore the potential for productivity and performance portability obtained by using locality abstractions. Fortunately, the trend emerging in recent literature on the topic alleviates many of the concerns that got in the way of their adoption by application developers. Data locality abstractions are available in the forms of libraries, data structures, languages and runtime systems; a common theme is increasing productivity without sacrificing performance. This paper examines these trends and identifies commonalities that can combine various locality concepts to develop a comprehensive approach to expressing and managing data locality on future large-scale high-performance computing systems.

Stencils in Scientific Computations

Stencils occur in many areas, but they are ubiquitous in scientific computing. They range from the simple Jacobi iterations to the extremely complex ones used in the solution of highly nonlinear partial differential equations (PDE). High level programming languages typically used in implementation of scientific software, by not providing explicit support for stencils, force each implementation to make choices about expressing its specifics such as dimensionality, data layout, order of access and order of operations. These choices often hide the opportunity for optimizations from the compilers. Therehave been attempts to provide abstractions for simpler stencils, and they have met with success in some areas, but multiphysics scientific applications present challenges that cannot be met by simple stencil abstractions. The applications may have hierarchy, or non-uniformity, or both in their discretizations which cannot be expressed by stencils describing uniform discretizations. The physics operators being applied maybe non-linear which would demand composability of stencils. As the order of the solution method increases, the size and the reach of stencil also increases, and there may be conditions that imply the application of the stencil to an arbitrary subset of the discretized points. And finally, if there are multiple steps involved in an update, intermediate results need to be managed. AMR Shift Calculus, (Phil Colella and Brian Van Straalen 2014), provides a generalized abstraction that addresses many of these concerns. It provides a means of expressing stencil computations in the form of a collection of shift operations combined with associated weights, that can be applied to a specified collection of discretized points. The shift calculus also addresses the hierarchy in the discretization, and defines operators on stencils that allow more complex stencils to be composed from simpler ones. Because the shift calculus makes it possible to express the computation concisely and precisely, it gets around the problem of false dependencies. Additionally, the composability of the stencil operators exposes possibilities of loop or even function fusion, and the granularity for holding intermediate values to the compiler for better optimization opportunities. The included slide presentation is organized in five sections. The first section gives examples of discretization from simple Poisson to complex compressible Navier-Stokes (CNS) equations and addresses thelevel of abstraction needed to express the computations on these discretizations. The second section outlines several challenges that are unique to scientific applications, and the ways in which many abstractions that have proved useful elsewhere fail to work with scientific computing. The third section goes on to describe the AMR shift calculus with emphasis on features that are typically not found in other approaches to stencils based abstractions, but are necessary for the solving complex PDEs in scientific computing. The fourth section provides an example of applying one aspect of the shift calculus to a CNS solver implemented in F90. The example replaces loop-nests that explicitly implement the first and second derivative operators being applied to various field variables in the solver. It not only collapses the line-count in the code, but also removes unnecessary specificity of the order of floating point operations in the implementation. Finally, the fifth section talks about the opportunities that are made available to compilers for optimizations by expressing the computations in a high level abstraction.

Granularity and the cost of error recovery in resilient AMR scientific applications

Supercomputing platforms are expected to have larger failure rates in the future because of scaling and power concerns. The memory and performance impact may vary with error types and failure modes. Therefore, localized recovery schemes will be important for scientific computations, including failure modes where application intervention is suitable for recovery. We present a resiliency methodology for applications using structured adaptive mesh refinement, where failure modes map to granularities within the application for detection and correction. This approach also enables parameterization of cost for differentiated recovery. The cost model is built with tuning parameters that can be used to customize the strategy for different failure rates in different computing environments. We also show that this approach can make recovery cost proportional to the failure rate.

Experiences from Software Engineering of Large Scale AMR Multiphysics Code Frameworks

Many research problems are being pursued through simulations that require multi-physics capabilities in codes, that are also able to run on HPC platforms. Multiphysics implies many solvers with divergent, and sometimes conflicting, demands on the software infrastructure. Additionally many multiphysics simulation codes make use of structured adaptive mesh refinement to achieve maximum resolution where needed within resource constraints, which places even more demands on software infrastructure. The software architecture and process for these codes, therefore, is a challenging task. In this experience report we detail the challenges faced, design choices made, and insights from two such major software efforts, FLASH and Chombo.

Fault tolerance using lower fidelity data in adaptive mesh applications

Many high performance scientific simulation codes use checkpointing for multiple reasons. In addition to having the flexibility to complete the simulation in multiple job submissions, it has also provided an adequate recovery mechanism up to the current generation of platforms. With the advent of million-way parallelism, application codes are looking for additional options for recovery that may or may not be transparent to the applications. In many instances the applications can make the best judgement about the acceptability of the recovered solution. In this paper, we explore one option for recovering from multiple faults in codes using block-structured adaptive mesh refinement (AMR). The AMR codes have easy access to low-fidelity solution in the same physical space where they are also computing higher-fidelity solution. When a fault occurs, this low-fidelity solution can be used to reconstruct the higher fidelity solution in-flight. We report our findings from one implementation of such a strategy in FLASH, a block-structured adaptive mesh refinement community code for simulation of reactive compressible flows. In all our experiments the mechanism proved to be within the error bounds of the considered applications.

Parallel Algorithms for Using Lagrangian Markers in Immersed Boundary Method with Adaptive Mesh Refinement in FLASH

Computational fluid dynamics (CFD) are at the forefront of computational mechanics in requiring large-scale computational resources associated with high performance computing (HPC). Many flows of practical interest also include moving and deforming boundaries. High fidelity computations of fluid-structure interactions (FSI) are amongst the most challenging problems in computational mechanics. Additionally, many FSI applications have different resolution requirements in different parts of the domain and therefore requirement adaptive mesh refinement (AMR) for computational efficiency. FLASH is a well established AMR code with an existing Lagrangian framework which could be augmented and exploited to implement an immersed boundary method for simulating fluid-structure interactions atop an existing infrastructure. This paper describes the augmentations to the Lagrangian framework, and the new parallel algorithms added to the FLASH infrastructure that enabled the implementation of immersed boundary method in FLASH. The paper also presents scaling behavior and performance analysis of the implementations.

Methodology for building granular testing in multicomponent scientific software

Computational science and engineering communities develop complex application software with multiple mathematical models that need to interact with one another. Partly due to complexity of verifying scientific software, and partly because of the way incentives work in science, there has been insufficient testing of these codes. With a spotlight on the results produced with scientific software, and increasing awareness of software testing and verification as a critical contributor to the reliability of these results, testing is gaining more attention by the developing teams. However, many science teams struggle to find a good solution for themselves due either to lack of training or lack of resources within the team. In this experience paper we describe test development methodologies utilized in two different scenarios: one explains a methodology for building granular tests where none existed before, while the second demonstrates a methodology for selecting test cases that build confidence in the software through a process similar to scaffolding. The common insight from both the experiences is that testing should be a part of software design from the beginning for better software and scientific productivity.