Bobby Philip

Solution of Equilibrium Radiation Diffusion Problems Using Implicit Adaptive Mesh Refinement

Diffusion approximations to radiation transport feature a nonlinear conduction coefficient that leads to formation of a sharp front, or Marshak wave, under suitable initial and boundary conditions. The front can vary several orders of magnitude over a very short distance. Resolving the shape of the Marshak wave is essential, but using a global fine mesh can be prohibitively expensive. In such circumstances it is natural to consider using adaptive mesh refinement (AMR) to place a fine mesh only in the vicinity of the propagating front. In addition, to avoid any loss of accuracy due to linearization, implicit time integration should be used to solve the equilibrium radiation diffusion equation. Implicit time integration on AMR grids introduces a new challenge, as algorithmic complexity must be controlled to fully realize the performance benefits of AMR\@. A Newton--Krylov method together with a multigrid preconditioner addresses this latter issue on a uniform grid. A straightforward generalization is to use a multilevel preconditioner that is tuned to the structure of the AMR grid, such as the fast adaptive composite grid (FAC) method. We describe the resulting Newton--Krylov-FAC method and demonstrate its performance on simple equilibrium radiation diffusion problems.

Accelerated application development

Display Omitted Lessons learned are given for moving applications to the GPU-based Titan system.A carefully managed readiness effort is essential to preparing for new hardware.Applications typically require code restructuring to port to accelerators.Exposing more parallelism and minimizing data traffic are common porting themes.Performance gains of 2X-7X have been realized for application codes on Titan. The use of computational accelerators such as NVIDIA GPUs and Intel Xeon Phi processors is now widespread in the high performance computing community, with many applications delivering impressive performance gains. However, programming these systems for high performance, performance portability and software maintainability has been a challenge. In this paper we discuss experiences porting applications to the Titan system. Titan, which began planning in 2009 and was deployed for general use in 2013, was the first multi-petaflop system based on accelerator hardware. To ready applications for accelerated computing, a preparedness effort was undertaken prior to delivery of Titan. In this paper we report experiences and lessons learned from this process and describe how users are currently making use of computational accelerators on Titan.

Adaptive algebraic smoothers

This paper will present a new method of adaptively constructing block iterative methods based on Local Sensitivity Analysis (LSA). The method can be used in the context of geometric and algebraic multigrid methods for constructing smoothers, and in the context of Krylov methods for constructing block preconditioners. It is suitable for both constant and variable coefficient problems. Furthermore, the method can be applied to systems arising from both scalar and coupled system partial differential equations (PDEs), as well as linear systems that do not arise from PDEs. The simplicity of the method will allow it to be easily incorporated into existing multigrid and Krylov solvers while providing a powerful tool for adaptively constructing methods tuned to a problem.

Geometric multigrid for an implicit-time immersed boundary method

The immersed boundary (IB) method is an approach to fluid-structure interaction that uses Lagrangian variables to describe the deformations and resulting forces of the structure and Eulerian variables to describe the motion and forces of the fluid. Explicit time stepping schemes for the IB method require solvers only for Eulerian equations, for which fast Cartesian grid solution methods are available. Such methods are relatively straightforward to develop and are widely used in practice but often require very small time steps to maintain stability. Implicit-time IB methods permit the stable use of large time steps, but efficient implementations of such methods require significantly more complex solvers that effectively treat both Lagrangian and Eulerian variables simultaneously. Several different approaches to solving the coupled Lagrangian-Eulerian equations have been proposed, but a complete understanding of this problem is still emerging. This paper presents a geometric multigrid method for an implicit-time discretization of the IB equations. This multigrid scheme uses a generalization of box relaxation that is shown to handle problems in which the physical stiffness of the structure is very large. Numerical examples are provided to illustrate the effectiveness and efficiency of the algorithms described herein. These tests show that using multigrid as a preconditioner for a Krylov method yields improvements in both robustness and efficiency as compared to using multigrid as a solver. They also demonstrate that with a time step 100–1000 times larger than that permitted by an explicit IB method, the multigrid-preconditioned implicit IB method is approximately 50–200 times more efficient than the explicit method.

An assessment of coupling algorithms for nuclear reactor core physics simulations

This paper evaluates the performance of multiphysics coupling algorithms applied to a light water nuclear reactor core simulation. The simulation couples the k-eigenvalue form of the neutron transport equation with heat conduction and subchannel flow equations. We compare Picard iteration (block Gauss-Seidel) to Anderson acceleration and multiple variants of preconditioned Jacobian-free Newton-Krylov (JFNK). The performance of the methods are evaluated over a range of energy group structures and core power levels. A novel physics-based approximation to a Jacobian-vector product has been developed to mitigate the impact of expensive on-line cross section processing steps. Numerical simulations demonstrating the efficiency of JFNK and Anderson acceleration relative to standard Picard iteration are performed on a 3D model of a nuclear fuel assembly. Both criticality (k-eigenvalue) and critical boron search problems are considered.

A parallel multi-domain solution methodology applied to nonlinear thermal transport problems in nuclear fuel pins

This paper describes an efficient and nonlinearly consistent parallel solution methodology for solving coupled nonlinear thermal transport problems that occur in nuclear reactor applications over hundreds of individual 3D physical subdomains. Efficiency is obtained by leveraging knowledge of the physical domains, the physics on individual domains, and the couplings between them for preconditioning within a Jacobian Free Newton Krylov method. Details of the computational infrastructure that enabled this work, namely the open source Advanced Multi-Physics (AMP) package developed by the authors is described. Details of verification and validation experiments, and parallel performance analysis in weak and strong scaling studies demonstrating the achieved efficiency of the algorithm are presented. Furthermore, numerical experiments demonstrate that the preconditioner developed is independent of the number of fuel subdomains in a fuel rod, which is particularly important when simulating different types of fuel rods. Finally, we demonstrate the power of the coupling methodology by considering problems with couplings between surface and volume physics and coupling of nonlinear thermal transport in fuel rods to an external radiation transport code.

Dynamic implicit 3D adaptive mesh refinement for non-equilibrium radiation diffusion

The time dependent non-equilibrium radiation diffusion equations are important for solving the transport of energy through radiation in optically thick regimes and find applications in several fields including astrophysics and inertial confinement fusion. The associated initial boundary value problems that are encountered often exhibit a wide range of scales in space and time and are extremely challenging to solve. To efficiently and accurately simulate these systems we describe our research on combining techniques that will also find use more broadly for long term time integration of nonlinear multi-physics systems: implicit time integration for efficient long term time integration of stiff multi-physics systems, local control theory based step size control to minimize the required global number of time steps while controlling accuracy, dynamic 3D adaptive mesh refinement (AMR) to minimize memory and computational costs, Jacobian Free Newton-Krylov methods on AMR grids for efficient nonlinear solution, and optimal multilevel preconditioner components that provide level independent solver convergence.

A Validation Study of Pin Heat Transfer for MOX Fuel Based on the IFA-597 Experiments

Abstract The Integrated Fuel Assessment IFA-432 experiments from the International Fuel Performance Experiments database were designed to study the effects of gap size, fuel density, and fuel densification on fuel centerline temperature in light water reactor fuel. An evaluation of nuclear fuel pin heat transfer in the FRAPCON-3.4 and Exnihilo codes for uranium dioxide (UO2) fuel systems was performed, with a focus on the densification stage (2.2 GWd/tonne UO2). In addition, sensitivity studies were performed to evaluate the effect of the radial power shape and approximations to the geometry to account for the thermocouple hole. The analysis demonstrated excellent agreement for rods 1, 2, 3, and 5 (varying gap thicknesses and density with traditional fuel), demonstrating the accuracy of the codes and their underlying material models for traditional fuel. For rod 6, which contained unstable fuel that densified an order of magnitude more than traditional, stable fuel, the magnitude of densification was overpredicted, and the temperatures were outside the experimental uncertainty. The radial power shape within the fuel was shown to have a significant impact on the predicted centerline temperatures, whereas the effect of modeling the fuel at the thermocouple location as either annular or solid was relatively negligible. This has provided an initial benchmarking of the pin heat transfer capability of Exnihilo for UO2 fuel with respect to a well-validated nuclear fuel performance code.

Communication Characterization and Optimization of Applications Using Topology-Aware Task Mapping on Large Supercomputers

On large supercomputers, the job scheduling systems may assign a non-contiguous node allocation for user applications depending on available resources. With parallel applications using MPI (Message Passing Interface), the default process ordering does not take into account the actual physical node layout available to the application. This contributes to non-locality in terms of physical network topology and impacts communication performance of the application. In order to mitigate such performance penalties, this work describes techniques to identify suitable task mapping that takes the layout of the allocated nodes as well as the applications communication behavior into account. During the first phase of this research, we instrumented and collected performance data to characterize communication behavior of critical US DOE (United States - Department of Energy) applications using an augmented version of the mpiP tool. Subsequently, we developed several reordering methods (spectral bisection, neighbor join tree etc.) to combine node layout and application communication data for optimized task placement. We developed a tool called mpiAproxy to facilitate detailed evaluation of the various reordering algorithms without requiring full application executions. This work presents a comprehensive performance evaluation (14,000 experiments) of the various task mapping techniques in lowering communication costs on Titan, the leadership class supercomputer at Oak Ridge National Laboratory.

Block-Relaxation Methods for 3D Constant-Coefficient Stencils on GPUs and Multicore CPUs

Block iterative methods are extremely important as smoothers for multigrid methods, as preconditioners for Krylov methods, and as solvers for diagonally dominant linear systems. Developing robust and efficient smoother algorithms suitable for current and evolving GPU and multicore CPU systems is a significant challenge. We address this issue in the case of constant-coefficient stencils arising in the solution of elliptic partial differential equations on structured 3D uniform and adaptively refined block structured grids. Robust, highly parallel implementations of block Jacobi and chaotic block Gauss-Seidel algorithms with exact inversion of the blocks are developed using different parallelization techniques. Experimental results for NVIDIA Fermi/Kepler GPUs and AMD multicore systems are presented.