Thomas M. Evans

Denovo: A New Three-Dimensional Parallel Discrete Ordinates Code in SCALE

Abstract Denovo is a new, three-dimensional, discrete ordinates (SN) transport code that uses state-of-the-art solution methods to obtain accurate solutions to the Boltzmann transport equation. Denovo uses the Koch-Baker-Alcouffe parallel sweep algorithm to obtain high parallel efficiency on O(100) processors on XYZ orthogonal meshes. As opposed to traditional SN codes that use source iteration, Denovo uses nonstationary Krylov methods to solve the within-group equations. Krylov methods are far more efficient than stationary schemes. Additionally, classic acceleration schemes (diffusion synthetic acceleration) do not suffer stability problems when used as a preconditioner to a Krylov solver. Denovo’s generic programming framework allows multiple spatial discretization schemes and solution methodologies. Denovo currently provides diamond-difference, theta-weighted diamond-difference, linear-discontinuous finite element, trilinear-discontinuous finite element, and step characteristics spatial differencing schemes. Also, users have the option of running traditional source iteration instead of Krylov iteration. Multigroup upscatter problems can be solved using Gauss-Seidel iteration with transport, two-grid acceleration. A parallel first-collision source is also available. Denovo solutions to the Kobayashi benchmarks are in excellent agreement with published results. Parallel performance shows excellent weak scaling up to 20000 cores and good scaling up to 40000 cores.

SIMULTANEOUS OPTIMIZATION OF TALLIES IN DIFFICULT SHIELDING PROBLEMS

Abstract Monte Carlo is quite useful for calculating specific quantities in complex transport problems. Many variance reduction strategies have been developed that accelerate Monte Carlo calculations for specific tallies. However, when trying to calculate multiple tallies or a mesh tally, users have had to accept different levels of relative uncertainty among the tallies or run separate calculations optimized for each individual tally. To address this limitation, an extension of the Consistent Adjoint Driven Importance Sampling (CADIS) method, which is used for difficult source/detector problems, has been developed to optimize several tallies or the cells of a mesh tally simultaneously. The basis for this method is the development of an importance function that represents the importance of particles to the objective of uniform Monte Carlo particle density in the desired tally regions. This method utilizes the results of a forward discrete ordinates solution, which may be based on a quick coarse-mesh calculation, to develop a forward-weighted source for the adjoint calculation. The importance map and the biased source computed from the adjoint flux are then used in the forward Monte Carlo calculation to obtain approximately uniform relative uncertainties for the desired tallies. This extension is called forward-weighted CADIS, or FW-CADIS.

ADVANTG An Automated Variance Reduction Parameter Generator

The primary objective of ADVANTG is to reduce both the user effort and the computational time required to obtain accurate and precise tally estimates across a broad range of challenging transport applications. ADVANTG has been applied to simulations of real-world radiation shielding, detection, and neutron activation problems. Examples of shielding applications include material damage and dose rate analyses of the Oak Ridge National Laboratory (ORNL) Spallation Neutron Source and High Flux Isotope Reactor (Risner and Blakeman 2013) and the ITER Tokamak (Ibrahim et al. 2011). ADVANTG has been applied to a suite of radiation detection, safeguards, and special nuclear material movement detection test problems (Shaver et al. 2011). ADVANTG has also been used in the prediction of activation rates within light water reactor facilities (Pantelias and Mosher 2013). In these projects, ADVANTG was demonstrated to significantly increase the tally figure of merit (FOM) relative to an analog MCNP simulation. The ADVANTG-generated parameters were also shown to be more effective than manually generated geometry splitting parameters.

Massively Parallel, Three-Dimensional Transport Solutions for the k-Eigenvalue Problem

Abstract We have implemented a new multilevel parallel decomposition in the Denovo discrete ordinates radiation transport code. In concert with Krylov subspace iterative solvers, the multilevel decomposition allows concurrency over energy in addition to space-angle, enabling scalability beyond the limits imposed by the traditional Koch-Baker-Alcouffe (KBA) space-angle partitioning. Furthermore, a new Arnoldi-based k-eigenvalue solver has been implemented. The added phase-space concurrency combined with the high-performance Krylov and Arnoldi solvers has enabled weak scaling to O(105) cores on the Titan XK7 supercomputer. The multilevel decomposition provides a mechanism for scaling to exascale computing and beyond.

AUTOMATED VARIANCE REDUCTION APPLIED TO NUCLEAR WELL-LOGGING PROBLEMS

Abstract Simulating nuclear well-logging devices with Monte Carlo methods is computationally challenging and requires significant variance reduction to compute detector responses with low statistical uncertainties in reasonable lengths of time. The consistent adjoint-driven importance sampling (CADIS) method, which provides consistent source and transport biasing parameters based on a deterministic adjoint (importance) function, has been demonstrated to be very effective for well-logging simulations and other deep-penetration problems. A recent extension to the CADIS method, FW-CADIS (forward-weighted CADIS), is designed to optimize the calculation of several tallies at once by using an adjoint function based on an adjoint source weighted by the inverse of the forward flux. These advanced variance reduction methods have been incorporated and automated into the MAVRIC sequence of SCALE, making them very easy to use. The CADIS and FW-CADIS methods are demonstrated and compared on simple benchmark models of both neutron- and photon-based well-logging devices. Both advanced variance reduction methods offer a substantial reduction in computing time, compared to analog simulation, for these applications.

Multigrid in energy preconditioner for Krylov solvers

We have added a new multigrid in energy (MGE) preconditioner to the Denovo discrete-ordinates radiation transport code. This preconditioner takes advantage of a new multilevel parallel decomposition. A multigroup Krylov subspace iterative solver that is decomposed in energy as well as space-angle forms the backbone of the transport solves in Denovo. The space-angle-energy decomposition facilitates scaling to hundreds of thousands of cores. The multigrid in energy preconditioner scales well in the energy dimension and significantly reduces the number of Krylov iterations required for convergence. This preconditioner is well-suited for use with advanced eigenvalue solvers such as Rayleigh Quotient Iteration and Arnoldi.

High performance radiation transport simulations: preparing for Titan

In this paper we describe the Denovo code system. Denovo solves the six-dimensional, steady-state, linear Boltzmann transport equation, of central importance to nuclear technology applications such as reactor core analysis (neutronics), radiation shielding, nuclear forensics and radiation detection. The code features multiple spatial differencing schemes, state-of-the-art linear solvers, the Koch-Baker-Alcouffe (KBA) parallel-wavefront sweep algorithm for inverting the transport operator, a new multilevel energy decomposition method scaling to hundreds of thousands of processing cores, and a modern, novel code architecture that supports straightforward integration of new features. In this paper we discuss the performance of Denovo on the 20+ petaflop ORNL GPU-based system, Titan. We describe algorithms and techniques used to exploit the capabilities of Titans heterogeneous compute node architecture and the challenges of obtaining good parallel performance for this sparse hyperbolic PDE solver containing inherently sequential computations. Numerical results demonstrating Denovo performance on early Titan hardware are presented.

Full core reactor analysis: Running Denovo on Jaguar

Abstract Fully consistent, full core, three-dimensional, deterministic neutron transport simulations using the orthogonal mesh code Denovo were run on the massively parallel computing architecture Jaguar XT5. Using energy and spatial parallelization schemes, Denovo was able to efficiently scale to more than 160 000 processors. Cell-homogenized cross sections were used with step characteristics, linear discontinuous finite element, and trilinear discontinuous finite element spatial methods. It was determined that using the finite element methods gave considerably more accurate eigenvalue solutions for large–aspect ratio meshes than using step characteristics.

ITER Neutronics Modeling Using Hybrid Monte Carlo/Deterministic and CAD-Based Monte Carlo Methods

Abstract The immense size and complex geometry of the ITER experimental fusion reactor require the development of special techniques that can accurately and efficiently perform neutronics simulations with minimal human effort. This paper shows the effect of the hybrid Monte Carlo (MC)/deterministic techniques—Consistent Adjoint Driven Importance Sampling (CADIS) and Forward-Weighted CADIS (FW-CADIS)—in enhancing the efficiency of the neutronics modeling of ITER and demonstrates the applicability of coupling these methods with computer-aided-design-based MC. Three quantities were calculated in this analysis: the total nuclear heating in the inboard leg of the toroidal field coils (TFCs), the prompt dose outside the biological shield, and the total neutron and gamma fluxes over a mesh tally covering the entire reactor. The use of FW-CADIS in estimating the nuclear heating in the inboard TFCs resulted in a factor of ˜275 increase in the MC figure of merit (FOM) compared with analog MC and a factor of ˜9 compared with the traditional methods of variance reduction. By providing a factor of ˜21 000 increase in the MC FOM, the radiation dose calculation showed how the CADIS method can be effectively used in the simulation of problems that are practically impossible using analog MC. The total flux calculation demonstrated the ability of FW-CADIS to simultaneously enhance the MC statistical precision throughout the entire ITER geometry. Collectively, these calculations demonstrate the ability of the hybrid techniques to accurately model very challenging shielding problems in reasonable execution times.

A Hybrid Transport-Diffusion Algorithm for Monte Carlo Radiation-Transport Simulations on Adaptive-Refinement Meshes in XY Geometry

Abstract Discrete Diffusion Monte Carlo (DDMC) is a technique for increasing the efficiency of Monte Carlo simulations in diffusive media. If standard Monte Carlo is employed in such a regime, particle histories will consist of many small steps, a situation that results in a computationally inefficient calculation. In DDMC, particles take discrete steps between spatial cells according to a discretized diffusion equation. Each discrete step replaces many smaller Monte Carlo steps, thus increasing the efficiency of the simulation. In addition, because DDMC is based on the diffusion approximation, it should yield accurate solutions if used judiciously. In this paper, we present a new DDMC method for linear, steady-state radiation transport on adaptive-refinement meshes in two-dimensional Cartesian geometry. Adaptive-refinement meshes are characterized by local refinement such that a spatial cell may have multiple neighboring cells across each face. We specifically examine the cases of (a) a regular mesh structure without refinement, (b) a refined mesh structure where neighboring cells differ in refinement, and (c) a boundary mesh structure representing the interface between a diffusive region (where DDMC is used) and a nondiffusive region (where standard Monte Carlo is employed). With numerical examples, we demonstrate that our new DDMC technique is accurate and can provide efficiency gains of two orders of magnitude over standard Monte Carlo.