Stefan P. Domino

A Turbulence Model for Buoyant Flows Based on Vorticity Generation

A turbulence model for buoyant flows has been developed in the context of a k-{var_epsilon} turbulence modeling approach. A production term is added to the turbulent kinetic energy equation based on dimensional reasoning using an appropriate time scale for buoyancy-induced turbulence taken from the vorticity conservation equation. The resulting turbulence model is calibrated against far field helium-air spread rate data, and validated with near source, strongly buoyant helium plume data sets. This model is more numerically stable and gives better predictions over a much broader range of mesh densities than the standard k-{var_epsilon} model for these strongly buoyant flows.

TOWARDS EXTREME-SCALE SIMULATIONS FOR LOW MACH FLUIDS WITH SECOND-GENERATION TRILINOS

Trilinos is an object-oriented software framework for the solution of large-scale, complex multi-physics engineering and scientific problems. While Trilinos was originally designed for scalable solutions of large problems, the fidelity needed by many simulations is significantly greater than what one could have envisioned two decades ago. When problem sizes exceed a billion elements even scalable applications and solver stacks require a complete revision. The second-generation Trilinos employs C++ templates in order to solve arbitrarily large problems. We present a case study of the integration of Trilinos with a low Mach fluids engineering application (SIERRA low Mach module/Nalu). Through the use of improved algorithms and better software engineering practices, we demonstrate good weak scaling for up to a nine billion element large eddy simulation (LES) problem on unstructured meshes with a 27 billion row matrix on 524,288 cores of an IBM Blue Gene/Q platform.

Towards Extreme-Scale Simulations with Next-Generation Trilinos: A Low Mach Fluid Application Case Study

Trilinos is an object-oriented software framework for the solution of large-scale, complex multi-physics engineering and scientific problems. While the original version of Trilinos was designed for highly scalable solutions for large problems, the need for increasingly higher fidelity simulations has pushed the problem sizes beyond what could have been envisioned two decades ago. When problem sizes exceed a billion elements even highly scalable applications and solver stacks require a complete revision. The next-generation Trilinos employs C++ templates in order to solve arbitrarily large problems and enable extreme-scale simulations. We present a case study that involves integration of Trilinos with an engineering application (Sierra low Mach module/Nalu), involving the simulation of low Mach fluid flow for problems of size up to nine billion elements. Through the use of improved algorithms and better software engineering practices, we demonstrate good weak scaling for the matrix assembly and solve for the engineering application for up to a nine billion element fluid flow large eddy simulation (LES) problem on unstructured meshes with a 27 billion row matrix on 131,072 cores of a Cray XE6 platform.

State space sensitivity to a prescribed probability density function shape in coal combustion systems: Joint β-PDF versus clipped Gaussian PDF

The turbulent transport of three coal off-gas mixture fractions is coupled to a prescribed joint β -probability density function ( β -PDF) mixing model. This physical transport and subgrid joint β -PDF mixing model is used to explore the incorporation of coal off-gas compositional disparities between the devolatilization and the char oxidation regime in detailed pulverized-coal combustion simulations. A simulation study of the University of Utah pulverized-coal research furnace is presented to evaluate the sensitivity of different mixing model assumptions. These simulation studies indicate that using a variable composition to characterize the process of coal combustion does not appreciably change the predicted gas-phase temperature field. Moreover, neglecting fluctuations in the char off-gas stream was found to change gas-phase temperature predictions by approximately 15%. State space variable sensitivity to the assumed shape of the PDF (clipped Gaussian vs. joint β ) is presented. Simulation results indicate differences in temperature profiles of as much as 20% depending on the chosen shape of the PDF. Integration accuracy issues for the joint β -PDF are presented and are found to be acceptable. A robust β -PDF function evaluation procedure is presented that accommodates arbitrarily high β -PDF distribution factors. This robust algorithm simply transforms the joint β -PDF function evaluation into a logarithmic form. The assumption that a joint PDF, as rigorously required within a prescribed subgrid mixing model, can be written as the product of N − 1 statistically independent probability density functions is quantified and shown to be less accurate.

Validation and uncertainty quantification of Fuego simulations of calorimeter heating in a wind-driven hydrocarbon pool fire.

The objective of this work is to perform an uncertainty quantification (UQ) and model validation analysis of simulations of tests in the cross-wind test facility (XTF) at Sandia National Laboratories. In these tests, a calorimeter was subjected to a fire and the thermal response was measured via thermocouples. The UQ and validation analysis pertains to the experimental and predicted thermal response of the calorimeter. The calculations were performed using Sierra/Fuego/Syrinx/Calore, an Advanced Simulation and Computing (ASC) code capable of predicting object thermal response to a fire environment. Based on the validation results at eight diversely representative TC locations on the calorimeter the predicted calorimeter temperatures effectively bound the experimental temperatures. This post-validates Sandias first integrated use of fire modeling with thermal response modeling and associated uncertainty estimates in an abnormal-thermal QMU analysis.

Supercomputer and Cluster Performance Modeling and Analysis Efforts: 2004-2006

This report describes efforts by the Performance Modeling and Analysis Team to investigate performance characteristics of Sandias engineering and scientific applications on the ASC capability and advanced architecture supercomputers, and Sandias capacity Linux clusters. Efforts to model various aspects of these computers are also discussed. The goals of these efforts are to quantify and compare Sandias supercomputer and cluster performance characteristics; to reveal strengths and weaknesses in such systems; and to predict performance characteristics of, and provide guidelines for, future acquisitions and follow-on systems. Described herein are the results obtained from running benchmarks and applications to extract performance characteristics and comparisons, as well as modeling efforts, obtained during the time period 2004-2006. The format of the report, with hypertext links to numerous additional documents, purposefully minimizes the document size needed to disseminate the extensive results from our research.

Multiscale models of nuclear waste reprocessing : from the mesoscale to the plant-scale.

Nuclear waste reprocessing and nonproliferation models are needed to support the renaissance in nuclear energy. This report summarizes an LDRD project to develop predictive capabilities to aid the next-generation nuclear fuel reprocessing, in SIERRA Mechanics, Sandia’s high performance computing multiphysics code suite and Cantera, an open source software product for thermodynamics and kinetic modeling. Much of the focus of the project has been to develop a moving conformal decomposition finite element method (CDFEM) method applicable to mass transport at the water/oil droplet interface that occurs in the turbulent emulsion of droplets within the contactor. Contactor-scale models were developed using SIERRA Mechanics turbulence modeling capability. Unit operations occur at the column-scale where many contactors are connected in series. Population balance models

Highly Scalable Linear Solvers on Thousands of Processors

In this report we summarize research into new parallel algebraic multigrid (AMG) methods. We first provide a introduction to parallel AMG. We then discuss our research in parallel AMG algorithms for very large scale platforms. We detail significant improvements in the AMG setup phase to a matrix-matrix multiplication kernel. We present a smoothed aggregation AMG algorithm with fewer communication synchronization points, and discuss its links to domain decomposition methods. Finally, we discuss a multigrid smoothing technique that utilizes two message passing layers for use on multicore processors.

Design-order, non-conformal low-Mach fluid algorithms using a hybrid CVFEM/DG approach

Abstract A hybrid, design-order sliding mesh algorithm, which uses a control volume finite element method (CVFEM), in conjunction with a discontinuous Galerkin (DG) approach at non-conformal interfaces, is outlined in the context of a low-Mach fluid dynamics equation set. This novel hybrid DG approach is also demonstrated to be compatible with a classic edge-based vertex centered (EBVC) scheme. For the CVFEM, element polynomial, P , promotion is used to extend the low-order P = 1 CVFEM method to higher-order, i.e., P = 2 . An equal-order low-Mach pressure-stabilized methodology, with emphasis on the non-conformal interface boundary condition, is presented. A fully implicit matrix solver approach that accounts for the full stencil connectivity across the non-conformal interface is employed. A complete suite of formal verification studies using the method of manufactured solutions (MMS) is performed to verify the order of accuracy of the underlying methodology. The chosen suite of analytical verification cases range from a simple steady diffusion system to a traveling viscous vortex across mixed-order non-conformal interfaces. Results from all verification studies demonstrate either second- or third-order spatial accuracy and, for transient solutions, second-order temporal accuracy. Significant accuracy gains in manufactured solution error norms are noted even with modest promotion of the underlying polynomial order. The paper also demonstrates the CVFEM/DG methodology on two production-like simulation cases that include an inner block subjected to solid rotation, i.e., each of the simulations include a sliding mesh, non-conformal interface. The first production case presented is a turbulent flow past a high-rate-of-rotation cube ( Re , 4000; RPM, 3600) on like and mixed-order polynomial interfaces. The final simulation case is a full-scale Vestas V27 225 kW wind turbine (tower and nacelle omitted) in which a hybrid topology, low-order mesh is used. Both production simulations provide confidence in the underlying capability and demonstrate the viability of this hybrid method for deployment towards high-fidelity wind energy validation and analysis.