Ramanan Sankaran

Terascale direct numerical simulations of turbulent combustion using S3D

Computational science is paramount to the understanding of underlying processes in internal combustion engines of the future that will utilize non-petroleum-based alternative fuels, including carbon-neutral biofuels, and burn in new combustion regimes that will attain high efficiency while minimizing emissions of particulates and nitrogen oxides. Next-generation engines will likely operate at higher pressures, with greater amounts of dilution and utilize alternative fuels that exhibit a wide range of chemical and physical properties. Therefore, there is a significant role for high-fidelity simulations, direct numerical simulations (DNS), specifically designed to capture key turbulence-chemistry interactions in these relatively uncharted combustion regimes, and in particular, that can discriminate the effects of differences in fuel properties. In DNS, all of the relevant turbulence and flame scales are resolved numerically using high-order accurate numerical algorithms. As a consequence terascale DNS are computationally intensive, require massive amounts of computing power and generate tens of terabytes of data. Recent results from terascale DNS of turbulent flames are presented here, illustrating its role in elucidating flame stabilization mechanisms in a lifted turbulent hydrogen/air jet flame in a hot air coflow, and the flame structure of a fuel-lean turbulent premixed jet flame. Computing at this scale requires close collaborations between computer and combustion scientists to provide optimized scaleable algorithms and software for terascale simulations, efficient collective parallel I/O, tools for volume visualization of multiscale, multivariate data and automating the combustion workflow. The enabling computer science, applied to combustion science, is also required in many other terascale physics and engineering simulations. In particular, performance monitoring is used to identify the performance of key kernels in the DNS code, S3D and especially memory intensive loops in the code. Through the careful application of loop transformations, data reuse in cache is exploited thereby reducing memory bandwidth needs, and hence, improving S3Ds nodal performance. To enhance collective parallel I/O in S3D, an MPI-I/O caching design is used to construct a two-stage write-behind method for improving the performance of write-only operations. The simulations generate tens of terabytes of data requiring analysis. Interactive exploration of the simulation data is enabled by multivariate time-varying volume visualization. The visualization highlights spatial and temporal correlations between multiple reactive scalar fields using an intuitive user interface based on parallel coordinates and time histogram. Finally, an automated combustion workflow is designed using Kepler to manage large-scale data movement, data morphing, and archival and to provide a graphical display of run-time diagnostics.

Turbulent flame–wall interaction: a direct numerical simulation study

A turbulent flame-wall interaction (FWI) configuration is studied using three-dimensional direct numerical simulation (DNS) and detailed chemical kinetics. The simulations are used to investigate the effects of the wall turbulent boundary layer (i) on the structure of a hydrogen-air premixed flame, (ii) on its near-wall propagation characteristics and (iii) on the spatial and temporal patterns of the convective wall heat flux. Results show that the local flame thickness and propagation speed vary between the core flow and the boundary layer, resulting in a regime change from flamelet near the channel centreline to a thickened flame at the wall. This finding has strong implications for the modelling of turbulent combustion using Reynolds-averaged Navier-Stokes or large-eddy simulation techniques. Moreover, the DNS results suggest that the near-wall coherent turbulent structures play an important role on the convective wall heat transfer by pushing the hot reactive zone towards the cold solid surface. At the wall, exothermic radical recombination reactions become important, and are responsible for approximately 70 % of the overall heat release rate at the wall. Spectral analysis of the convective wall heat flux provides an unambiguous picture of its spatial and temporal patterns, previously unobserved, that is directly related to the spatial and temporal characteristic scalings of the coherent near-wall turbulent structures.

Direct numerical simulation of turbulent combustion: fundamental insights towards predictive models

The advancement of our basic understanding of turbulent combustion processes and the development of physics-based predictive tools for design and optimization of the next generation of combustion devices are strategic areas of research for the development of a secure, environmentally sound energy infrastructure. In direct numerical simulation (DNS) approaches, all scales of the reacting flow problem are resolved. However, because of the magnitude of this task, DNS of practical high Reynolds number turbulent hydrocarbon flames is out of reach of even terascale computing. For the foreseeable future, the approach to this complex multi-scale problem is to employ distinct but synergistic approaches to tackle smaller sub-ranges of the complete problem, which then require models for the small scale interactions. With full access to the spatially and temporally resolved fields, DNS can play a major role in the development of these models and in the development of fundamental understanding of the micro-physics of turbulence-chemistry interactions. Two examples, from simulations performed at terascale Office of Science computing facilities, are presented to illustrate the role of DNS in delivering new insights to advance the predictive capability of models. Results are presented from new three-dimensional DNS with detailed chemistry of turbulent non-premixed jet flames, revealing the differences between mixing of passive and reacting scalars, and determining an optimal lower dimensional representation of the full thermochemical state space.

Early evaluation of IBM BlueGene/P

BlueGene/P (BG/P) is the second generation BlueGene architecture from IBM, succeeding BlueGene/L (BG/L). BG/P is a system-on-a-chip (SoC) design that uses four PowerPC 450 cores operating at 850 MHz with a double precision, dual pipe floating point unit per core. These chips are connected with multiple interconnection networks including a 3-D torus, a global collective network, and a global barrier network. The design is intended to provide a highly scalable, physically dense system with relatively low power requirements per flop. In this paper, we report on our examination of BG/P, presented in the context of a set of important scientific applications, and as compared to other major large scale supercomputers in use today. Our investigation confirms that BG/P has good scalability with an expected lower performance per processor when compared to the Cray XT4s Opteron. We also find that BG/P uses very low power per floating point operation for certain kernels, yet it has less of a power advantage when considering science driven metrics for mission applications.

Cray XT4: an early evaluation for petascale scientific simulation

The scientific simulation capabilities of next generation high-end computing technology will depend on striking a balance among memory, processor, I/O, and local and global network performance across the breadth of the scientific simulation space. The Cray XT4 combines commodity AMD dual core Opteron processor technology with the second generation of Crays custom communication accelerator in a system design whose balance is claimed to be driven by the demands of scientific simulation. This paper presents an evaluation of the Cray XT4 using micro-benchmarks to develop a controlled understanding of individual system components, providing the context for analyzing and comprehending the performance of several petascale-ready applications. Results gathered from several strategic application domains are compared with observations on the previous generation Cray XT3 and other high-end computing systems, demonstrating performance improvements across a wide variety of application benchmark problems.

Dynamic flammability limits of methane/air premixed flames with mixture composition fluctuations

As a fundamental study in the application to direct-injection spark-ignition engines or gas turbines, in which mixture stratification and partial quenching are of serious concerns, unsteady premixed methane/ air flames subjected to time-varying composition fluctuations are investigated computationally. The code OPUS employs an unsteady opposed-flow combustion configuration, including detailed chemical kinetics, transport, and radiation models, using an adaptive time integration method for a stiff system of differential-algebraic equations with a high index. The primary issue of the study is to establish the concept of the dynamic flammability limit , defined as the minimum equivalence ratio above which the unsteady flame can sustain combustion. For the weak and strong strain rate cases studied, it is observed that the dynamic flammability limit depends on the mean and frequency of the composition fluctuation. The parametric study demonstrated that the flammability limit of an unsteady premixed flame is further extended to a leaner condition as the frequency or mean equivalence ratio fluctuation increases. It is also found that, under all conditions, the mean equivalence ratio and the minimum flame temperature must be higher than those at the steady flammability limit to sustain combustion. It is further shown that the dynamic flammability limit is primarily determined by the instantaneous, branching-termination balance at the reaction zone. The behavior of the flame response attenuation with increasing frequency is found to scale properly using the normalized frequency based on the imposed flow strain rate, which represents the characteristic time scale of the transport process within the flame.

Hybridizing S3D into an exascale application using OpenACC: an approach for moving to multi-petaflops and beyond

Hybridization is the process of converting an application with a single level of parallelism to an application with multiple levels of parallelism. Over the past 15 years a majority of the applications that run on High Performance Computing systems have employed MPI for all of the parallelism within the application. In the Peta-Exascale computing regime, effective utilization of the hardware requires multiple levels of parallelism matched to the macro architecture of the system to achieve good performance. A hybridized code base is performance portable when sufficient parallelism is expressed in an architecture agnostic form to achieve good performance on a range of available systems. The hybridized S3D code is performance portable across todays leading many core and GPU accelerated systems. The OpenACC framework allows a unified code base to be deployed for either (Manycore CPU or Manycore CPU+GPU) while permitting architecture specific optimizations to expose new dimensions of parallelism to be utilized.

EFFECTS OF HYDROGEN ADDITION ON THE MARKSTEIN LENGTH AND FLAMMABILITY LIMIT OF STRETCHED METHANE/AIR PREMIXED FLAMES

A computational study is performed to investigate the effects of hydrogen addition on the fundamental characteristics of steady and unsteady stretched methane/air premixed flame in an opposed flow configuration. The problem is of interest as a potential application to gas turbines and sparkignition engines, where the addition of a small amount of hydrogen allows combustion at leaner conditions to achieve lower NOx emission. The flame response is first studied under steady conditions with different levels of hydrogen addition. The effective Markstein length is found to exhibit a nonmonotonic function of the level of blending due to the competing effects between the Zeldovich and Lewis number variations. The results also show that the lean flammability limit is significantly extended due to the presence of hydrogen in the mixture, consistent with previous studies. On the other hand, the consumption speed and time scale of the flame at the extinction condition are found to be rather insensitive to the extent of blending. Unsteady flame response is subsequently studied by imposing oscillatory equivalence ratio at the boundary, as a means to characterize the effects of mixture stratification at various time scales. Consistent with the steady results, the attenuation of the dynamic flammability limits in a normalized scale collapse very well for various levels of hydrogen blending, implying that the unsteady flame response depends strongly on the characteristic chemical time scale irrespective of the amount of fuel blending. A simple analytical solution is derived to predict consistent qualitative behavior. The net effects of unsteady composition fluctuation on the NOx formation are also discussed.

Direct numerical simulations of turbulent lean premixed combustion

In recent years, due to the advent of high-performance computers and advanced numerical algorithms, direct numerical simulation (DNS) of combustion has emerged as a valuable computational research tool, in concert with experimentation. The role of DNS in delivering new Scientific insight into turbulent combustion is illustrated using results from a recent 3D turbulent premixed flame simulation. To understand the influence of turbulence on the flame structure, a 3D fully-resolved DNS of a spatially-developing lean methane-air turbulent Bunsen flame was performed in the thin reaction zones regime. A reduced chemical model for methane-air chemistry consisting of 13 resolved species, 4 quasi-steady state species and 73 elementary reactions was developed specifically for the current simulation. The data is analyzed to study possible influences of turbulence on the flame thickness. The results show that the average flame thickness increases, in qualitative agreement with several experimental results.

Accelerating S3D: a GPGPU case study

The graphics processor (GPU) has evolved into an appealing choice for high performance computing due to its superior memory bandwidth, raw processing power, and flexible programmability. As such, GPUs represent an excellent platform for accelerating scientific applications. This paper explores a methodology for identifying applications which present significant potential for acceleration. In particular, this work focuses on experiences from accelerating S3D, a high-fidelity turbulent reacting flow solver. The acceleration process is examined from a holistic viewpoint, and includes details that arise from different phases of the conversion. This paper also addresses the issue of floating point accuracy and precision on the GPU, a topic of immense importance to scientific computing. Several performance experiments are conducted, and results are presented from the NVIDIA Tesla C1060 GPU. We generalize from our experiences to provide a roadmap for deploying existing scientific applications on heterogeneous GPU platforms.