Duane Rosenberg

A hybrid MPI–OpenMP scheme for scalable parallel pseudospectral computations for fluid turbulence

Abstract A hybrid scheme that utilizes MPI for distributed memory parallelism and OpenMP for shared memory parallelism is presented. The work is motivated by the desire to achieve exceptionally high Reynolds numbers in pseudospectral computations of fluid turbulence on emerging petascale, high core-count, massively parallel processing systems. The hybrid implementation derives from and augments a well-tested scalable MPI-parallelized pseudospectral code. The hybrid paradigm leads to a new picture for the domain decomposition of the pseudospectral grids, which is helpful in understanding, among other things, the 3D transpose of the global data that is necessary for the parallel fast Fourier transforms that are the central component of the numerical discretizations. Details of the hybrid implementation are provided, and performance tests illustrate the utility of the method. It is shown that the hybrid scheme achieves good scalability up to ∼20,000 compute cores with a maximum efficiency of 89%, and a mean of 79%. Data are presented that help guide the choice of the optimal number of MPI tasks and OpenMP threads in order to maximize code performance on two different platforms.

Lack of universality in decaying magnetohydrodynamic turbulence.

Using computations of three-dimensional magnetohydrodynamic (MHD) turbulence with a Taylor-Green flow, whose inherent time-independent symmetries are implemented numerically, and in the absence of either a forcing function or an imposed uniform magnetic field, we show that three different inertial ranges for the energy spectrum may emerge for three different initial magnetic fields, the selecting parameter being the ratio of nonlinear eddy to Alfvén time. Equivalent computational grids range from 128(3) to 2048(3) points with a unit magnetic Prandtl number and a Taylor Reynolds number of up to 1500 at the peak of dissipation. We also show a convergence of our results with Reynolds number. Our study is consistent with previous findings of a variety of energy spectra in MHD turbulence by studies performed in the presence of both a forcing term with a given correlation time and a strong, uniform magnetic field. However, in contrast to the previous studies, here the ratio of characteristic time scales can only be ascribed to the intrinsic nonlinear dynamics of the paradigmatic flows under study.

Geophysical-astrophysical spectral-element adaptive refinement (GASpAR): Object-oriented h-adaptive fluid dynamics simulation

We present an object-oriented geophysical and astrophysical spectral-element adaptive refinement (GASpAR) code for application to turbulent flows. Like most spectralelement codes, GASpAR combines finite-element efficiency with spectral-method accuracy. It is also designed to be flexible enough for a range of geophysics and astrophysics applications where turbulence or other complex multiscale problems arise. For extensibility and flexibilty the code is designed in an object-oriented manner. The computational core is based on spectral-element operators, which are represented as objects. The formalism accommodates both conforming and non

Anisotropy and nonuniversality in scaling laws of the large-scale energy spectrum in rotating turbulence.

Rapidly rotating turbulent flow is characterized by the emergence of columnar structures that are representative of quasi-two-dimensional behavior of the flow. It is known that when energy is injected into the fluid at an intermediate scale Lf, it cascades towards smaller as well as larger scales. In this paper we analyze the flow in the inverse cascade range at a small but fixed Rossby number, Rof≈0.05. Several numerical simulations with helical and nonhelical forcing functions are considered in periodic boxes with unit aspect ratio. In order to resolve the inverse cascade range with reasonably large Reynolds number, the analysis is based on large eddy simulations which include the effect of helicity on eddy viscosity and eddy noise. Thus, we model the small scales and resolve explicitly the large scales. We show that the large-scale energy spectrum has at least two solutions: one that is consistent with Kolmogorov-Kraichnan-Batchelor-Leith phenomenology for the inverse cascade of energy in two-dimensional (2D) turbulence with a ∼k⊥-5/3 scaling, and the other that corresponds to a steeper ∼k⊥-3 spectrum in which the three-dimensional (3D) modes release a substantial fraction of their energy per unit time to the 2D modes. The spectrum that emerges depends on the anisotropy of the forcing function, the former solution prevailing for forcings in which more energy is injected into the 2D modes while the latter prevails for isotropic forcing. In the case of anisotropic forcing, whence the energy goes from the 2D to the 3D modes at low wave numbers, large-scale shear is created, resulting in a time scale τsh, associated with shear, thereby producing a ∼k-1 spectrum for the total energy with the horizontal energy of the 2D modes still following a ∼k⊥-5/3 scaling.

Ideal evolution of magnetohydrodynamic turbulence when imposing Taylor-Green symmetries

We investigate the ideal and incompressible magnetohydrodynamic (MHD) equations in three space dimensions for the development of potentially singular structures. The methodology consists in implementing the fourfold symmetries of the Taylor-Green vortex generalized to MHD, leading to substantial computer time and memory savings at a given resolution; we also use a regridding method that allows for lower-resolution runs at early times, with no loss of spectral accuracy. One magnetic configuration is examined at an equivalent resolution of 6144(3) points and three different configurations on grids of 4096(3) points. At the highest resolution, two different current and vorticity sheet systems are found to collide, producing two successive accelerations in the development of small scales. At the latest time, a convergence of magnetic field lines to the location of maximum current is probably leading locally to a strong bending and directional variability of such lines. A novel analytical method, based on sharp analysis inequalities, is used to assess the validity of the finite-time singularity scenario. This method allows one to rule out spurious singularities by evaluating the rate at which the logarithmic decrement of the analyticity-strip method goes to zero. The result is that the finite-time singularity scenario cannot be ruled out, and the singularity time could be somewhere between t=2.33 and t=2.70. More robust conclusions will require higher resolution runs and grid-point interpolation measurements of maximum current and vorticity.

Large-scale anisotropy in stably stratified rotating flows

We present results from direct numerical simulations of the Boussinesq equations in the presence of rotation and/or stratification, both in the vertical direction. The runs are forced isotropically and randomly at small scales and have spatial resolutions of up to 1024(3) grid points and Reynolds numbers of ≈1000. We first show that solutions with negative energy flux and inverse cascades develop in rotating turbulence, whether or not stratification is present. However, the purely stratified case is characterized instead by an early-time, highly anisotropic transfer to large scales with almost zero net isotropic energy flux. This is consistent with previous studies that observed the development of vertically sheared horizontal winds, although only at substantially later times. However, and unlike previous works, when sufficient scale separation is allowed between the forcing scale and the domain size, the kinetic energy displays a perpendicular (horizontal) spectrum with power-law behavior compatible with ∼k(⊥)(-5/3), including in the absence of rotation. In this latter purely stratified case, such a spectrum is the result of a direct cascade of the energy contained in the large-scale horizontal wind, as is evidenced by a strong positive flux of energy in the parallel direction at all scales including the largest resolved scales.

Interplay of waves and eddies in rotating stratified turbulence and the link with kinetic-potential energy partition

The interplay between waves and eddies in stably stratified rotating flows is investigated by means of world-class direct numerical simulations using up to

Helical Turbulence Prevails over Inertial Waves in Forced Rotating Flows at High Reynolds and Low Rossby Numbers

3072^3

Emergence of helicity in rotating stratified turbulence

grid points. Strikingly, we find that the shift from vortex to wave dominated dynamics occurs at a wavenumber

Adaptive mesh refinement with spectral accuracy for magnetohydrodynamics in two space dimensions

k_R