Oana Marin

Characterization of the secondary flow in hexagonal ducts

In this work we report the results of DNSs and LESs of the turbulent flow through hexagonal ducts at friction Reynolds numbers based on centerplane wall shear and duct half-height Reτ,c ≃ 180, 360, and 550. The evolution of the Fanning friction factor f with Re is in very good agreement with experimental measurements. A significant disagreement between the DNS and previous RANS simulations was found in the prediction of the in-plane velocity, and is explained through the inability of the RANS model to properly reproduce the secondary flow present in the hexagon. The kinetic energy of the secondary flow integrated over the cross-sectional area 〈K〉yz decreases with Re in the hexagon, whereas it remains constant with Re in square ducts at comparable Reynolds numbers. Close connection between the values of Reynolds stress uw¯ on the horizontal wall close to the corner and the interaction of bursting events between the horizontal and inclined walls is found. This interaction leads to the formation of the secon...

On the Strong Scaling of the Spectral Element Solver Nek5000 on Petascale Systems

The present work is targeted at performing a strong scaling study of the high-order spectral element fluid dynamics solver Nek5000. Prior studies such as [5] indicated a recommendable metric for strong scalability from a theoretical viewpoint, which we test here extensively on three parallel machines with different performance characteristics and interconnect networks, namely Mira (IBM Blue Gene/Q), Beskow (Cray XC40) and Titan (Cray XK7). The test cases considered for the simulations correspond to a turbulent flow in a straight pipe at four different friction Reynolds numbers Reτ = 180, 360, 550 and 1000. Considering the linear model for parallel communication we quantify the machine characteristics in order to better assess the scaling behaviors of the code. Subsequently sampling and profiling tools are used to measure the computation and communication times over a large range of compute cores. We also study the effect of the two coarse grid solvers XXT and AMG on the computational time. Super-linear scaling due to a reduction in cache misses is observed on each computer. The strong scaling limit is attained for roughly 5000-10,000 degrees of freedom per core on Mira, 30,000 - 50,0000 on Beskow, with only a small impact of the problem size for both machines, and ranges between 10,000 and 220,000 depending on the problem size on Titan. This work aims at being a reference for Nek5000 users and also serves as a basis for potential issues to address as the community heads towards exascale supercomputers.

Turbulence statistics in a spectral element code: a toolbox for High-Fidelity Simulations

In the present document we describe a toolbox for the spectral-element code Nek5000 Fischer et al. (2008), aimed at computing turbulence statistics. The toolbox is presented for a small test case, namely a periodic hill with Lx = 9h, Ly = 3.035h and Lz = 4.5h, where x, y and z are the horizontal, vertical and spanwise directions, respectively, and h is the hill height. The number of elements in the xy−plane is 442, and the number of elements in z is 19, leading to a total of 8,398 spectral elements. A polynomial order of N = 5 is chosen, and the mesh is generated using ICEM-CFD. In this case, transition is triggered by tripping the boundary layers on both surfaces by means of a random-volume force implemented in the subroutine userf in per hill.usr. The toolbox presented here allows to compute mean-velocity components, the Reynoldsstress tensor as well as turbulent kinetic energy (TKE) and Reynolds-stress budgets. Note that the present toolbox allows to compute turbulence statistics in turbulent flows with one homogeneous direction (where the statistics are based on time-averaging and averaging in the homogeneous direction), and also in fully three-dimensional flows (with no periodic directions, where only time-averaging is considered). The structure of the StatsToolbox folder is as follows:

Asynchronous Two-level Checkpointing Scheme for Large-scale Adjoints in the Spectral-Element Solver Nek5000

Abstract Adjoints are an important computational tool for large-scale sensitivity evaluation, uncertainty quantification, and derivative-based optimization. An essential component of their performance is the storage/recomputation balance in which efficient adjoint checkpointing strategies play a key role. We introduce a novel asynchronous two-level adjoint checkpointing scheme for numerical time discretizations targeted at large-scale numerical simulations. The checkpointing scheme combines bandwidth-limited disk checkpointing and space-limited binomial memory checkpointing. Based on assumptions about the target petascale systems, which we later demonstrate to be realistic on the IBM Blue Gene/Q system Mira, we create a model of the predicted performance of the adjoint computation and validate it using the highly scalable Navier-Stokes spectral-element solver Nek5000 on small to moderate subsystems of the Mira supercomputer.

The three-dimensional structure of swirl-switching in bent pipe flow

Swirl-switching is a low-frequency oscillatory phenomenon which affects the Dean vortices in bent pipes and may cause fatigue in piping systems. Despite thirty years worth of research, the mechanis ...

Transitional and Turbulent Bent Pipes

We review a number of aspects of the transitional and turbulent flow in bent pipes, obtained at KTH using the spectral-element code Nek5000. This flow, sometimes also called Dean flow, is characterised by the appearance of Dean vortices, which arise due to the action of the centrifugal force in the bend. We start with reviewing recent stability analysis in the toroidal flow, and conclude that for all curvatures \(\delta >0\) an exponential instability is present at a bulk Reynolds number of about 4000. Further increasing the Reynolds number lets the flow go through a region with potential sub straight and sublaminar drag. An analysis using proper orthogonal decomposition (POD) reveals that wave-like motions are still present in the otherwise turbulent flow. Upon further increasing Re, the in-plane Dean vortices lead to a modulation of turbulence depending on the azimuthal position. The flow is then dominated by low-frequency so-called swirl-switching motion. This motion is studied in both a periodic and spatially developing framework. Finally, the effect of Dean vortices on Lagrangian inertial particles is studied.