Hadrien Calmet

Alya: Multiphysics engineering simulation toward exascale

Alya is a multi-physics simulation code developed at Barcelona Supercomputing Center (BSC). From its inception Alya code is designed using advanced High Performance Computing programming techniques to solve coupled problems on supercomputers efficiently. The target domain is engineering, with all its particular features: complex geometries and unstructured meshes, coupled multi-physics with exotic coupling schemes and physical models, ill-posed problems, flexibility needs for rapidly including new models, etc. Since its beginnings in 2004, Alya has scaled well in an increasing number of processors when solving single-physics problems such as fluid mechanics, solid mechanics, acoustics, etc. Over time, we have made a concerted effort to maintain and even improve scalability for multi-physics problems. This poses challenges on multiple fronts, including: numerical models, parallel implementation, physical coupling models, algorithms and solution schemes, meshing process, etc. In this paper, we introduce Alyas main features and focus particularly on its solvers. We present Alyas performance up to 100.000 processors in Blue Waters, the NCSA supercomputer with selected multi-physics tests that are representative of the engineering world. The tests are incompressible flow in a human respiratory system, low Mach combustion problem in a kiln furnace, and coupled electro-mechanical contraction of the heart. We show scalability plots for all cases and discuss all aspects of such simulations, including solver convergence.

Large-scale CFD simulations of the transitional and turbulent regime for the large human airways during rapid inhalation

The dynamics of unsteady flow in the human large airways during a rapid inhalation were investigated using highly detailed large-scale computational fluid dynamics on a subject-specific geometry. The simulations were performed to resolve all the spatial and temporal scales of the flow, thanks to the use of massive computational resources. A highly parallel finite element code was used, running on two supercomputers, solving the transient incompressible Navier-Stokes equations on unstructured meshes. Given that the finest mesh contained 350 million elements, the study sets a precedent for large-scale simulations of the respiratory system, proposing an analysis strategy for mean flow, fluctuations and wall shear stresses on a rapid and short inhalation (a so-called sniff). The geometry used encompasses the exterior face and the airways from the nasal cavity, through the trachea and up to the third lung bifurcation; it was derived from a contrast-enhanced computed tomography (CT) scan of a 48-year-old male. The transient inflow produces complex flows over a wide range of Reynolds numbers (Re). Thanks to the high fidelity simulations, many features involving the flow transition were observed, with the level of turbulence clearly higher in the throat than in the nose. Spectral analysis revealed turbulent characteristics persisting downstream of the glottis, and were captured even with a medium mesh resolution. However a fine mesh resolution was found necessary in the nasal cavity to observe transitional features. This work indicates the potential of large-scale simulations to further understanding of airway physiological mechanics, which is essential to guide clinical diagnosis; better understanding of the flow also has implications for the design of interventions such as aerosol drug delivery.

Dynamics of airflow in a short inhalation.

During a rapid inhalation, such as a sniff, the flow in the airways accelerates and decays quickly. The consequences for flow development and convective transport of an inhaled gas were investigated in a subject geometry extending from the nose to the bronchi. The progress of flow transition and the advance of an inhaled non-absorbed gas were determined using highly resolved simulations of a sniff 0.5 s long, 1 l s−1 peak flow, 364 ml inhaled volume. In the nose, the distribution of airflow evolved through three phases: (i) an initial transient of about 50 ms, roughly the filling time for a nasal volume, (ii) quasi-equilibrium over the majority of the inhalation, and (iii) a terminating phase. Flow transition commenced in the supraglottic region within 20 ms, resulting in large-amplitude fluctuations persisting throughout the inhalation; in the nose, fluctuations that arose nearer peak flow were of much reduced intensity and diminished in the flow decay phase. Measures of gas concentration showed non-uniform build-up and wash-out of the inhaled gas in the nose. At the carina, the form of the temporal concentration profile reflected both shear dispersion and airway filling defects owing to recirculation regions.

Real-space density functional theory and time dependent density functional theory using finite/infinite element methods

Abstract We present a numerical approach using the finite element method to discretize the equations that allow getting a first-principles description of multi-electronic systems within DFT and TD-DFT formalisms. A strictly local polynomial function basis set is used in order to represent the entire real-space domain. Infinite elements are introduced to model the infinite external boundaries in the case of Hartree’s equation. The diagonal mass matrix is obtained using a close integration rule, reducing the generalized eigenvalue problem to a standard one. This framework of electronic structure calculation is embedded in a high performance computing environment with a very good parallel behavior.

Experience in Parallel Computational Mechanics on MareNostrum

We present in this paper the experience of the authors in solving very large problems of computational mechanics on a supercomputer. The authors are researchers of the Barcelona Supercomputing Center-Centro Nacional de Supercomputacion (BSC-CNS), a brand new research center in Spain, which hosts the fastest supercomputer in Europe and the fifth in the world: MareNostrum (TOP500, November 2006). A brief presentation of MareNostrum is given in the first section. In the next section, we describe the physical problems we are faced with, followed by the section on the associated numerical strategies. Next section presents the parallelization strategy employed; emphasis will be put on the important aspects to obtain good speed-up results on thousands of processors. We follow the discussion by describing some applications together with speed-up results. We finalize the paper by describing the performance tool used at BSC-CNS and their application in computational mechanics.