Harald Klimach

Fluid-Acoustics Interaction on Massively Parallel Systems

To simulate fluid-acoustics interaction, we couple inviscid Euler equations in the near-field, which is relevant for noise generation, to linearized Euler equations in the far-field. This allows us to separate the critical scales and treat each domain with an individual discretization. Both fields are computed by the high-order discontinuous Galerkin solver Ateles, while we couple the solvers at the interface by the library preCICE. We discuss a detailed performance analysis of the coupled simulation on massively parallel systems. Furthermore, to show the full potential of our approach, we simulate a flow around a sphere.

Efficient Coupling of Fluid and Acoustic Interaction on Massive Parallel Systems

We present and compare two coupling approaches for direct aeroacoustic simulations. Direct aeroacoustic simulations pose a multi-scale problem, as the generation of sound in a flow field occurs at small spatial scales with high energy, while its propagation in the farfield has to be observed on a large spatial scale with only low energy. The challenge of different scales can be addressed by employing different numerical schemes in the individual spatial areas with an interaction between them on the surfaces. Two implementation strategies of this coupling approach are presented. The first coupling strategy employs a library that allows a wide range of different applications to be coupled with minimal changes to the individual solvers. Hence, this is a very flexible approach but limited access to information and therefore cope with loss of potential performance. Further this strategy involves the handling of multiple executables on today supercomputer. This multi-solver approach requires data interpolation at the coupling interface which introduce another numerical error. In contrast, the second approach is fully integrated within one numerical framework. Thereby the solvers are invoked as a library by the coupling application and only one single applications must be handled. Tethering high order solvers, fully access to the data implies that no additional data interpolation is required which promise better numerical results. This tight integration allows for the exploitation of knowledge about internal data structures and therefore yield performance benefits accompany with less flexibility. Both strategies will be compared with respect to numerical error due to data interpolation at the coupling interface as well as scalability and performance on modern supercomputer.

Implementation of the compact interpolation within the octree based Lattice Boltzmann solver Musubi

Abstract A sparse octree based parallel implementation of the Lattice Boltzmann method for non-uniform meshes is presented in this paper. To couple grids of different resolutions, a second order accurate, compact interpolation is employed and further extended into three dimensions. The compact interpolation requires only four source elements from the adjacent level for both, two and three dimensions, thus improves the computational efficiency. Moreover, a weight based domain decomposition algorithm and the level-wise arrangement of elements is explained in detail. The second order convergence for both velocity and strain rate are validated numerically in the Taylor–Green vortex test case. Additionally, the 3D flow around a sphere in the Reynolds number range of 100–1000 is investigated. Good agreement between simulated results and those from literature is observed, which provides further evidence for the accuracy of our method.

Highly Efficient Integrated Simulation of Electro-membrane Processes for Desalination of Sea Water

Electrodialysis is an efficient process for sea water desalination. In this process, sea water flows through the channel with a complex spacer structure that separates the ion exchange membranes. The multi-species lattice Boltzmann model for liquid mixture modeling has been chosen to study the transport phenomena of ionized liquids in the spacer filled channel. This model is implemented in the highly scalable simulation framework APES based on octree meshes. In this paper, the performance and scalability of our implementation on the Cray XE6 system Hermit in Stuttgart are presented. The performance analysis is performed on periodic cubic domains with different compilers and communication patterns. The scalability of our solver with spacer structure for single-fluid LBM and multi-species LBM with three species mixture are presented. The spacer structure is scaled from single spacer element to full length of laboratory experiment spacer.

Aeroacoustic Simulation of Flow Through Porous Media Based on Lattice Boltzmann Method

This work presents the simulation of a flow through a porous silencer on the parallel super computing system Hornet at the HLRS in Stuttgart. This engineering problem poses a challenging task due to the complexity given the presence of multiple scales in space and time. We highlight the computational requirements for this simulation and the need for large scale data processing. The simulation is performed using our flow solver Musubi, which is based on the Lattice Boltzmann Method. We explain the design features of Musubi and show how these allow to exploit large scale parallel systems with distributed memory efficiently. Performance and scalability of Musubi is evaluated on Hornet with up to 2048 nodes. Using an interpolation supplemented local mesh refinement technique enables the simultaneous flow simulation inside the micro-porous structure and the sound wave propagation in bulk space. Some preliminary simulation results with this approach is finally provided, showing sound generation and propagation in this direct aero-acoustic setup.

Coupled Simulation with Two Coupling Approaches on Parallel Systems

The reduction of noise is one of the challenging tasks in the field of engineering. The interaction between flow, structure, and an acoustic field involves multiple scales. Simulating the whole domain with one solver is not feasible and out of range on todays supercomputer. Since the involving physics appear on different scales, the effects can be spatially separated into different domains. The interaction between the domains is realised with coupling approaches via boundaries. Different interpolation methods at the coupling interfaces are reviewed in this paper. The methods include the Nearest-Neighbor Interpolation (first order), the Radial-Basis Function (second order) as well as the direct evaluation of the state representation at the requested points (arbitrary order). We show which interpolation method provides less error, when compared to the monolithic solution of the result. We present how the two coupling approaches preCICE and APESmate can be used. The coupling tool preCICE is based on a black box coupling, where just the point values at the surface of the coupling domains are known. In contrast APESmate has knowledge about the numerical schemes within the domain. Thus, preCICE needs to interpolate values, while APESmate can evaluate the high order polynomials of the underlying Discontinous Galerkin scheme. Hence, the preCICE approach is more generally applicable, while the APESmate approach is more efficient, especially in the context of high order schemes.

Vectorization of High-Order DG in Ateles for the NEC SX-ACE

In this chapter, we investigate the possibilities of deploying a high-order, modal, discontinuous Galerkin scheme on the SX-ACE. Our implementation Ateles is written in modern Fortran and requires the new sxf03 compiler from NEC. It is based on an unstructured mesh representation that necessitates indirect addressing, but allows for a large flexibility in the representation of geometries. However, the degrees of freedom within the elements are stored in a rigid, structured array. For sufficiently high-order approximations these data structures within the elements can be exploited for vectorization.

Partitioned Fluid–Structure–Acoustics Interaction on Distributed Data: Numerical Results and Visualization

We present a coupled simulation approach for fluid–structure–acoustic interactions (FSAI) as an example for strongly surface coupled multi-physics problems. In addition to the multi-physics character, FSAI feature multi-scale properties as a further challenge. In our partitioned approach, the problem is split into spatially separated subdomains interacting via coupling surfaces. Within each subdomain, scalable, single-physics solvers are used to solve the respective equation systems. The surface coupling between them is realized with the scalable open-source coupling tool preCICE described in the “Partitioned Fluid–Structure–Acoustics Interaction on Distributed Data: Coupling via preCICE”. We show how this approach enables the use of existing solvers and present the overall scaling behavior for a three-dimensional test case with a bending tower generating acoustic waves. We run this simulation with different solvers demonstrating the performance of various solvers and the flexibility of the partitioned approach with the coupling tool preCICE. An efficient and scalable in-situ visualization reducing the amount of data in place at the simulation processors before sending them over the network or to a file system completes the simulation environment.

APES on SX-ACE

We report on first experiences in deploying the APES framework on the NEC SX-ACE vector system. In APES there are two solvers available, implementing different numerical schemes. Musubi is a Lattice-Boltzmann solver that can be used to simulate incompressible flows. This numerical method is attractive as it allows the explicit computation for incompressible flows with good scalability and robust treatment of highly comlex geometries. The second solver, Ateles, implements a high-order Discontinuous Galerkin method and can be used to solve hyperbolic conservation laws, including linear equations like acoustics and non-linear equations like compressible Navier-Stokes. The NEC SX-ACE vector system offers a memory bandwidth to operation ratio of 1 Byte per floating point operation, which is an interesting deployment option for many numerical schemes. Though, there are a lot of experiences with earlier systems of the SX series, the latest installment comes with new features and an overhaul of the programming environment.

Dealing with Non-linear Terms in a Modal High-Order Discontinuous Galerkin Method

The Discontinuous Galerkin (DG) method utilizes a mesh of elements with local functions like traditional continuous finite element methods, together with a flux approximation between elements like finite volume methods. This combination yields a high locality of the overall scheme, especially for high-order representations within elements. Two local operations need to be mainly considered. One is the application of the mass matrix and the other is the stiffness matrix. With an appropriate orthogonal basis as choice for the local functions both operations can be computed with minimal complexity. In this contribution we are concerned with a DG implementation that makes use of a Legendre polynomial basis with an application to non-linear equation systems. For non-linear systems a complication is introduced by the scheme by the necessity to compute the non-linear flux operation, which generally can not be done in the optimal modal basis. Instead, a pointwise evaluation of the non-linear operations is usually performed. Combining the fast evaluation of the integrals in the modal scheme with the pointwise evaluation of the non-linear terms requires a transformation between these two. Many methods have been developed for a fast transformation from Legendre modes to nodal values [1]. However, most of those methods for fast polynomial transformations are designed for extremely high polynomial degrees in the range of several hundreds. In three-dimensional DG simulations the polynomial degree in each dimension is more limited, and we are looking for methods that are fast but suitable for polynomials in the range up to a maximal degree of one hundred. We discuss some approaches to the fast transformation, especially the method proposed by Alpert and Rokhlin [2], and compare our implementation of this method to a straight forward \(L_2\) projection. The implementation specifically addresses also the hybrid parallelism with MPI and OpenMP for the three-dimensional DG elements.