Jan Götz

WaLBerla: HPC software design for computational engineering simulations

Abstract WaLBerla (Widely applicable Lattice-Boltzmann from Erlangen) is a massively parallel software framework supporting a wide range of physical phenomena. This article describes the software designs realizing the major goal of the framework, a good balance between expandability and scalable, highly optimized, hardware-dependent, special purpose kernels. To demonstrate our designs, we discuss the coupling of our Lattice-Boltzmann fluid flow solver and a method for fluid structure interaction. Additionally, we show a software design for heterogeneous computations on GPU and CPU utilizing optimized kernels. Finally, we estimate the software quality of the framework on the basis of software quality factors.

Permeability of porous materials determined from the Euler characteristic

We study the permeability of quasi-two-dimensional porous structures of randomly placed overlapping monodisperse circular and elliptical grains. Measurements in microfluidic devices and lattice Boltzmann simulations demonstrate that the permeability is determined by the Euler characteristic of the conducting phase. We obtain an expression for the permeability that is independent of the percolation threshold and shows agreement with experimental and simulated data over a wide range of porosities. Our approach suggests that the permeability explicitly depends on the overlapping probability of grains rather than their shape.

Coupling multibody dynamics and computational fluid dynamics on 8192 processor cores

This paper describes a method for the fully resolved simulation of particle laden flows. For this purpose, we discuss the parallelization of large scale coupled fluid structure interaction with up to 37 million geometrically modeled moving objects incorporated in the flow. The simulation is performed using a 3D lattice Boltzmann solver for the fluid flow and a so-called rigid body physics engine for the treatment of the objects. The numerical algorithms and the parallelization are discussed in detail. Furthermore, performance results are presented for test cases on up to 8192 processor cores running on an SGI Altix supercomputer. The approach enables a detailed simulation of large scale particulate flows that are relevant for many industrial applications.

Direct Numerical Simulation of Particulate Flows on 294912 Processor Cores

This paper describes computational models for particle-laden flows based on a fully resolved fluid-structure interaction. The flow simulation uses the Lattice Boltzmann method, while the particles are handled by a rigid body dynamics algorithm. The particles can have individual non-spherical shapes, creating the need for a non-trivial collision detection and special contact models. An explicit coupling algorithm transfers momenta from the fluid to the particles in each time step, while the particles impose moving boundaries for the flow solver. All algorithms and their interaction are fully parallelized. Scaling experiments and a careful performance analysis are presented for up to 294912 processor cores of the Blue Gene at the Jülich Supercomputing center. The largest simulations involve 264 million particles that are coupled to a fluid which is simultaneously resolved by 150 billion cells for the Lattice Boltzmann method. The paper will conclude with a computational experiment for the segregation of suspensions of particles of different density, as an example of the many industrial applications that are enabled by this new methodology.

WaLBerla: Exploiting Massively Parallel Systems for Lattice Boltzmann Simulations

In this chapter, a software concept for massively parallel computational fluid dynamics (CFD) applications is introduced. The focus thereby lies on the parallelization, which is based on a domain partitioning scheme named patch concept. This concept also enables a seamless specialization of the partitions to different application features as well as the possibility for further optimization such as memory reduction. It is discussed in detail how our design ensures an efficient and flexible implementation. The suitability and efficiency of this concept is demonstrated and evaluated with the waLBerla project, which aims at the development of an efficient massively parallel lattice Boltzmann framework providing the necessary features for several CFD applications. To discuss the suitability of the parallelization for massively parallel usage, various test scenarios have been investigated on different architectures. These tests include serial, weak and strong scaling experiments up to 810 cores and up to a domain size of 15303 lattice cells.

Fluid flow simulation on the Cell Broadband Engine using the lattice Boltzmann method

In this paper we present a fast lattice Boltzmann fluid solver that has been performance optimized and tailored for the Cell Broadband Engine Architecture. Many design decisions were motivated by the long range objective to simulate blood flow in human blood vessels, especially in aneurysms, but have proven to be much more generally applicable. After explaining implementation details and how they were influenced by the target platform, the performance and memory requirements of this prototype solver are evaluated.

All good things come in threes—Three beads learn to swim with lattice Boltzmann and a rigid body solver

We simulate the self-propulsion of devices in a fluid in the regime of low Reynolds numbers. Each device consists of three bodies (spheres or capsules) connected with two damped harmonic springs. Sinusoidal driving forces compress the springs which are resolved within a rigid body physics engine. The latter is consistently coupled to a 3D lattice Boltzmann framework for the fluid dynamics. In simulations of three-sphere devices, we find that the propulsion velocity agrees well with theoretical predictions. In simulations where some or all spheres are replaced by capsules, we find that the asymmetry of the design strongly affects the propelling efficiency.

Localized Parallel Algorithm for Bubble Coalescence in Free Surface Lattice-Boltzmann Method

The lattice Boltzmann method is a popular method from computational fluid dynamics. An extension of this method handling liquid flows with free surfaces can be used to simulate bubbly flows. It is based on a volume-of-fluids approach and an explicit tracking of the interface, including a reconstruction of the curvature to model surface tension. When this algorithm is parallelized, complicated data exchange is required, in particular when bubbles extend across several subdomains and when topological changes occur through coalescence of bubbles. In a previous implementation this was handled by using all-to-all MPI communication in each time step, restricting the scalability of the simulations to a moderate parallelism on a small number of processors. In this paper, a new parallel bubble merge algorithm will be introduced that communicates updates of the bubble status only locally in a restricted neighborhood. This results in better scalability and is suitable for massive parallelism. The algorithm has been implemented in the lattice Boltzmann software framework waLBerla , resulting in parallel efficiency of 90% on up to 4080 cores.

waLBerla: The Need for Large-Scale Super Computers

The simulation of complex real-life scenarios in fluid dynamics demands a vast amount of computing time and memory that can only be provided by the latest supercomputers. With the access to HLRB II we now have the opportunity to exploit its resources by computing very large-scale lattice Boltzmann simulations of various kinds of interesting problems in fluid dynamics. To be able to benefit from the parallel architecture the target of our software project waLBerla is to provide a parallel, highly scalable and performance-optimized lattice Boltzmann solver. In this paper we present five different fluid dynamics applications that are integrated in waLBerla and that will fully demand the capacities of HLRB II. At its early stage of development, waLBerla has already shown promising results on up to 812 cores. In the course of this project we will further develop the software to be able to take advantage of heterogeneous computer architectures consisting of multi-core CPUs, cell processors and graphics cards.

waLBerla: Optimization for Itanium-based Systems with Thousands of Processors

Performance optimization is an issue at different levels, in particular for computing and communication intensive codes like free surface lattice Boltzmann. This method is used to simulate liquid-gas flow phenomena such as bubbly flows and foams. Due to a special treatment of the gas phase, an aggregation of bubble volume data is necessary in every time step. In order to accomplish efficient parallel scaling, the all-to-all communication schemes used up to now had to be replaced with more sophisticated patterns that work in a local vicinity. With this approach, scaling could be improved such that simulation runs on up to 9 152 processor cores are possible with more than 90% efficiency. Due to the computation of surface tension effects, this method is also computational intensive. Therefore, also optimization of single core performance plays a tremendous role. The characteristics of the Itanium processor require programming techniques that assist the compiler in efficient code vectorization, especially for complex C++ codes like the waLBerla framework. An approach using variable length arrays shows promising results.