Christian Feichtinger

WaLBerla: HPC software design for computational engineering simulations

Abstract WaLBerla (Widely applicable Lattice-Boltzmann from Erlangen) is a massively parallel software framework supporting a wide range of physical phenomena. This article describes the software designs realizing the major goal of the framework, a good balance between expandability and scalable, highly optimized, hardware-dependent, special purpose kernels. To demonstrate our designs, we discuss the coupling of our Lattice-Boltzmann fluid flow solver and a method for fluid structure interaction. Additionally, we show a software design for heterogeneous computations on GPU and CPU utilizing optimized kernels. Finally, we estimate the software quality of the framework on the basis of software quality factors.

A flexible Patch-based lattice Boltzmann parallelization approach for heterogeneous GPU-CPU clusters

Sustaining a large fraction of single GPU performance in parallel computations is considered to be the major problem of GPU-based clusters. We address this issue in the context of a lattice Boltzmann flow solver that is integrated in the WaLBerla software framework. Our multi-GPU implementation uses a block-structured MPI parallelization and is suitable for load balancing and heterogeneous computations on CPUs and GPUs. The overhead required for multi-GPU simulations is discussed in detail. It is demonstrated that a large fraction of the kernel performance can be sustained for weak scaling on InfiniBand clusters, leading to excellent parallel efficiency. However, in strong scaling scenarios using multiple GPUs is much less efficient than running CPU-only simulations on IBM BG/P and x86-based clusters. Hence, a cost analysis must determine the best course of action for a particular simulation task and hardware configuration. Finally we present weak scaling results of heterogeneous simulations conducted on CPUs and GPUs simultaneously, using clusters equipped with varying node configurations.

Performance Modeling and Analysis of Heterogeneous Lattice Boltzmann Simulations on CPU-GPU Clusters

Abstract Computational fluid dynamic simulations are in general very compute intensive. Only by parallel simulations on modern supercomputers the computational demands of complex simulation tasks can be satisfied. Facing these computational demands GPUs offer high performance, as they provide the high floating point performance and memory to processor chip bandwidth. To successfully utilize GPU clusters for the daily business of a large community, usable software frameworks must be established on these clusters. The development of such software frameworks is only feasible with maintainable software designs that consider performance as a design objective right from the start. For this work we extend the software design concepts to achieve more efficient and highly scalable multi-GPU parallelization within our software framework waLBerla for multi-physics simulations centered around the lattice Boltzmann method. Our software designs now also support a pure-MPI and a hybrid parallelization approach capable of heterogeneous simulations using CPUs and GPUs in parallel. For the first time weak and strong scaling performance results obtained on the Tsubame 2.0 cluster for more than 1000 GPUs are presented using waLBerla. With the help of a new communication model the parallel efficiency of our implementation is investigated and analyzed in a detailed and structured performance analysis. The suitability of the waLBerla framework for production runs on large GPU clusters is demonstrated. As one possible application we show results of strong scaling experiments for flows through a porous medium.

Coupling multibody dynamics and computational fluid dynamics on 8192 processor cores

This paper describes a method for the fully resolved simulation of particle laden flows. For this purpose, we discuss the parallelization of large scale coupled fluid structure interaction with up to 37 million geometrically modeled moving objects incorporated in the flow. The simulation is performed using a 3D lattice Boltzmann solver for the fluid flow and a so-called rigid body physics engine for the treatment of the objects. The numerical algorithms and the parallelization are discussed in detail. Furthermore, performance results are presented for test cases on up to 8192 processor cores running on an SGI Altix supercomputer. The approach enables a detailed simulation of large scale particulate flows that are relevant for many industrial applications.

WaLBerla: Exploiting Massively Parallel Systems for Lattice Boltzmann Simulations

In this chapter, a software concept for massively parallel computational fluid dynamics (CFD) applications is introduced. The focus thereby lies on the parallelization, which is based on a domain partitioning scheme named patch concept. This concept also enables a seamless specialization of the partitions to different application features as well as the possibility for further optimization such as memory reduction. It is discussed in detail how our design ensures an efficient and flexible implementation. The suitability and efficiency of this concept is demonstrated and evaluated with the waLBerla project, which aims at the development of an efficient massively parallel lattice Boltzmann framework providing the necessary features for several CFD applications. To discuss the suitability of the parallelization for massively parallel usage, various test scenarios have been investigated on different architectures. These tests include serial, weak and strong scaling experiments up to 810 cores and up to a domain size of 15303 lattice cells.

Hybrid lattice-Boltzmann and finite-difference simulation of electroosmotic flow in a microchannel

A three-dimensional (3D) transient mathematical model is developed to simulate electroosmotic flows (EOFs) in a homogeneous, square cross-section microchannel, with and without considering the effects of axial pressure gradients. The general governing equations for electroosmotic transport are incompressible Navier?Stokes equations for fluid flow and the nonlinear Poisson?Boltzmann (PB) equation for electric potential distribution within the channel. In the present numerical approach, the hydrodynamic equations are solved using a lattice-Boltzmann (LB) algorithm and the PB equation is solved using a finite-difference (FD) method. The hybrid LB?FD numerical scheme is implemented on an iterative framework solving the system of coupled time-dependent partial differential equations subjected to the pertinent boundary conditions. Transient behavior of the EOF and effects due to the variations of different physicochemical parameters on the electroosmotic velocity profile are investigated. Transport characteristics for the case of combined electroosmotic- and pressure-driven microflows are also examined with the present model. For the sake of comparison, the cases of both favorable and adverse pressure gradients are considered. EOF behaviors of the non-Newtonian fluid are studied through implementation of the power-law model in the 3D LB algorithm devised for the fluid flow analysis. Numerical simulations reveal that the rheological characteristic of the fluid changes the EOF pattern to a considerable extent and can have significant consequences in the design of electroosmotically actuated bio-microfluidic systems. To improve the performance of the numerical solver, the proposed algorithm is implemented for parallel computing architectures and the overall parallel performance is found to improve with the number of processors.

Effects of surface roughness and electrokinetic heterogeneity on electroosmotic flow in microchannel

In this paper, a hybrid lattice-Boltzmann and finite-difference (LB-FD) model is applied to simulate the effects of three-dimensional surface roughness and electrokinetic heterogeneity on electroosmotic flow (EOF) in a microchannel. The lattice-Boltzmann (LB) method has been employed to obtain the flow field and a finite-difference (FD) method is used to solve the Poisson-Boltzmann (PB) equation for the electrostatic potential distribution. Numerical simulation of flow through a square cross-section microchannel with designed roughness is conducted and the results are critically analysed. The effects of surface heterogeneity on the electroosmotic transport are investigated for different roughness height, width, roughness interval spacing, and roughness surface potential. Numerical simulations reveal that the presence of surface roughness changes the nature of electroosmotic transport through the microchannel. It is found that the electroosmotic velocity decreases with the increase in roughness height and the velocity profile becomes asymmetric. For the same height of the roughness elements, the EOF velocity rises with the increase in roughness width. For the heterogeneously charged rough channel, the velocity profile shows a distinct deviation from the conventional plug-like flow pattern. The simulation results also indicate locally induced flow vortices which can be utilized to enhance the flow and mixing within the microchannel. The present study has important implications towards electrokinetic flow control in the microchannel, and can provide an efficient way to design a microfluidic system of practical interest.

Localized Parallel Algorithm for Bubble Coalescence in Free Surface Lattice-Boltzmann Method

The lattice Boltzmann method is a popular method from computational fluid dynamics. An extension of this method handling liquid flows with free surfaces can be used to simulate bubbly flows. It is based on a volume-of-fluids approach and an explicit tracking of the interface, including a reconstruction of the curvature to model surface tension. When this algorithm is parallelized, complicated data exchange is required, in particular when bubbles extend across several subdomains and when topological changes occur through coalescence of bubbles. In a previous implementation this was handled by using all-to-all MPI communication in each time step, restricting the scalability of the simulations to a moderate parallelism on a small number of processors. In this paper, a new parallel bubble merge algorithm will be introduced that communicates updates of the bubble status only locally in a restricted neighborhood. This results in better scalability and is suitable for massive parallelism. The algorithm has been implemented in the lattice Boltzmann software framework waLBerla , resulting in parallel efficiency of 90% on up to 4080 cores.

waLBerla: The Need for Large-Scale Super Computers

The simulation of complex real-life scenarios in fluid dynamics demands a vast amount of computing time and memory that can only be provided by the latest supercomputers. With the access to HLRB II we now have the opportunity to exploit its resources by computing very large-scale lattice Boltzmann simulations of various kinds of interesting problems in fluid dynamics. To be able to benefit from the parallel architecture the target of our software project waLBerla is to provide a parallel, highly scalable and performance-optimized lattice Boltzmann solver. In this paper we present five different fluid dynamics applications that are integrated in waLBerla and that will fully demand the capacities of HLRB II. At its early stage of development, waLBerla has already shown promising results on up to 812 cores. In the course of this project we will further develop the software to be able to take advantage of heterogeneous computer architectures consisting of multi-core CPUs, cell processors and graphics cards.

A Geometric Multigrid Solver on Tsubame 2.0

Tsubame 2.0 is currently one of the largest installed GPU clusters and number 5 in the Top 500 list ranking the fastest supercomputers in the world. In order to make use of Tsubame, there is a need to adapt existing software design concepts to multi-GPU environments. We have developed a modular and easily extensible software framework called waLBerla that covers a wide range of applications ranging from particulate flows over free surface flows to nano fluids coupled with temperature simulations and medical imaging. In this article we report on our experiences to extend waLBerla in order to support geometric multigrid algorithms for the numerical solution of partial differential equations (PDEs) on multi-GPU clusters. We discuss the software and performance engineering concepts necessary to integrate efficient compute kernels into our waLBerla framework and show first weak and strong scaling results on Tsubame for up to 1029 GPUs for our multigrid solver.