Johannes Hötzer

Parallel computing for phase-field models

The phase-field method is gradually exploiting new research areas, and the demand for the ability to simulate larger domains is increasing continuously with the move towards massive parallel computation. We emphasize the efficient usage of high-performance computing resources by investigating the scaling behavior of 1D domain decomposition, 3D domain decomposition and the runtime behavior of the commonly used moving simulation domain as well as 1D load balancing for a finite difference phase-field implementation. A simple performance model for blocking communication and measurements shows that it is necessary to apply 3D domain decomposition for a 3D domain to scale on high-performance clusters.

Calibration of a multi-phase field model with quantitative angle measurement

Over the last years, the phase-field method has been established to model capillarity-induced microstructural evolution in various material systems. Several phase-field models were introduced and different studies proved that the microstructure evolution is crucially affected by the triple junction (TJ’s) mobilities as well as the evolution of the dihedral angles. In order to understand basic mechanisms in multi-phase systems, we are interested in the time evolution of TJ’s, especially in the contact angles in these regions. Since the considered multi-phase systems consist of a high number of grains, it is not feasible to measure the angles at all TJ’s by hand. In this work, we present a method enabling the localization of TJ’s and the measurement of dihedral contact angles in the diffuse interface inherent in the phase-field model. Based on this contact angle measurement method, we show how to calibrate the phase-field model in order to satisfy Young’s law for different contact angles.

Massively parallel phase-field simulations for ternary eutectic directional solidification

Microstructures forming during ternary eutectic directional solidification processes have significant influence on the macroscopic mechanical properties of metal alloys. For a realistic simulation, we use the well established thermodynamically consistent phase-field method and improve it with a new grand potential formulation to couple the concentration evolution. This extension is very compute intensive due to a temperature dependent diffusive concentration. We significantly extend previous simulations that have used simpler phase-field models or were performed on smaller domain sizes. The new method has been implemented within the massively parallel HPC framework waLBerla that is designed to exploit current supercomputers efficiently. We apply various optimization techniques, including buffering techniques, explicit SIMD kernel vectorization, and communication hiding. Simulations utilizing up to 262,144 cores have been run on three different supercomputing architectures and weak scalability results are shown. Additionally, a hierarchical, mesh-based data reduction strategy is developed to keep the I/O problem manageable at scale.

The parallel multi-physics phase-field framework PACE3D

Abstract The phase-field method has been established for the numerical investigation of various microstructure evolution processes. The accurate description of these complex processes requires large domains and suitable models, allowing to couple several physical fields in statistical representative volume elements. To simplify the implementation of new models and to reduce the simulation run time, different frameworks have been developed. In this work, the parallel multi-physics phase-field framework Pace3D is introduced. The general structure of the solver, its modules and the parallelization are described. For increasing the performance of the implemented phase-field models, various optimization techniques are outlined. To efficiently store the simulation results, different data formats and parallel writing mechanisms are presented. The performance of an optimized implementation for a specific phase-field model is analyzed on a single core, showing a good peak performance. For a single node, the memory bandwidth is analyzed and ruled out as possible bottleneck. In addition, a proper weak scaling behavior is demonstrated on the three supercomputers ForHLR I, ForHLR II and Hazel Hen, for up to 96 100 cores.

Quantitative Comparison of Ternary Eutectic Phase-Field Simulations with Analytical 3D Jackson–Hunt Approaches

For the analytical description of the relationship between undercoolings, lamellar spacings and growth velocities during the directional solidification of ternary eutectics in 2D and 3D, different extensions based on the theory of Jackson and Hunt are reported in the literature. Besides analytical approaches, the phase-field method has been established to study the spatially complex microstructure evolution during the solidification of eutectic alloys. The understanding of the fundamental mechanisms controlling the morphology development in multiphase, multicomponent systems is of high interest. For this purpose, a comparison is made between the analytical extensions and three-dimensional phase-field simulations of directional solidification in an ideal ternary eutectic system. Based on the observed accordance in two-dimensional validation cases, the experimentally reported, inherently three-dimensional chain-like pattern is investigated in extensive simulation studies. The results are quantitatively compared with the analytical results reported in the literature, and with a newly derived approach which uses equal undercoolings. A good accordance of the undercooling–spacing characteristics between simulations and the analytical Jackson–Hunt apporaches are found. The results show that the applied phase-field model, which is based on the Grand potential approach, is able to describe the analytically predicted relationship between the undercooling and the lamellar arrangements during the directional solidification of a ternary eutectic system in 3D.

Metallic foam structures, dendrites and implementation optimizations for phase-field modeling

We present our current work in the field of computational materials science with the phase-field method on the high performance cluster XC 4000 of the KIT (Karlsruhe Institute of Technology). Our investigations include heat conduction of open cell metal foams, dendritic growth and optimizations of the concurrent processing with the message passing interface (MPI) standard. Large scale simulations are applied to identify relevant parameters of heat conduction and dendrite growth. Our overall goal is to continuously develop our models, numerical solution techniques and software implementations. The basic model and parallelization scheme is described. Disadvantages of 1D domain decomposition compared to 3D domain decomposition for large 3D simulation domains are explained and a detailed analysis of the new 3D decomposition needs to be performed. The data throughput of parallel file IO operations is measured and system specific differences have been found which need further investigations.

Massiv-parallele und großskalige Phasenfeldsimulationen zur Untersuchung der Mikrostrukturentwicklung

Fur masgeschneiderte Bauteile mit definierten Eigenschaften ist ein detailliertes Verstandnis der Mikrostrukturentwicklung notwendig. Phasenfeldsimulationen erlauben es, gezielt den Einfluss von verschiedenen physikalischen Parametern sowie von Prozessparametern auf die Mikrostrukturentwicklung zu untersuchen. Im ersten Teil wird die Mikrostrukturentwicklung bei der ternaren eutektischen gerichteten Erstarrung untersucht. Hierzu wird die hoch optimierte Umsetzung des Phasenfeldmodells auf Basis des Groskanonischen Potentialansatzes im massiv-parallelen waLBerla-Framework gezeigt. In Messungen wird damit auf einem Rechenkern eine Peak Performance von 27,1% sowie ein nahezu ideales Skalierungsverhalten mit bis zu 1048576 Prozessen erreicht. Ausgehend von Simulationsstudien in 2D zu gekipptem Wachstum wird das in Experimenten vermutete und raumlich komplexe Wachstum von Spiralen in grosskaligen 3D-Simulationen nachgewiesen. Die Musterbildung vor und nach der Loslichkeitsanderung wahrend der Erstarrung im System Al-Ag-Cu sowie der Einfluss von verschiedenen Geschwindigkeitswechseln auf die Musterbildung wird qualitativ wie auch quantitativ untersucht. Hierbei wird eine gute Ubereinstimmung mit Experimenten gefunden. Im zweiten Teil wird die Mikrostrukturentwicklung unter dem Einfluss von Poren an Korngrenzen wahrend des Endstadiums des Sinterprozesses analysiert. Der Druck in den Poren wird hierzu uber das ideale Gasgesetz im Phasenfeldmodell modelliert. Zur Beschreibung der Poren-Poren-Interaktion wird im PACE-Loser ein effizienter und paralleler Algorithmus zur Beschreibung von Topologieanderungen auf Basis von Zusammenhangskomponenten umgesetzt. Fur die Untersuchung des Abloseverhaltens der Poren von Korngrenzen werden idealisierte sowie realistische Mikrostrukturen mit analytischen Gleichungen verglichen. Zudem wird eine Separationskarte abhangig von der durchschnittlichen Korngrose und dem Porenabstand erstellt.

Application of Large-Scale Phase-Field Simulations in the Context of High-Performance Computing

In material science, simulations became a common tool for the understanding of the underlying behaviour of different classes of materials. Due to the growing complexity of problems at hand, the simulation domains, and therefore the computational effort is steadily increasing. We presents various application of the phase-field method; ranging from the solidification of ternary eutectics and pure ice systems to the interaction of multiple liquid phases on fibers. All these topics have in common, that they need a large number of cores to investigate the decisive physical effects in adequate time. We show an overview of the results for this wide range of applications and the scaling behaviour of the used software frameworks.

Compound Droplets on Fibers

Droplets on fibers have been extensively studied in the recent years. Although the equilibrium shapes of simple droplets on fibers are well established, the situation becomes more complex for compound fluidic systems. Through experimental and numerical investigations, we show herein that compound droplets can be formed on fibers and that they adopt specific geometries. We focus on the various contact lines formed at the meeting of the different phases and we study their equilibrium state. It appears that, depending on the surface tensions, the triple contact lines can remain separate or merge together and form quadruple lines. The nature of the contact lines influences the behavior of the compound droplets on fibers. Indeed, both experimental and numerical results show that, during the detachment process, depending on whether the contact lines are triple or quadruple, the characteristic length is the inner droplet radius or the fiber radius.

A scalable and extensible checkpointing scheme for massively parallel simulations

Realistic simulations in engineering or in the materials sciences can consume enormous computing resources and thus require the use of massively parallel supercomputers. The probability of a failure increases both with the runtime and with the number of system components. For future exascale systems, it is therefore considered critical that strategies are developed to make software resilient against failures. In this article, we present a scalable, distributed, diskless, and resilient checkpointing scheme that can create and recover snapshots of a partitioned simulation domain. We demonstrate the efficiency and scalability of the checkpoint strategy for simulations with up to 40 billion computational cells executing on more than 400 billion floating point values. A checkpoint creation is shown to require only a few seconds and the new checkpointing scheme scales almost perfectly up to more than 260, 000 (218) processes. To recover from a diskless checkpoint during runtime, we realize the recovery algorithms using ULFM MPI. The checkpointing mechanism is fully integrated in a state-of-the-art high-performance multi-physics simulation framework. We demonstrate the efficiency and robustness of the method with a realistic phase-field simulation originating in the material sciences and with a lattice Boltzmann method implementation.