Hatem Ltaief

Fault tolerant algorithms for heat transfer problems

With the emergence of new massively parallel systems in the high performance computing area allowing scientific simulations to run on thousands of processors, the mean time between failures of large machines is decreasing from several weeks to a few minutes. The ability of hardware and software components to handle these singular events called process failures is therefore getting increasingly important. In order for a scientific code to continue despite a process failure, the application must be able to retrieve the lost data items. The recovery procedure after failures might be fairly straightforward for elliptic and linear hyperbolic problems. However, the reversibility in time for parabolic problems appears to be the most challenging part because it is an ill-posed problem. This paper focuses on new fault-tolerant numerical schemes for the time integration of parabolic problems. The new algorithm allows the application to recover from process failures and to reconstruct numerically the lost data of the failed process(es) avoiding the expensive roll-back operation required in most checkpoint/restart schemes. As a fault tolerant communication library, we use the fault tolerant message passing interface developed by the Innovative Computing Laboratory at the University of Tennessee. Experimental results show promising performances. Indeed, the three-dimensional parabolic benchmark code is able to recover and to keep on running after failures, adding only a very small penalty to the overall time of execution.

Fault Tolerant Domain Decomposition for Parabolic Problems

and that Ω is partitioned into N subdomains Ωj , j = 1..N . The computation of these subdomains is distributed among N Processing Units (PUs) or computers. We anticipate that one or several PUs may stall or get disconnected. We complement the distributed architecture of these N PUs, with S additional PUs called spare processing units. The problem in designing a Fault Tolerant (FT) code that can survive to several failures of PUs decomposes as follows:

The compute and control for adaptive optics (CACAO) real-time control software package

The compute and control for adaptive optics (cacao) package is an open-source modular software environment for real-time control of modern adaptive optics system. By leveraging many-core CPU and GPU hardware, it can scale up to meet the demanding computing requirements of current and future high frame rate, high actuator count adaptive optics (AO) systems. cacao’s modular design enables both simple/barebone operation, and complex full-featured AO control systems. cacao’s design is centered on data streams that hold real-time data in shared memory along with a synchronization mechanism for computing processes. Users and programmers can add additional features by coding modules that interact with cacao’s data stream format. We describe cacao’s architecture and its design approach. We show that accurate timing knowledge is key to many of cacao’s advanced operation modes. We discuss current and future development priorities, including support for machine learning to provide real-time optimization of complex AO systems.

Scalable soft real-time supervisor for tomographic AO

Implementations of AO tomography for the next generation of Extremely Large Telescopes (ELTs) is challenging because of the extremely large number of degrees of freedom of such systems, in particular when it comes to the tomographic reconstructor computation, due to its size. The computation of this matrix, via the supervisor module, requires leveraging high performance computing techniques, on shared or distributed memory systems, to comply with the specifications of tomographic AO systems, which prescribe an update rate of the order of few minutes. In the scope of the Green-Flash project, we are exploring several approaches to optimize the execution of this soft real-time supervision pipeline. This includes low-rank techniques to reduce the computational load. We have tested several compression schemes to optimize the linear algebra involved in the tomographic reconstructor as well as the computation of the covariance matrices involved in this process. We present, in this paper, the scalable and portable pipeline we have developed to address these issues. Performance in terms of time to solution and scalability are reported. Additionally, the case of low-rank algorithms is stressed as both an attempt to address the computation challenge of the tomographic reconstructor for the supervisor module, and a way to reduce the computational load (hence the overall RTC system latency) at the level of the real-time data pipeline.

A Parallel Aitken-Additive Schwarz Waveform Relaxation Method for Parabolic Problems

The objective of this paper is to describe a parallel acceleration framework of the additive schwarz waveform relaxation for parabolic problems. The problem is in three space dimension and time. This new parallel domain decomposition algorithm generalizes the Aitken-like Acceleration method of the additive Schwarz algorithm for elliptic problems. Although the standard Schwarz Waveform Relaxation algorithm has a linear rate of convergence and low numerical efficiency, it is beneficent to cache use and scales with the memory. The combination with the Aitken-like Acceleration method transforms the Schwarz algorithm into a direct solver for the heat operator. This solver combines all in one the load balancing, the efficiency, the scalability and the fault tolerance features which make it suitable for grid environments.

Parallel fault tolerant algorithms for parabolic problems

With increasing number of processors available on nowadays high performance computing systems, the mean time between failure of these machines is decreasing. The ability of hardware and software components to handle process failures is therefore getting increasingly important. The objective of this paper is to present a fault tolerant approach for the implicit forward time integration of parabolic problems using explicit formulas. This technique allows the application to recover from process failures and to reconstruct the lost data of the failed process(es) avoiding the roll-back operation required in most checkpoint-restart schemes. The benchmark used to highlight the new algorithms is the two dimensional heat equation solved with a first order implicit Euler scheme.

Efficient Supervision Strategy for Tomographic AO Systems on E-ELT

A critical subsystem of the tomographic AO RTC is the supervisor module. Its role is to feed the challenging real-time data pipeline with a new reconstructor matrix at a regular rate, computed from a statistical analysis of the measurements, to optimize the performance of the AO system. This process involves solving a system of linear equations defined by the covariance matrix of the wave front sensors’ measurements, the size of which may be up to 90k×90k for the E-ELT’s first light instruments using tomographic AO modules. The computational load for the solver of this dense symmetric matrix system is quite significant at this scale but can be efficiently handled using state-of-the-art energy-efficient manycore x86 or accelerator-based architectures, such as KNLs or GPUs, respectively. As part of the Green Flash project, we develop a supervisor module and demonstrate its portability by deploying it on each aforementioned hardware system. Finally, we describe different implementations and their trade-offs in terms of performance and accuracy and show preliminary results on the possible impact of hierarchically low-rank approximation methods on the overall supervisor module.

An Hemodynamic Application on a Distributed Computing Environment

Grids deliver to scientists and researchers an incredible amount of resources geographically spread but often, the lack of accessible and communicative interfaces appears to be a restricting factor for many standard users. The objective of this paper is to present a Biomedical application, based on an incompressible Navier-Stokes code, running on a grid environment which requires the integration of multidisciplinary tools and methods to achieve a high level of performance. The idea is to help doctors and surgeons to establish eciently their diagnosis by simulating blood flows close to real time and therefore, to ensure better therapy for patients. Moreover, Biomedical applications necessitate many computational resources and are very known to be time-consuming. As a matter of fact, our approach is based on parallel distributed numerical methods which relies on the proper combination of three techniques that are the L2 penalty method to deal with complex geometry, a level set method to detect the geometry and a fast domain decomposition solver. A central computer unit has been built to be used as a easy gateway to access the diverse resources remotely, to perform the needed computations and finally, to visualize the simulation in three dimensional space.

Performance Analysis of Fault Tolerant Algorithms for the Heat Equation in Three Space Dimensions

Publisher Summary Based on distributed and uncoordinated check pointing, numerical methods presented in this chapter can reconstruct a consistent state in parallel application, despite storing checkpoints of various processes at different time steps. The main purpose of these algorithms is to avoid the expensive rollback operation to the last consistent distributed checkpoint, losing all the subsequent work and adding a significant overhead for applications running on thousands of processors because of coordinated checkpoints. The first method, the forward implicit scheme, requires for the reconstruction procedure, the boundary variables of each time step to be stored along with the current solution; the second method, based on explicit space/time marching, requires check pointing the solution of each process every time step. To stabilize the scheme, a hyperbolic regularization such as the telegraph equation that is a perturbation of the heat equation may be added. Performance results comparing both methods with respect to the checkpoints overhead have been presented. The checkpointing infrastructure implemented in the 3D-heat equation uses two groups of processes a solver group composed by processes that will solve the problem itself and a spare group of processes whose main function is to store the local data from solver processes.