James D. Teresco

Adaptive Local Refinement with Octree Load Balancing for the Parallel Solution of Three-Dimensional Conservation Laws

Conservation laws are solved by a local Galerkin finite element procedure with adaptive space-time mesh refinement and explicit time integration. The Courant stability condition is used to select smaller time steps on smaller elements of the mesh, thereby greatly increasing efficiency relative to methods having a single global time step. Processor load imbalances, introduced at adaptive enrichment steps, are corrected by using traversals of an octree representing a spatial decomposition of the domain. To accommodate the variable time steps, octree partitioning is extended to use weights derived from element size. Partition boundary smoothing reduces the communications volume of partitioning procedures for a modest cost. Computational results comparing parallel octree and inertial partitioning procedures are presented for the three-dimensional Euler equations of compressible flow solved on an IBM SP2 computer.

Parallel structures and dynamic load balancing for adaptive finite element computation

Abstract An adaptive technique for a partial differential system automatically adjusts a computational mesh or varies the order of a numerical procedure to obtain a solution satisfying prescribed accuracy criteria in an optimal fashion. We describe data structures for distributed storage of finite element mesh data as well as software for mesh adaptation, load balancing, and solving compressible flow problems. Processor load imbalances are introduced at adaptive enrichment steps during the course of a parallel computation. To correct this, we have developed three dynamic load balancing procedures based, respectively, on load imbalance trees, moment of inertia, and octree traversal. Computational results on an IBM SP2 computer are presented for steady and transient solutions of the three-dimensional Euler equations of compressible flow.

Partitioning and Dynamic Load Balancing for the Numerical Solution of Partial Differential Equations

In parallel simulations, partitioning and load-balancing algorithms compute the distribution of application data and work to processors. The effectiveness of this distribution greatly influences the performance of a parallel simulation. Decompositions that balance processor loads while keeping the application’s communication costs low are preferred. Although a wide variety of partitioning and load-balancing algorithms have been developed, their effectiveness depends on the characteristics of the application using them. In this chapter, we review several partitioning algorithms, along with their strengths and weaknesses for various PDE applications. We also discuss current efforts toward improving partitioning algorithms for future applications and architectures.

A hierarchical partition model for adaptive finite element computation

Software tools for the solution of partial differential equations using parallel adaptive finite element methods have been developed. We describe the design and implementation of the parallel mesh structures within an adaptive framework. The most fundamental concept is that of a hierarchical partition model used to distribute finite element meshes and associated data on a parallel computer. The hierarchical model represents heterogeneous processor and network speeds, and may be used to represent processes in any parallel computing environment, including an SMP, a distributed-memory computer, a network of workstations, or some combination of these. Using this model to segment the computation into chunks which can fit into cache memory provides a potential efficiency gain from an increased cache hit rate, even in a single processor environment. The information about different processor speeds, memory sizes, and the corresponding interconnection network can be useful in a dynamic load balancing algorithm which seeks to achieve a good balance with minimal interprocessor communication penalties when a slow interconnection network is involved.

Resource-aware scientific computation on a heterogeneous cluster

Although researchers can develop software on small, local clusters and move it later to larger clusters and supercomputers, the software must run efficiently in both environments. Two efforts aim to improve the efficiency of scientific computation on clusters through resource-aware dynamic load balancing. The popularity of cost-effective clusters built from commodity hardware has opened up a new platform for the execution of software originally designed for tightly coupled supercomputers. Because these clusters can be built to include any number of processors ranging from fewer than 10 to thousands, researchers in high-performance scientific computation at smaller institutions or in smaller departments can maintain local parallel computing resources to support software development and testing, then move the software to larger clusters and supercomputers. As promising as this ability is, it has also led to the need for local expertise and resources to set up and maintain these clusters. The software must execute efficiently both on smaller local clusters and on larger ones. These computing environments vary in the number of processors, speed of processing and communication resources, and size and speed of memory throughout the memory hierarchy as well as in the availability of support tools and preferred programming paradigms. Software developed and optimized using a particular computing environment might not be as efficient when its moved to another one. In this article, we describe a small cluster along with two efforts to improve the efficiency of parallel scientific computation on that cluster. Both approaches modify the dynamic load-balancing step of an adaptive solution procedure to tailor the distribution of data across the cooperating processes. This modification helps account for the heterogeneity and hierarchy in various computing environments.

Software for the parallel adaptive solution of conservation laws by discontinous Galerkin methods.

The authors develop software tools for the solution of conservation laws using parallel adaptive discontinuous Galerkin methods. In particular, the Rensselaer Partition Model (RPM) provides parallel mesh structures within an adaptive framework to solve the Euler equations of compressible flow by a discontinuous Galerkin method (LOCO). Results are presented for a Rayleigh-Taylor flow instability for computations performed on 128 processors of an IBM SP computer. In addition to managing the distributed data and maintaining a load balance, RPM provides information about the parallel environment that can be used to tailor partitions to a specific computational environment.

Hierarchical partitioning and dynamic load balancing for scientific computation

Cluster and grid computing has made hierarchical and heterogeneous computing systems increasingly common as target environments for large-scale scientific computation. A cluster may consist of a network of multiprocessors. A grid computation may involve communication across slow interfaces. Modern supercomputers are often large clusters with hierarchical network structures. For maximum efficiency, software must adapt to the computing environment. We focus on partitioning and dynamic load balancing, in particular on hierarchical procedures implemented within the Zoltan Toolkit, guided by DRUM, the Dynamic Resource Utilization Model. Here, different balancing procedures are used in different parts of the domain. Preliminary results show that hierarchical partitionings are competitive with the best traditional methods on a small hierarchical cluster.

Distributed Octree Data Structures and Local Refinement Method for the Parallel Solution of Three-Dimensional Conservation Laws

Conservation laws are solved by a local Galerkin finite element procedure with adaptive space-time mesh refinement and explicit time integration. A distributed octree structure representing a spatial decomposition of the domain is used for mesh generation, and later may be used to correct for processor load imbalances introduced at adaptive enrichment steps. A Courant stability condition is used to select smaller time steps on smaller elements of the mesh, thereby greatly increasing efficiency relative to methods having a single global time step. To accommodate the variable time steps, octree partitioning is extended to use weights derived from element size. Computational results are presented for the three-dimensional Euler equations of compressible flow solved on an IBM SP2 computer. The problem examined is the flow inside a perforated shock tube.

The Quality of Partitions Produced by an Iterative Load Balancer

We examine the quality of partitions produced by an iterative load balancer in parallel adaptive finite element calculations. We present several metrics which we use to evaluate the quality of a mesh partitioning, and report statistics generated from our analysis of adaptively refined meshes produced during the solution of computational fluid dynamics problems. Timings from the finite element solution phase for runs involving these meshes on 16 and 32 processors of an IBM SP2 are also presented.

Adaptive computation over dynamic and heterogeneous networks

Over the last two decades, efficient message passing libraries have been developed for parallel scientific computation. Concurrently, programming languages have been created supporting dynamically reconfigurable distributed systems over the heterogeneous Internet. In this paper, we introduce SALSA-MPI, an actor programming language approach to scientific computing that extends MPI with a checkpointing and migration API and a runtime system that manages both periodic checkpoints and process or application migration. The goal is to enable dynamic network reconfiguration and load balancing without sacrificing application performance or requiring extensive code modifications. As driving technology for this effort of unifying parallel and distributed computing, we plan to use adaptive solvers of partial differential equations. Fields as diverse as fluid dynamics, material science, biomechanics, and ecology make use of parallel adaptive computation, but target architectures have traditionally been supercomputers and tightly-coupled clusters. SALSA-MPI is intended to allow these computations to make efficient use of more distributed and dynamic computing resources.