Richard Frederick Barrett

Characterization of Scientific Workloads on Systems with Multi-Core Processors

Multi-core processors are planned for virtually all next-generation HPC systems. In a preliminary evaluation of AMD Opteron Dual-Core processor systems, we investigated the scaling behavior of a set of micro-benchmarks, kernels, and applications. In addition, we evaluated a number of processor affinity techniques for managing memory placement on these multi-core systems. We discovered that an appropriate selection of MPI task and memory placement schemes can result in over 25% performance improvement for key scientific calculations. We collected detailed performance data for several large-scale scientific applications. Analyses of the application performance results confirmed our micro-benchmark and scaling results

Early evaluation of IBM BlueGene/P

BlueGene/P (BG/P) is the second generation BlueGene architecture from IBM, succeeding BlueGene/L (BG/L). BG/P is a system-on-a-chip (SoC) design that uses four PowerPC 450 cores operating at 850 MHz with a double precision, dual pipe floating point unit per core. These chips are connected with multiple interconnection networks including a 3-D torus, a global collective network, and a global barrier network. The design is intended to provide a highly scalable, physically dense system with relatively low power requirements per flop. In this paper, we report on our examination of BG/P, presented in the context of a set of important scientific applications, and as compared to other major large scale supercomputers in use today. Our investigation confirms that BG/P has good scalability with an expected lower performance per processor when compared to the Cray XT4s Opteron. We also find that BG/P uses very low power per floating point operation for certain kernels, yet it has less of a power advantage when considering science driven metrics for mission applications.

Cray XT4: an early evaluation for petascale scientific simulation

The scientific simulation capabilities of next generation high-end computing technology will depend on striking a balance among memory, processor, I/O, and local and global network performance across the breadth of the scientific simulation space. The Cray XT4 combines commodity AMD dual core Opteron processor technology with the second generation of Crays custom communication accelerator in a system design whose balance is claimed to be driven by the demands of scientific simulation. This paper presents an evaluation of the Cray XT4 using micro-benchmarks to develop a controlled understanding of individual system components, providing the context for analyzing and comprehending the performance of several petascale-ready applications. Results gathered from several strategic application domains are compared with observations on the previous generation Cray XT3 and other high-end computing systems, demonstrating performance improvements across a wide variety of application benchmark problems.

Self-consistent full-wave and Fokker-Planck calculations for ion cyclotron heating in non-Maxwellian plasmas

Magnetically confined plasmas can contain significant concentrations of nonthermal plasma particles arising from fusion reactions, neutral beam injection, and wave-driven diffusion in velocity space. Initial studies in one-dimensional and experimental results show that nonthermal energetic ions can significantly affect wave propagation and heating in the ion cyclotron range of frequencies. In addition, these ions can absorb power at high harmonics of the cyclotron frequency where conventional two-dimensional global-wave models are not valid. In this work, the all-orders global-wave solver AORSA [E. F. Jaeger et al., Phys. Rev. Lett. 90, 195001 (2003)] is generalized to treat non-Maxwellian velocity distributions. Quasilinear diffusion coefficients are derived directly from the wave fields and used to calculate energetic ion velocity distributions with the CQL3D Fokker-Planck code [R. W. Harvey and M. G. McCoy, Proceedings of the IAEA Technical Committee Meeting on Simulation and Modeling of Thermonuclear ...

An Evaluation of the Oak Ridge National Laboratory Cray XT3

In 2005, Oak Ridge National Laboratory (ORNL) received delivery of a 5294 processor Cray XT3. The XT3 is Crays third-generation massively parallel processing system. The ORNL system uses a single-processor node built around the AMD Opteron and uses a custom chip—called SeaStar—for interprocessor communication. The system uses a lightweight operating system called Catamount on its compute nodes. This paper provides a performance evaluation of the Cray XT3, including measurements for micro-benchmark, kernel, and application benchmarks. In particular, we provide performance results for strategic Department of Energy applications areas including climate, biology, astrophysics, combustion, and fusion. Our results, on up to 4096 processors, demonstrate that the Cray XT3 provides competitive processor performance, high interconnect bandwidth, and high parallel efficiency on a diverse application workload, typical in the DOE Office of Science.

Simulation of high-power electromagnetic wave heating in the ITER burning plasma

The next step toward fusion as a practical energy source is the design and construction of ITER [R. Aymar et al., Nucl. Fusion 41, 1301 (2001)], a device capable of producing and controlling the high-performance plasma required for self-sustaining fusion reactions, i.e., “burning plasma.” ITER relies in part on ion-cyclotron radio frequency power to heat the deuterium and tritium fuel to fusion temperatures. In order to heat effectively, the radio frequency wave fields must couple efficiently to the dense core plasma. Calculations in this paper support the argument that this will be the case. Three-dimensional full-wave simulations show that fast magnetosonic waves in ITER propagate radially inward with strong central focusing and little toroidal spreading. Energy deposition, current drive, and plasma flow are all highly localized near the plasma center. Very high resolution, two-dimensional calculations reveal the presence of mode conversion layers, where fast waves can be converted to slow ion cyclotron...

Global-wave solutions with self-consistent velocity distributions in ion cyclotron heated plasmas

Global wave solutions with self-consistent velocity distributions are calculated for ion cyclotron heating in non-Maxwellian plasmas. The all-orders spectral algorithm (AORSA) global wave solver is generalized to treat non-thermal velocity distributions arising from fusion reactions, neutral beam injection and wave driven diffusion in velocity space. Quasi-linear diffusion coefficients are derived directly from the wave electric field and used to calculate ion velocity distribution functions with the CQL3D Fokker–Planck code. Alternatively, the quasi-linear coefficients can be calculated numerically by integrating the Lorentz force equations along particle orbits. Self-consistency between the wave electric field and resonant ion distribution function is achieved by iterating between the global-wave and Fokker–Planck solutions.

REDUCING THE BULK IN THE BULK SYNCHRONOUS PARALLEL MODEL

For over two decades the dominant means for enabling portable performance of computational science and engineering applications on parallel processing architectures has been the bulk-synchronous parallel programming (BSP) model. Code developers, motivated by performance considerations to minimize the number of messages transmitted, have typically pursued a strategy of aggregating message data into fewer, larger messages. Emerging and future high-performance architectures, especially those seen as targeting Exascale capabilities, provide motivation and capabilities for revisiting this approach. In this paper we explore alternative configurations within the context of a large-scale complex multi-physics application and a proxy that represents its behavior, presenting results that demonstrate some important advantages as the number of processors increases in scale.

Report of experiments and evidence for ASC L2 milestone 4467 : demonstration of a legacy application's path to exascale.

This report documents thirteen of Sandias contributions to the Computational Systems and Software Environment (CSSE) within the Advanced Simulation and Computing (ASC) program between fiscal years 2009 and 2012. It describes their impact on ASC applications. Most contributions are implemented in lower software levels allowing for application improvement without source code changes. Improvements are identified in such areas as reduced run time, characterizing power usage, and Input/Output (I/O). Other experiments are more forward looking, demonstrating potential bottlenecks using mini-application versions of the legacy codes and simulating their network activity on Exascale-class hardware. The purpose of this report is to prove that the team has completed milestone 4467-Demonstration of a Legacy Applications Path to Exascale. Cielo is expected to be the last capability system on which existing ASC codes can run without significant modifications. This assertion will be tested to determine where the breaking point is for an existing highly scalable application. The goal is to stretch the performance boundaries of the application by applying recent CSSE RD in areas such as resilience, power, I/O, visualization services, SMARTMAP, lightweight LWKs, virtualization, simulation, and feedback loops. Dedicated system time reservations and/or CCC allocations will be used to quantify the impact of system-level changes to extend the life and performance of the ASC code base. Finally, a simulation of anticipated exascale-class hardware will be performed using SST to supplement the calculations. Determine where the breaking point is for an existing highly scalable application: Chapter 15 presented the CSSE work that sought to identify the breaking point in two ASC legacy applications-Charon and CTH. Their mini-app versions were also employed to complete the task. There is no single breaking point as more than one issue was found with the two codes. The results were that applications can expect to encounter performance issues related to the computing environment, system software, and algorithms. Careful profiling of runtime performance will be needed to identify the source of an issue, in strong combination with knowledge of system software and application source code.

Performance analysis and projections for Petascale applications on Cray XT series systems

The Petascale Cray XT5 system at the Oak Ridge National Laboratory (ORNL) Leadership Computing Facility (LCF) shares a number of system and software features with its predecessor, the Cray XT4 system including the quad-core AMD processor and a multi-core aware MPI library. We analyze performance of scalable scientific applications on the quad-core Cray XT4 system as part of the early system access using a combination of micro-benchmarks and Petascale ready applications. Particularly, we evaluate impact of key changes that occurred during the dual-core to quad-core processor upgrade on applications behavior and provide projections for the next-generation massively-parallel platforms with multi-core processors, specifically for proposed Petascale Cray XT5 system. We compare and contrast the quad-core XT4 system features with the upcoming XT5 system and discuss strategies for improving scaling and performance for our target applications.