Shawn A. Hall

Packaging the Blue Gene/L supercomputer

As 1999 ended, IBM announced its intention to construct a one-petaflop supercomputer. The construction of this system was based on a cellular architecture--the use of relatively small but powerful building blocks used together in sufficient quantities to construct large systems. The first step on the road to a petaflop machine (one quadrillion floating-point operations in a second) is the Blue Gene®/L supercomputer. Blue Gene/L combines a low-power processor with a highly parallel architecture to achieve unparalleled computing performance per unit volume. Implementing the Blue Gene/L packaging involved trading off considerations of cost, power, cooling, signaling, electromagnetic radiation, mechanics, component selection, cabling, reliability, service strategy, risk, and schedule. This paper describes how 1,024 dual-processor compute application-specific integrated circuits (ASICs) are packaged in a scalable rack, and how racks are combined and augmented with host computers and remote storage. The Blue Gene/L interconnect, power, cooling, and control systems are described individually and as part of the synergistic whole.

The blue gene/L supercomputer: a hardware and software story

The Blue Gene/L system at the Department of Energy Lawrence Livermore National Laboratory in Livermore, California is the world’s most powerful supercomputer. It has achieved groundbreaking performance in both standard benchmarks as well as real scientific applications. In that process, it has enabled new science that simply could not be done before. Blue Gene/L was developed by a relatively small team of dedicated scientists and engineers. This article is both a description of the Blue Gene/L supercomputer as well as an account of how that system was designed, developed, and delivered. It reports on the technical characteristics of the system that made it possible to build such a powerful supercomputer. It also reports on how teams across the world worked around the clock to accomplish this milestone of high-performance computing.

Power and performance optimization at the system level

The BlueGene/L supercomputer has been designed with a focus on power/performance efficiency to achieve high application performance under the thermal constraints of common data centers. To achieve this goal, emphasis was put on system solutions to engineer a power-efficient system. To exploit thread level parallelism, the BlueGene/L system can scale to 64 racks with a total of 65536 computer nodes consisting of a single compute ASIC integrating all system functions with two industry-standard PowerPC microprocessor cores in a chip multiprocessor configuration. Each PowerPC processor exploits data-level parallelism with a high-performance SIMD oating point unitTo support good application scaling on such a massive system, special emphasis was put on efficient communication primitives by including five highly optimized communification networks. After an initial introduction of the Blue-Gene/L system architecture, we analyze power/performance efficiency for the BlueGene system using performance and power characteristics for the overall system performance (as exemplified by peak performance numbers.To understand application scaling behavior, and its impact on performance and power/performance efficiency, we analyze the NAMD molecular dynamics package using the ApoA1 benchmark. We find that even for strong scaling problems, BlueGene/L systems can deliver superior performance scaling and deliver significant power/performance efficiency. Application benchmark power/performance scaling for the voltage-invariant energy delay 2 power/performance metric demonstrates that choosing a power-efficient 700MHz embedded PowerPC processor core and relying on application parallelism was the right decision to build a powerful, and power/performance efficient system

Pulse-width-modulating control of a nonlinear electromagnetic actuator

The performance of a fast, nonlinear, clapper-type electromagnetic actuator used in impact printing is controlled by real-time measurement and feedback. The objective is to regulate flight time, which is the time from start to actuation to impact. Toward this end, control is applied digitally via pulse-width modulation of a series of coil-driving pulses, which together propel the armature through its trajectory. Each pulse is modulated individually based on state-variable errors measured on its rising edge. The functional relationships between the measured state-variable errors and the required pulse-width modulations are derived systematically, using a computer-controlled method involving trial-and-error experimentation followed by statistical regression. The resulting control law accounts for both mechanical and electrical perturbations and is expressed in an analytic format that can be applied either by look-up tables or by direct computation. Using look-up tables, typical closed-loop operation is shown to achieve dramatic reductions in flight-time error when compared with open-loop operation. >

Energy-Efficient Cooling of Liquid-Cooled Electronics Having Temperature-Dependent Leakage

Energy conservation in data centers with liquid-cooled electronics is considered, taking into account the combined power consumption of both the electronics and the associated cooling equipment, particularly chillers. The energy-saving technique called “free cooling,” which is most effective at minimizing or eliminating the need for chillers when the temperature of the coolant delivered to the electronics is high, is shown to be at odds with the electronics itself, because of subthreshold leakage in complementary metal-oxide semiconductor (CMOS) transistors, which is minimized when coolant temperature is low. A mathematical model is developed to investigate this trade-off, to discover what liquid-coolant temperature is optimal overall, and thereby to determine what combination of free cooling and traditional chiller cooling should be used for minimal overall power consumption. As an example of how to apply the mathematical model, parameters are chosen in accordance with experimental measurements on an early prototype of a state-of-the-art, water-cooled computer (International Business Machiness (IBM) BlueGene/Q). Results for total data-center power (computer + cooling equipment) are obtained as a function of coolant temperature, the computational state of the computer (which affects CMOS leakage), and weather (which affects the ability to employ free cooling). For the type of system considered, the optimal coolant temperature is found to be a discontinuous function of the other parameters, and traditional chiller cooling is found sometimes to be more energy efficient than free cooling.

Packaging the IBM Blue Gene/Q supercomputer

The IBM Blue Gene®/Q supercomputer is designed for highly efficient computing for problems dominated by floating-point computation. Its target mean time between failures for a 96-rack, 98, 304-node system is three days, allowing tasks requiring computation for many days to run at scale, with little time wasted on checkpoint-restart operations. This paper describes various elements of the compute application-specific integrated circuit and the system package, and how they contribute to low power consumption and high reliability.

Real-time control of a nonlinear electromagnetic actuator

Abstract This paper describes theoretical and experimental investigations aimed at controlling the trajectory of a nonlinear, clapper-type electromagnetic actuator. The vehicle for the investigations is the three-piece actuator used in the IBM 4248 impact line printer. The control algorithm seeks to regulate flight time—the time required for the actuator to move from its rest position to its point of impact—by modulating the pulse width delivered to the coil as a function of one measurement of position and velocity. The actuator is modelled mathematically, and a control law based on initial conditions is numerically derived. A similar control law based on mid-flight conditions is derived experimentally, by computer-controlled adaptation, and its ability to regulate flight time is demonstrated.

A Holistic Approach to System Reliability in Blue Gene

Optimizing supercomputer performance requires a balance between objectives for processor performance, network performance, power delivery and cooling, cost and reliability. In particular, scaling a system to a large number of processors poses challenges for reliability, availability and serviceability. Given the power and thermal constraints of data centers, the BlueGene/L supercomputer has been designed with a focus on maximizing floating point operations per second per Watt (FLOPS/Watt). This results in a drastic reduction in FLOPS/m2 floor space and FLOPS/dollar, allowing for affordable scale-up. The BlueGene/L system has been scaled to a total of 65,536 compute nodes in 64 racks. A system approach was used to minimize power at all levels, from the processor to the cooling plant. A BlueGene/L compute node consists of a single ASIC and associated memory. The ASIC integrates all system functions including processors, the memory subsystem and communication, thereby minimizing chip count, interfaces, and power dissipation. As the number of components increases, even a low failure rate per-component leads to an unacceptable system failure rate. Additional mechanisms have to be deployed to achieve sufficient reliability at the system level. In particular, the data transfer volume in the communication networks of a massively parallel system poses significant challenges on bit error rates and recovery mechanisms in the communication links. Low power dissipation and high performance, along with reliability, availability and serviceability were prime considerations in BlueGene/L hardware architecture, system design, and packaging. A high-performance software stack, consisting of operating system services, compilers, libraries and middleware, completes the system, while enhancing reliability and data integrity

Coupling efficiency of a batch-aligned laser-fiber array

Coupling efficiencies for 32-wide, batch-aligned arrays of laser-fiber interconnects were measured and compared to the efficiency obtainable with individual alignment. The tests were carried out using 1300 nm, InP ridge lasers and Corning SMF-28, single-mode optical fibers. In this paper, the associated fabrication and alignment techniques are described briefly. Results are given showing how coupling efficiency depends on misalignment, on laser-to-fiber distance, on fiber bevel angle, and on piece-to-piece variations among lasers and fibers. A direct comparison of batch alignment versus individual alignment is also given, which shows that the average batch-aligned efficiency is about 88 percent as large as the individually aligned (optimal) efficiency. This result, which reflects a particular packaging methodology and best-effort alignment, gives a good indication of the coupling efficiencies which may be typically expected for wide, batch-aligned arrays.