Gary F. Goth

An Overview of the IBM Power 775 Supercomputer Water Cooling System

In 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load: the heat load to air. The next generation of this platform, the Power 775 Supercomputing node/system, is a monumental leap forward in computing performance and energy efficiency. The computer node and system were designed from the start with water cooling in mind. The result, a system with greater than 96% of its heat load conducted directly to water, is a system that, together with a rear door heat exchanger, removes 100% of its heat load to water with no requirement for room air conditioning. In addition to the processor, the memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an interboard air-to-water heat exchanger. This paper will give an overview of the water cooling system featuring the water conditioning unit and rack manifolds. Advances in technology over this system’s predecessor will be highlighted. An overview of the cooling assemblies within the server drawer (i.e., central electronics complex,) the disc enclosure, and the centralized (bulk) power conversion system will also be given. Furthermore, techniques to enhance performance and energy efficiency will also be described.

Hybrid cooling with cycle steering in the IBM eServer z990

The IBM eServerTM z990 introduces a new mode for cooling multichip processor modules that enables significantly more processors to be refrigerant-cooled than previously. In recent IBM zSeries® offerings, including G4, G5, G6, and z900, chip junctions in a single muhichip module (MCM) located in a central electronic complex (CEC) frame were cooled for reliability and performance benefits, using refrigerant technology, to temperatures lower than those achievable with air cooling. In the z990 system, a hybrid cooling approach is used, allowing refrigeration to be extended to four MCMs in a single CEC, which makes possible denser systems and greater power efficiency compared with prior modular refrigeration technologies used. In the event of a malfunction of the primary refrigeration cooling system, a backup air-cooling System is automatically engaged until the refrigeration problem is fixed. System sensors monitor the cooling state at all times. When air cooling is required, the chip circuit temperatures increase and the logic clocks are optimally adjusted to match the new junction temperatures. When refrigeration cooling is restored, the clocks are adjusted back to their fast speed. This technique allows the z990 system to match the processor density of direct-air-cooled systems while retaining a system performance and reliability benefit from refrigeration.

Dual-tapered piston (DTP) module cooling for IBM Enterprise System/9000 system

The water-cooled thermal conduction modules (TCMs) in the IBM Enterprise System/9000™ (ES/9000™) systems require a fourfold thermal improvement over TCMs in the 3090™ system. An examination of the thermal/mechanical tolerance relationships among the chips, substrate, and cooling hardware showed that a cylindrical piston would not meet this requirement. The piston was redesigned with a cylindrical center section and a taper on each end. This shape minimizes the gap between the piston and “hat” while retaining intimate contact between the piston face and chip surface during all assembly conditions. Numerical and analytical models demonstrate that this new piston shape, coupled with improved conductivity of the cooling hardware materials, exceeds ES/9000 system needs. These models were verified by tests conducted on single-site and full-scale modules in the laboratory and by tests on actual ES/9000 systems.

Packaging design of the IBM system z10 enterprise class platform central electronic complex

The IBM System z10™ Enterprise Class mainframe addresses the modern data center requirements for minimizing floor space while increasing computing power efficiency. These objectives placed challenges on the z10™ packaging design as a result of significantly increased demand on system packaging density, power delivery, and logic and power cooling efficiency compared with the recent IBM System z9® and z990 mainframe generations. Several innovations were implemented to successfully meet these challenges: a more powerful multichip module (MCM) that delivers denser computing capability and a 64-way system; a vertically mated processor unit (PU) book structure that achieves a more efficient thermal implementation and a higher signal bandwidth between processors; and a PU book-centric dc-dc power delivery design that is more efficient. This paper presents the key elements to achieve this design: the novel mechanical load transmission paths and the connector technologies for the MCM, PU book, I/O, and power regulation components; an innovative cooling and thermal design that includes component-level tolerance of failures; and improved power delivery and power code developments to maximize the overall z10 compute efficiency.

A power, packaging, and cooling overview of the IBM eServer z900

This paper provides an overview of the power, packaging, and cooling aspects of the IBM eServer z900 design. The semiconductor processor chips must be supported and protected in a mechanical structure that has to provide electrical interconnects while maintaining the chip junction temperature within specified limits. The mechanical structure should be able to withstand shock and vibrations during transportation or events such as earthquakes. The processor chips require electrical power at well-regulated voltages, unaffected by the ac-line voltage and load current fluctuations. The acoustical and electromagnetic noise produced by the hardware must be within the limits set by national regulatory agencies, and the electronic operations must be adequately protected from disruption caused by electromagnetic radiation. For high availability, the power, packaging, and cooling hardware must have redundancy and the ability to be maintained while the system is operating. This paper first overviews the packaging hardware, followed by a description of the first- and second-level packaging, which includes the mother board and the multichip module. Thermal management is discussed from the point of view of both the multichip module and the overall system. Power conversion, management, and distribution are presented next. Finally, the design aspects involved with meeting the requirements of electromagnetic compatibility, acoustics, and immunity to shock, vibration, and earthquakes are discussed.

Thermal and mechanical analysis and design of the IBM Power 775 water cooled supercomputing central electronics complex

Back in 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load; the heat load to air. Water cooling enabled a compute node with 34% greater performance (Flops), resulted in a processor temperature 20-30°C lower than that typically provided with air cooling, and reduced the power consumed in the data center to transfer the IT heat to the outside ambient by as much as 45%. The next generation of this platform, the Power 775 Supercomputing node/system, is a significant leap forward in computing performance and energy efficiency. The compute node and system were designed from the start with water cooling in mind. The result, a system with greater than 95% of its heat load conducted directly to water; a system that, together with a rear door heat exchanger, removes 100% of its heat load to water with no requirement for room air conditioning. In addition to the processor, memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an inter-board air-to-water heat exchanger. This paper will detail key thermal and mechanical design issues associated with the Power 775 server drawer or central electronics complex (CEC). Topics to be addressed include processor and optical I/O Hub Module thermal design (including thermal interfaces); water cooled memory design; module cold plate designs; CEC level water distribution; module level structural analyses for thermal performance; module/board land grid array (LGA) load distribution; effect of load distribution on module thermal interfaces; and the effect of cold plate tubing on module (LGA) loading.

High-speed interconnect and packaging design of the IBM System z9 processor cage

This paper describes the system packaging and technologies of the IBM System z9TM enterprise-class server. The central electronic complex of the system consists of four nodes, each housing a multichip module (MCM) with 16 chips consuming up to 1,200 W. The z9TM server doubles the multiprocessor performance of the System z990 by increasing the central processing unit (CPU) configuration and using an internally developed elastic interface to increase interconnect speed on all high-speed buses. In contrast to all previous zSeries® designs, which were running at half of the processor speed, the packaging interconnects on the multichip module run at the same speed as the processor (1.72 GHz). High frequencies and massively parallel connectivity lead to a raw packaging bandwidth of up to 1,764 GB/s between processors and cache within a single frame for a fully configured four-node z9 system.

IBM zEnterprise energy management

Data centers are facing serious energy challenges. Increasing energy costs make the operation and cooling of servers more significant cost factors. Furthermore, improvements in technology have led to processor chips and systems with rapidly increasing power density. The resulting power consumption and cooling requirements of these systems are pushing many existing data centers to the limits of their power distribution capability and cooling capacity. Improvements in energy efficiency and management are needed at the chip and the system level to counteract this trend. This paper provides a comprehensive description of the hardware and firmware improvements implemented with IBM zEnterprise® 196 to stop the growth of and even reduce its energy footprint compared with previous IBM System z® servers. These include more power-efficient chips; power conversion and distribution; new sensors; cooling control firmware; new energy management functions; integrated hybrid energy management for power saving and power capping across the whole hybrid system; and data center energy-efficiency improvements resulting from options for water cooling, high-voltage DC (HVDC) power, and overhead cabling.

Mechanical packaging, power, and cooling design for the IBM z13

The system-level packaging of the IBM z13™ supports the implementation of a new drawer-based Central Processor Complex (CPC). Departing from previous IBM z Systems™ designs, the introduction of distributed land-grid-array (LGA) attached single-chip modules (SCMs) required new mechanical, power, and cooling designs to address specified performance requirements and to provide enhanced reliability, availability, and serviceability (RAS) attributes. Building upon the designs created for the IBM zEnterprise® BC12 (zBC12), new CPC drawer and frame mechanical designs were created to significantly increase overall packaging density. Similar to its predecessor, the IBM zEnterprise EC12 (zEC12), the z13 utilizes water-cooling of the processors, but in contrast to the single input and return flow used to cool the multi-chip module (MCM) in the zEC12, the z13 accomplishes its processor cooling using a flexible hose internal manifold design that provides parallel input and return fluid flow to each SCM. The use of flexible hose also enabled SCM field replacement, new to high-end IBM z Systems. A new internal cooling loop unit and an updated external (building-chilled) modular water-conditioning unit were designed utilizing customized water delivery manifold systems to feed the common CPC drawer design. Revised power delivery and service control structures were also created to address the distributed nature of the z13 system design.

An overview of the IBM zEnterprise EC12 processor cooling system

On September 19, 2012 IBM announced its latest System z Enterprise Class zServer, the IBM zEnterprise EC12 (zEC12). This server uses a 96 mm glass ceramic substrate to interconnect processors and related cache chips on a multi-chip module (MCM). In rare applications, the power in these MCMs can exceed 2000W, well beyond air cooling capability. This paper describes a new cooling methodology IBM employs in zEC12 to cool its processor MCMs. From the IBM S/390 G4, which first shipped in 1997, through z196 which is EC12s enterprise class predecessor, IBMs high end System z servers have utilized vapor compression refrigeration to cool its processor MCMs. In zEC12, the thermal solution employs an air to water heat exchanger to provide this function. This paper discusses the technical details of this cooling system. Thermal performance of each component of the cooling path from processor core to ambient, as well as comparison to prior cooling approaches in terms of temperatures, reliability, and energy efficiency will be reviewed. In summary, this technology shows considerable promise for cooling this class of server.