Amilcar R. Arvelo

An Overview of the IBM Power 775 Supercomputer Water Cooling System

In 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load: the heat load to air. The next generation of this platform, the Power 775 Supercomputing node/system, is a monumental leap forward in computing performance and energy efficiency. The computer node and system were designed from the start with water cooling in mind. The result, a system with greater than 96% of its heat load conducted directly to water, is a system that, together with a rear door heat exchanger, removes 100% of its heat load to water with no requirement for room air conditioning. In addition to the processor, the memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an interboard air-to-water heat exchanger. This paper will give an overview of the water cooling system featuring the water conditioning unit and rack manifolds. Advances in technology over this system’s predecessor will be highlighted. An overview of the cooling assemblies within the server drawer (i.e., central electronics complex,) the disc enclosure, and the centralized (bulk) power conversion system will also be given. Furthermore, techniques to enhance performance and energy efficiency will also be described.

Thermal and mechanical analysis and design of the IBM Power 775 water cooled supercomputing central electronics complex

Back in 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load; the heat load to air. Water cooling enabled a compute node with 34% greater performance (Flops), resulted in a processor temperature 20-30°C lower than that typically provided with air cooling, and reduced the power consumed in the data center to transfer the IT heat to the outside ambient by as much as 45%. The next generation of this platform, the Power 775 Supercomputing node/system, is a significant leap forward in computing performance and energy efficiency. The compute node and system were designed from the start with water cooling in mind. The result, a system with greater than 95% of its heat load conducted directly to water; a system that, together with a rear door heat exchanger, removes 100% of its heat load to water with no requirement for room air conditioning. In addition to the processor, memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an inter-board air-to-water heat exchanger. This paper will detail key thermal and mechanical design issues associated with the Power 775 server drawer or central electronics complex (CEC). Topics to be addressed include processor and optical I/O Hub Module thermal design (including thermal interfaces); water cooled memory design; module cold plate designs; CEC level water distribution; module level structural analyses for thermal performance; module/board land grid array (LGA) load distribution; effect of load distribution on module thermal interfaces; and the effect of cold plate tubing on module (LGA) loading.

Package cooling designs for a dual-chip electronic package with one high power chip

Dual-chip microelectronic packages (DCP) with one high power chip are being increasingly encountered in computer and other electronic systems where a common chip carrier, whether a ceramic or an organic laminate, has a central processing unit (CPU) accompanied by a memory chip. In this study, package cooling designs are developed and presented for cooling two product applications of the DCP, with one application having larger power dissipation on the CPU compared to the other. Thermal analysis was conducted to identify the encapsulation solutions for the DCP. Mechanical analysis was then conducted to identify any structural integrity concerns and include appropriate verification tests during reliability assurance testing. The encapsulation processes were optimized to ensure the reliability of the package under field operation. The reliability of the packaging structures was assured using thermal measurements, acoustic sonography, and shear and tensile strength measurements of using test vehicles and actual product DCPs.

Multi-chip package thermal management of IBM z-server systems

The recently announced IBM z9 server system presents unique cooling requirements from a packaging perspective. Cooling has to be achieved for sixteen chips mounted on a common glass ceramic chip carrier. Eight of the sixteen chips dissipate significant power. A recently described small gap technology (SGT) is used to attain customized chip to cap gaps. An advanced thermal compound (ATC) is used as the interface between the chips and the cap. The package thermal and mechanical design is first described. Design optimization is achieved by detailed finite element thermo-mechanical modeling. The complex encapsulation process to attain the correct chip to hat ATC gaps is outlined. Verification of the ATC gaps is an integral part of the assembly process. The reliability qualification is then discussed. Issues found during the qualification were the structural fragility of the glass ceramic chip carrier flange and ATC thermal degradation. The structural robustness of the chip carrier was improved by modifying its design. ATC degradation is quantitatively related to the shear strain

An Overview of the Power 775 Supercomputer Water Cooling System

Back in 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load; the heat load to air. The next generation of this platform, the Power 775 Supercomputing node/system, is a monumental leap forward in computing performance and energy efficiency. The compute node and system were designed from the start with water cooling in mind. The result, a system with greater than 96% of it’s heat load conducted directly to water; a system that, together with a rear door heat exchanger, removes 100% of it’s heat load to water with no requirement for room air conditioning. In addition to the processor, memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an interboard air-to-water heat exchanger. This paper will overview the water cooling system featuring the water conditioning unit and rack manifolds. Advances in technology over this system’s predecessor will be highlighted. An overview of the cooling assemblies within the server drawer (i.e. central electronics complex,) the disc enclosure, and the centralized (Bulk) power conversion system will also be given. Further, techniques to enhance performance and energy efficiency will also be described.Copyright

Hermetic Encapsulation Technique Developed for the IBM Z-Server Multi-Chip Module

The large MCM developed to package the main processor unit used in the IBM z9 Server makes use of a novel sealing design that imparts many desirable characteristics to the module assembly process, performance and reliability. These packages consist of a large ceramic chip carrier encapsulated using a copper cooling cap and a metal sealing ring. The sealing technique not only provides the hermetic environment needed to protect the non-underfilled devices contained within the module, but also allows for easy rework of the assembly. The seal used can withstand the thermally induced stresses and strains driven by the thermal expansion coefficient mismatch between the carrier and the cap. Depending on the system requirements or application, it can do this and reliably maintain the level of hermeticity needed to protect the encapsulated devices over a thousand or more thermal cycles. In addition to this, the seal and module design must compensate for mechanical tolerances of the carrier and devices that affect the assembled condition of the module. In the z-Server module design these considerations, as well as thermal performance factors, are all taken into account. This paper will cover the various aspects of the module design, focusing on the novel application of the hermetic seal employed. The seal will be described and its design parameters will be discussed. Seal, component and module level qualification testing that is performed to insure that the assembly meets the package reliability requirements will be presented.Copyright

Reliability analysis on epoxy based thermal interface subjected to moisture environment

A thermal challenge has been growing in microelectronics industry to dissipate more and more chip power. New thermal solutions have been developed in industry. High metal filled thermal compound is one of them. This kind of thermal compound often uses a polymer matrix such as epoxy as the metal carrier. This paper is to investigate the thermal reliability with the environmental exposure to temperature and humidity. A mathematical diffusion model with variables of temperature, humidity and exposure time is utilized to analyze the moisture content in the thermal compound. Experimental measurements on bonding strength have shown to decrease as the moisture level increases in the thermal interface bondline following an exponential curve. An acceleration model is also obtained for the moisture level in thermal interface. Combined with the mechanical modeling, the packaging products could be designed to achieve a reliable thermal performance in the field with the epoxy based thermal interface