Levi A. Campbell

The evolution of water cooling for IBM large server systems: Back to the future

This paper provides a technical perspective and review of water cooling technology as implemented through 5 generations of IBMs high performance computing systems from the S360/91 to the recently announced IBM Power 575 supercomputing system. The use of hybrid air-to-water cooling and then indirect (cold plate) water cooling in earlier IBM systems is described. Attention is given to how and why water-cooling was implemented to provide the required cooling capability while maintaining ease of serviceability at the module level. Also discussed is the use of a Cooling Distribution Unit (CDU) to control cooling system water temperature, distribute water to multiple racks and serve as a buffer between system water and customer facility water. Rising microprocessor power dissipation, increased heat loads at the data center level, and demands for increased cooling energy efficiency are presented as driving the need for the reintroduction of water cooling. The introduction of the rear door heat exchanger to respond to the challenge of rising heat loads at the data center level is discussed. Finally, the IBM Power 575 water cooling system is described. Included in the discussion are the coolant flow architecture and the incorporation of Modular Water cooling Units (MWUs) within the server frame replacing the remote CDU concept.

An Overview of the IBM Power 775 Supercomputer Water Cooling System

In 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load: the heat load to air. The next generation of this platform, the Power 775 Supercomputing node/system, is a monumental leap forward in computing performance and energy efficiency. The computer node and system were designed from the start with water cooling in mind. The result, a system with greater than 96% of its heat load conducted directly to water, is a system that, together with a rear door heat exchanger, removes 100% of its heat load to water with no requirement for room air conditioning. In addition to the processor, the memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an interboard air-to-water heat exchanger. This paper will give an overview of the water cooling system featuring the water conditioning unit and rack manifolds. Advances in technology over this system’s predecessor will be highlighted. An overview of the cooling assemblies within the server drawer (i.e., central electronics complex,) the disc enclosure, and the centralized (bulk) power conversion system will also be given. Furthermore, techniques to enhance performance and energy efficiency will also be described.

Numerical prediction of the junction-to-fluid thermal resistance of a 2-phase immersion-cooled IBM dual core POWER6 processor

The numerical model used in development of the CPU cold plates for the water-cooled IBM p575 supercomputer is used in this work to predict the junction-to-fluid performance capabilities of passive 2-phase immersion cooling for the same p575 chip module. Experimentally-determined boiling heat transfer coefficients for a porous copper boiling enhancement coating (BEC) were used as a convective boundary condition applied atop the lid in place of the cold plate and secondary thermal interface. The BEC produces 75% and 1500% increases, respectively, in the critical heat flux (CHF) and peak heat transfer coefficients relative to a smooth surface. Lid thicknesses, 3.75mm<;t<;10mm, were modeled at the peak module power of Qm=158W. A thickness of t=3.75mm eliminated regional dryout of the BEC and yielded the optimal sink-to-fluid thermal resistance based on the lid temperature over the centerline of the chip of Rsf=0.073°C/W, a value consistent with previous measurements based on electric heaters similar in size to the P6 core. The resultant average junction-to-fluid thermal resistance was Rjf=0.174°C/W, ~10% lower than the junction-to-water inlet resistance, Rjw,i previously modeled for a single water-cooled cold plate used in the production p575. Immersion system level performance was estimated by assuming that 50% of the volume used for heat sinks in an air-cooled version of the p575 node was available for condensation. The analysis showed roughly equivalent performance to the water-cooled node if the same isolated rack water is used to condense the vapor. If facility water is instead used to condense the vapor directly and at the rack scale, pumps and much of the cooling hardware could be eliminated and the facility water temperature could be raised.

Analysis and characterization of thermoelectric module and heat exchanger performance in a hybrid system cooling application

A thermoelectric chiller is a potential replacement for sub-ambient refrigeration for electronics cooling applications, where the reliance on vapor compression refrigeration results in risk of cooling failure due to the mechanical nature of the compressor and electronic expansion valve. Another benefit of a thermoelectric chiller is that controllable cooling of the electronic component can be achieved regardless of ambient conditions, and the ultimate heat sink can be either air or facility water. The goal of the work described herein is to study a thermoelectric chiller with reasonable capacity (in Watts), coefficient of performance (COP), and reliability (mean time between failures, MTBF), for electronics cooling applications. Four sets of tests are presented: a thermoelectric module tested with a heater block and a cold plate (Figure 2), and thermoelectric heat exchanger tests where the thermoelectric module hot and cold sides are arranged in segregated loops (Figure 5), a single serial loop (Figure 6), and parallel loops (Figure 7).

An Overview of the Power 775 Supercomputer Water Cooling System

Back in 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load; the heat load to air. The next generation of this platform, the Power 775 Supercomputing node/system, is a monumental leap forward in computing performance and energy efficiency. The compute node and system were designed from the start with water cooling in mind. The result, a system with greater than 96% of it’s heat load conducted directly to water; a system that, together with a rear door heat exchanger, removes 100% of it’s heat load to water with no requirement for room air conditioning. In addition to the processor, memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an interboard air-to-water heat exchanger. This paper will overview the water cooling system featuring the water conditioning unit and rack manifolds. Advances in technology over this system’s predecessor will be highlighted. An overview of the cooling assemblies within the server drawer (i.e. central electronics complex,) the disc enclosure, and the centralized (Bulk) power conversion system will also be given. Further, techniques to enhance performance and energy efficiency will also be described.Copyright

Analysis and testing of a thermoelectric heat exchanger configured for cooling a processor module heat load

The testing and subsequent data analysis of a liquid to liquid thermoelectric heat exchanger configured to provide cooled water to a heat load are discussed in the following paper. In the test apparatus, water flow rates and temperatures are recorded. From previously obtained heat transfer data for each heat exchanger plate, and from previous thermoelectric heat exchanger analysis work by Campbell et al. [1] and Luo [2], a model is developed and fit to the current data. Various scenarios are then developed to explore the performance of the thermoelectric heat exchanger in multiple operating modes.

Modeling blower flow characteristics and comparing to measurements

Thermal management for high performance electronic systems such as servers and I/O boxes has become increasingly challenging due to the ever growing demand for higher computing performance and packaging density. Proper modeling, design and characterization of the cooling system have become critical to the overall system performance, reliability and energy consumption. Air moving devices such as blowers (centrifugal fans) are key components of the air-cooled electronic systems. This paper focuses on the flow characteristics of blowers and the impact on the system air flow distribution. To capture the flow characteristics, different levels of numerical modeling methodology are considered using a commercially available tool. It is demonstrated that a simple compact model, typically used in system level models, is not sufficient to resolve the air flow distribution near the exhaust. A more detailed model which includes the actual geometry of the blower blades resolves the body forces using MRF and predicts the flow distribution with better agreement with measurement data. Comparing the different modeling methodologies for systems of different impedance characteristics, a general guideline is subsequently proposed.

Experimental Investigation of the Heat Transfer Performance of Arrays of Round Jets With Sharp-Edged Orifices and Peripheral Effluent: Convective Behavior of Water on a Heated Silicon Surface

In the present work, deionized water is impinged onto a heated silicon surface using square arrays of round jets. Various numbers of jets and jet diameters are used over a heated area of constant size with the orifice plate height above the heater held constant. In these experiments, the jet orifices are sharp-edged and the fluid exhaust direction is parallel to the heated surface and leaves the chip periphery through a manifold. The resulting temperature and flow data are presented in physical units as well as in groups of dimensionless parameters. A correlation is presented to reasonably predict the experimental results of this study. The techniques used for data reduction and for experimentation, including the construction of the test module, are given in detail, including a numerical conduction simulation based data reduction technique and uncertainty analysis. The results shown include flow rates ranging from 6.1 cc/s to 63.18 cc/s resulting in Reynolds numbers based on orifice diameter ranging from 141 to 6670. Jet diameters investigated in this study range from 377 μm to 1.01 mm, in square arrays of 16 to 324 orifices on an area of 18.52 mm × 18.59 mm. The resulting maximum spatially averaged effective heat transfer coefficient achieved is 7.94 W/cm2 K, and the maximum spatially averaged Nusselt number based on jet diameter is 79.4.© 2005 ASME

Numerical Modeling of Jet Impingement and Validation of Convection: Conduction Decoupling in Thermal Design

The steady rise in cooling requirements of commercial computer products mandates the development of aggressive thermal management techniques, as well as accurate design and analysis methodologies. Single phase direct liquid jet impingement, offers a controlled high performance alternative, by eliminating the need for a thermal interface, and by delivering the coolant directly to the surface of the chip. This paper characterizes the thermal performance of a specific direct liquid jet impingement scheme in which the hot fluid exhausts via return vents located in the immediate vicinity of the jet. The study also quantifies the error associated with using the average heat transfer coefficient as a thermal performance metric for the non-uniform cooling boundary condition that occurs in such jet impingement solutions. The thermal performance of the direct liquid jet impingement designs are characterized and studied, via CFD (Computational Fluid Dynamics) models constructed using a commercial numerical solver. The hydraulic performance is analytically estimated using a simple loss coefficient based model. A square 10×10×0.75 mm3 chip, dissipating 400W, and cooled by water at 32°C, is considered as the representative example for the analysis. The effect of jet density on thermal performance is characterized for 1–400 jets/cm2 , and for several feasible flow parameters, i.e. inlet jet velocities (5–10 m/s) and volumetric flow rates (946–1893 liters/minute). For the configurations explored, the optimal jet density was found to be 100 jets/cm2 . An engineering cut-off point for the use of the 1-D average heat transfer coefficient metric, was identified as 10 jets/cm2 . The error associated with use of a 1-D average heat transfer coefficient was shown to be in excess of 5% when the jet density is less than 10 jets/cm2 , as high as 20% for the single jet case, and less than 0.7% for jet densities greater than 36 jets/cm2 .© 2005 ASME

Theoretical (Ideal) Module Cooling and Module Cooling Effectiveness

When contemplating processor module cooling, the notion of maximum cooling capability is not simple or straight forward to estimate. There are a multitude of variables and constraints to consider; some more rigid or fixed than others. This paper proposes a theoretical maximum cooling capability predicated on the treatment of the module heat sink or cold plate as a heat exchanger with infinite conductive and convective behavior. The resulting theoretical minimum heat sink thermal resistance is a function of the bulk thermal transport of the fluid dependent only on the fluid’s density, specific heat (at constant pressure) and volumetric flow rate. An ideal module internal thermal resistance will also be defined. The sum of the two resistances constitutes the theoretical minimum total module thermal resistance and defines the ideal thermal performance of the module. Finally, a module cooling effectiveness relating the actual module thermal performance to the ideal thermal performance will defined. Examples of both air and water cooled modules will be given with discussion on the relevance and utility of this methodology.© 2015 ASME