Michael J. Ellsworth

Review of cooling technologies for computer products

This paper provides a broad review of the cooling technologies for computer products from desktop computers to large servers. For many years cooling technology has played a key role in enabling and facilitating the packaging and performance improvements in each new generation of computers. The role of internal and external thermal resistance in module level cooling is discussed in terms of heat removal from chips and module and examples are cited. The use of air-cooled heat sinks and liquid-cooled cold plates to improve module cooling is addressed. Immersion cooling as a scheme to accommodate high heat flux at the chip level is also discussed. Cooling at the system level is discussed in terms of air, hybrid, liquid, and refrigeration-cooled systems. The growing problem of data center thermal management is also considered. The paper concludes with a discussion of future challenges related to computer cooling technology.

The evolution of water cooling for IBM large server systems: Back to the future

This paper provides a technical perspective and review of water cooling technology as implemented through 5 generations of IBMs high performance computing systems from the S360/91 to the recently announced IBM Power 575 supercomputing system. The use of hybrid air-to-water cooling and then indirect (cold plate) water cooling in earlier IBM systems is described. Attention is given to how and why water-cooling was implemented to provide the required cooling capability while maintaining ease of serviceability at the module level. Also discussed is the use of a Cooling Distribution Unit (CDU) to control cooling system water temperature, distribute water to multiple racks and serve as a buffer between system water and customer facility water. Rising microprocessor power dissipation, increased heat loads at the data center level, and demands for increased cooling energy efficiency are presented as driving the need for the reintroduction of water cooling. The introduction of the rear door heat exchanger to respond to the challenge of rising heat loads at the data center level is discussed. Finally, the IBM Power 575 water cooling system is described. Included in the discussion are the coolant flow architecture and the incorporation of Modular Water cooling Units (MWUs) within the server frame replacing the remote CDU concept.

An Overview of the IBM Power 775 Supercomputer Water Cooling System

In 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load: the heat load to air. The next generation of this platform, the Power 775 Supercomputing node/system, is a monumental leap forward in computing performance and energy efficiency. The computer node and system were designed from the start with water cooling in mind. The result, a system with greater than 96% of its heat load conducted directly to water, is a system that, together with a rear door heat exchanger, removes 100% of its heat load to water with no requirement for room air conditioning. In addition to the processor, the memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an interboard air-to-water heat exchanger. This paper will give an overview of the water cooling system featuring the water conditioning unit and rack manifolds. Advances in technology over this system’s predecessor will be highlighted. An overview of the cooling assemblies within the server drawer (i.e., central electronics complex,) the disc enclosure, and the centralized (bulk) power conversion system will also be given. Furthermore, techniques to enhance performance and energy efficiency will also be described.

Chip power density and module cooling technology projections for the current decade

Several cooling technologies were examined with the intent of quantifying their cooling capabilities based on practical applications or constraints. Several air cooling concepts were examined: a heat sink attached to the module, a heat sink attached directly to the chip, and a heat pipe heat sink attached directly to the chip. Water cooling concepts were also examined: water cooled cold plate attached to a module and a bellows cooling concept that brings the water closer to the chip. And finally, liquid immersion cooling schemes were examined: Jet impingement, flow through a heat sink attached to the chip, and spray cooling. Both fluoroinerts cooling and water cooling were considered. Fluroinert cooling in general does not fair as well as water cooling. Water cooling by far has the greatest cooling extendibility, and when applied directly to the chip is the only technology considered in this study capable of handling heat fluxes in excess of 200 W/cm/sup 2/.

Thermal and mechanical analysis and design of the IBM Power 775 water cooled supercomputing central electronics complex

Back in 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load; the heat load to air. Water cooling enabled a compute node with 34% greater performance (Flops), resulted in a processor temperature 20-30°C lower than that typically provided with air cooling, and reduced the power consumed in the data center to transfer the IT heat to the outside ambient by as much as 45%. The next generation of this platform, the Power 775 Supercomputing node/system, is a significant leap forward in computing performance and energy efficiency. The compute node and system were designed from the start with water cooling in mind. The result, a system with greater than 95% of its heat load conducted directly to water; a system that, together with a rear door heat exchanger, removes 100% of its heat load to water with no requirement for room air conditioning. In addition to the processor, memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an inter-board air-to-water heat exchanger. This paper will detail key thermal and mechanical design issues associated with the Power 775 server drawer or central electronics complex (CEC). Topics to be addressed include processor and optical I/O Hub Module thermal design (including thermal interfaces); water cooled memory design; module cold plate designs; CEC level water distribution; module level structural analyses for thermal performance; module/board land grid array (LGA) load distribution; effect of load distribution on module thermal interfaces; and the effect of cold plate tubing on module (LGA) loading.

Flow network analysis of the IBM Power 775 supercomputer Water Cooling System

In 2011 IBM announced the Power 775 Supercomputing node/system which, for the time, was a monumental leap forward in computing performance and energy efficiency. The system was designed from the start with water cooling in mind. The result: a system with greater than 95% of its heat load conducted directly to water and a system that, together with a rear door heat exchanger, removes 100% of its heat load to water with no requirement for room air conditioning. In addition to direct water cooling the processor, the memory, power conversion, and I/O electronics also conduct their heat directly to water. Also included within the framework of the system is a disk storage unit (i.e. disc enclosure) containing an interboard air-to-water heat exchanger. A detailed flow network analysis was undertaken to assure adequate flow to all components in the system while minimizing pump power consumption for a wide range of system configurations. The analysis was validated by both early development experimentation as well as final system verification. The network analysis could therefore be used to establish pump speed tables for normal and abnormal modes of operation. This paper will outline the assumptions and methodology associated with the flow network analysis and demonstrates how the analysis was used for the design and operation of the water cooling system.

A Numerical Steady State and Dynamic Study in a Data Center Using Calibrated Fan Curves for CRACs and Servers

This study presents the results of a detailed parametric study for a data center that is air cooled using a set of four CRAC units in a cold/hot aisle raised floor configuration. The fans of the CRAC units and the servers are calibrated using their practical characteristics fan curves. A commercial CFD code is utilized for this purpose in which the buoyancy forces are taken into account. The k-epsilon model and the Boussinesq approximation are used to model the turbulent airflow and the buoyancy effect, respectively. A dynamic model is developed to take into account the changes in flow rates and power dissipation in the data center environment. The current dynamic model does not take into account the thermal mass of the CRAC units or the servers. The effect of the CRAC fan speed, instantaneous change in power dissipation, tiles perforation ratio, and servers fan speeds on the total flow rate in the room and the inlet temperatures of the racks are investigated. In the transient model, we investigate the effect of different CRAC failure scenarios on the time history of the temperatures and the flow pattern in the data center. Time constants and safe time are estimated from this study. A fundamental understanding of the effect of different data center entities on the flow and the temperatures is developed. Interesting flow patterns are observed in the case of different CRAC failures that could be used to recommend general design guidelines.Copyright

An Overview of the Power 775 Supercomputer Water Cooling System

Back in 2008 IBM reintroduced water cooling technology into its high performance computing platform, the Power 575 Supercomputing node/system. Water cooled cold plates were used to cool the processor modules which represented about half of the total system (rack) heat load. An air-to-liquid heat exchanger was also mounted in the rear door of the rack to remove a significant fraction of the other half of the rack heat load; the heat load to air. The next generation of this platform, the Power 775 Supercomputing node/system, is a monumental leap forward in computing performance and energy efficiency. The compute node and system were designed from the start with water cooling in mind. The result, a system with greater than 96% of it’s heat load conducted directly to water; a system that, together with a rear door heat exchanger, removes 100% of it’s heat load to water with no requirement for room air conditioning. In addition to the processor, memory, power conversion, and I/O electronics conduct their heat to water. Included within the framework of the system is a disk storage unit (disc enclosure) containing an interboard air-to-water heat exchanger. This paper will overview the water cooling system featuring the water conditioning unit and rack manifolds. Advances in technology over this system’s predecessor will be highlighted. An overview of the cooling assemblies within the server drawer (i.e. central electronics complex,) the disc enclosure, and the centralized (Bulk) power conversion system will also be given. Further, techniques to enhance performance and energy efficiency will also be described.Copyright

Performance of a Mixed-Refrigerant System Designed for Computer Cooling

CMOS-based computer processors will operate at increased frequencies when cooled to sub-ambient temperatures by a refrigeration system. This benefit must be weighed against many refrigerator considerations, the most noteworthy being thermal performance, efficiency, reliability, and cost. With this in mind a throttle-cycle, mixed-refrigerant cryocooler using a single-stage, oil-lubricated compressor was modified for increased capacity at 173K (-100°C) and its thermal performance was characterized. This mixedrefrigerant system cooled 119 Watts (with an average heat flux of better than 5 Watts/cm2) at an interface temperature of 173K, with a Coefficient of Performance (COP) of more than 22%. The maximum in cooling capacity was 140 Watts at 183K (-90°C). The ratio of the COP relative to Carnot (ideal) efficiency held very constant at 16% over the temperature range of 153 to 183K (-120 to -90°C). This performance (relative to Carnot) is comparable to that of refrigeration systems with similar capacity operating at much higher temperatures. These results establish the good efficiency obtainable using mixedrefrigerant technology, making cooled CMOS applications attractive. Based on experimental data and computer simulations, we anticipate that better than 20% of Carnot efficiency can be achieved with a more optimal design.

Combining computational fluid dynamics (CFD) and flow network modeling (FNM) for design of a multi-chip module (MCM) cold plate

The objective of the study is to improve on performance of the current liquid cooling solution for a Multi-Chip Module (MCM) through design of a chip-scale cold plate with quick and accurate thermal analysis. This can be achieved through application of Flow Network Modeling (FNM) and Computational Fluid Dynamics (CFD) in an interactive manner. Thermal analysis of the baseline cold plate design is performed using CFD to determine initial improvement in performance as compared to the original solution, in terms of thermal resistance and pumping power. Fluid flow through the solution is modeled using FNM and verified with results from the CFD analysis. In addition, CFD is employed to generate flow impedance curves of non-standard components within the cold plate, which are used as input for the Hardy Cross method in FNM. Using the verified flow network model, design parameters of different components in the cold plate are modified to promote uniform flow distribution to each active region in the chip-scale solution. Analysis of the resultant design using CFD determines additional improvement in performance over the original solution, if available. Thus, through complementary application of FNM and CFD, a robust cold plate can be designed without requiring expensive fabrication of prototypes and with minimal computational time and resources.Copyright