Roger R. Schmidt

Challenges of data center thermal management

The need for more performance from computer equipment in data centers has driven the power consumed to levels that are straining thermal management in the centers. When the computer industry switched from bipolar to CMOS transistors in the early 1990s, low-power CMOS technology was expected to resolve all problems associated with power and heat. However, equipment power consumption with CMOS has been rising at a rapid rate during the past 10 years and has surpassed power consumption from equipment installed with the bipolar technologies 10 to 15 years ago. Data centers are being designed with 15-20-year life spans, and customers must know how to plan for the power and cooling within these data centers. This paper provides an overview of some of the ongoing work to operate within the thermal environment of a data center. Some of the factors that affect the environmental conditions of data-communication (datacom) equipment within a data center are described. Since high-density racks clustered within a data center are of most concern, measurements are presented along with the conditions necessary to meet the datacom equipment environmental requirements. A number of numerical modeling experiments have been performed in order to describe the governing thermo-fluid mechanisms, and an attempt is made to quantify these processes through performance metrics.

A Practical Implementation of Silicon Microchannel Coolers for High Power Chips

This paper describes a practical implementation of a single-phase Si microchannel cooler designed for cooling very high power chips such as microprocessors. Through the use of multiple heat exchanger zones and optimized cooler fin designs, a unit thermal resistance 10.5 C-mm2 /W from the cooler surface to the inlet water was demonstrated with a fluid pressure drop of <35kPa. Further, cooling of a thermal test chip with a microchannel cooler bonded to it packaged in a single chip module was also demonstrated for a chip power density greater than 300W/cm2. Coolers of this design should be able to cool chips with average power densities of 400W/cm2 or more

Review of cooling technologies for computer products

This paper provides a broad review of the cooling technologies for computer products from desktop computers to large servers. For many years cooling technology has played a key role in enabling and facilitating the packaging and performance improvements in each new generation of computers. The role of internal and external thermal resistance in module level cooling is discussed in terms of heat removal from chips and module and examples are cited. The use of air-cooled heat sinks and liquid-cooled cold plates to improve module cooling is addressed. Immersion cooling as a scheme to accommodate high heat flux at the chip level is also discussed. Cooling at the system level is discussed in terms of air, hybrid, liquid, and refrigeration-cooled systems. The growing problem of data center thermal management is also considered. The paper concludes with a discussion of future challenges related to computer cooling technology.

High-end server low-temperature cooling

The IBM S/390® G4 CMOS system, first shipped in 1997, was the first high-end system to use refrigeration. The decision to employ refrigeration cooling instead of other cooling options such as high-flow air cooling or various water-cooling schemes focused on the potential system performance improvement obtainable by lowering coolant temperatures using a refrigeration system. This paper reviews the historical background of refrigeration from its use in the early 1800s to its implementation in computer systems in the early 1990s. The advantages and disadvantages of using refrigeration in the cooling of computer systems are examined. The advantages have outweighed the disadvantages, leading to the first use by IBM of refrigeration in cooling the S/390 G4 server. The design of the refrigeration system for the S/390 G4 system is described in detail, and some of the key parametric studies that contributed to the final design are described.

AIRFLOW UNIFORMITY THROUGH PERFORATED TILES IN A RAISED-FLOOR DATA CENTER

The maximum equipment power density (e.g. in power/rack or power/area) that may be deployed in a typical raised-floor data center is limited by perforated tile airflow. In the design of a data center cooling system, a simple estimate of mean airflow per perforated tile is typically made based on the number of CRACs and number of perforated tiles (and possibly a leakage airflow estimate). However, in practice, many perforated tiles may deliver substantially more or less than the mean, resulting in, at best, inefficiencies and, at worst, equipment failure due to inadequate cooling. Consequently, the data center designer needs to estimate the magnitude of variations in perforated tile airflow prior to construction or renovation. In this paper, over 240 CFD models are analyzed to determine the impact of data-center design parameters on perforated tile airflow uniformity. The CFD models are based on actual data center floor plans and the CFD model is verified by comparison to experimental test data. Perforated tile type and the presence of plenum obstructions have the greatest potential influence on airflow uniformity. Floor plan, plenum depth, and airflow leakage rate have modest effect on uniformity and total airflow rate (or average plenum pressure) has virtually no effect. Good uniformity may be realized by using more restrictive (e.g. 25%- open) perforated tiles, minimizing obstructions and leakage airflow, using deeper plenums, and using rectangular floor plans with standard hot aisle/cold aisle arrangements.

Challenges in Electronic Cooling—Opportunities for Enhanced Thermal Management Techniques—Microprocessor Liquid Cooled Minichannel Heat Sink

The volumetric heat dissipated by computer equipment at each level of the package from the chip to the chassis is having a tremendous impact on the thermal management of computer equipment. Because of the consumers insatiable desire for increased performance, the competitive pressures are driving the computer manufacturer to pack as much processor/memory performance within the smallest volume possible. The consumer views high performance in a compact package as a benefit. These market pressures seem to be in direct conflict with the desire to continue to provide air cooling solutions for the foreseeable future. Because of these trends in power and package design, other cooling technologies beside air are now becoming viable, techniques, but each must be weighed with many other factors that influence the cooling technology selected. These factors will be discussed along with two specific IBM server packages and their associated cooling technology employed. Finally a microprocessor liquid cooled minichannel heat sink will be described and its performance presented as it applies to a current microprocessor (IBM Power4) chip.

Comparative Analysis of Different Data Center Airflow Management Configurations

Increase in computing power resulting from high performance microprocessors, packages, and modules and the deployment of high heat load computer rack units in high density configurations, has escalated the thermal challenges in today’s data center systems. One of the key issues is the location of hot recirculation regions in the room and the mixing of hot rack exhaust air with the cold supply air. Along with many factors such as the rack heat load and the cooling capacity of the supply air, the data center thermal management architecture plays an important role in determining the reliability of the electronic equipment and the general thermal performance of the data center. There are several candidate configurations available for the air ducting designs for data centers. The overall energy efficiency of the system is highly dependant upon the selection of the specific configuration. This paper will summarize the results of a broad numerical study carried out to assess the effectiveness of different data center configurations. The numerical modeling is performed using a commercial computational fluid dynamics (CFD) code based on finite volume approach. The configurations studied include different combinations of raised floor and ceiling supply and return vent location subject to specific constraints. The performance of the data center has been characterized on the basis of average and maximum mean region rack inlet air temperature. Among the seven different configurations compared, the raised floor/ceiling return type configuration is found to be the most effective configuration for the given set of constraints and assumptions.Copyright

Server liquid cooling with chiller-less data center design to enable significant energy savings

This paper summarizes the concept design and hardware build efforts as part of a US Department of Energy cost shared grant, two year project (2010-2012) that was undertaken to develop highly energy efficient, warm liquid cooled servers for use in chiller-less data centers. Significant savings are expected in data center energy, refrigerant and make up water use. The technologies being developed include liquid cooling hardware for high volume servers, advanced thermal interface materials, and dry air heat exchanger (chiller-less with all year “economizer”) based facility level cooling systems that reject the Information Technology (IT) equipment heat load directly to the outside ambient air. Substantial effort has also been devoted towards exploring the use of high volume manufacturable components and cost optimized cooling designs that address high volume market design points. Demonstration hardware for server liquid cooling and data center economizer based cooling has been built and is operational for a 15 kW rack fully populated with liquid cooled servers. This design allows the use of up to 45 °C liquid coolant to the rack. Data collection has commenced to document the system thermal performance and energy usage using sophisticated instrumentation and data collection software methodologies. The anticipated benefits of such energy-centric configurations are significant energy savings at the data center level of as much as 30% and energy-proportional cooling in real time based on IT load and ambient air temperatures. The objective of this project is to reduce the cooling energy to 5% or less of a comparable typical air cooled chiller based total data center energy. Additional energy savings can be realized by reducing the IT power itself through reduced server fan power and potentially less leakage power due to lower device temperatures on average for most locations. This paper focuses on the server liquid cooling, the rack enclosure with heat exchanger cooling and liquid distribution, and the data center level cooling infrastructure. A sample of recently collected energy-efficiency data is also presented to provide experimental validation of the concept demonstrating cooling energy use to be less than 3.5% of the IT power for a hot summer day in New York.

Optimization of Data Center Room Layout to Minimize Rack Inlet Air Temperature

In a typical raised floor data center with alternating hot and cold aisles, air enters the front of each rack over the entire height of the rack. Since the heat loads of data processing equipment continue to increase at a rapid rate, it is a challenge to maintain the temperature of all the racks within the stated requirement. A facility manager has discretion in deciding the data center room layout, but a wrong decision will eventually lead to equipment failure. There are many complex decisions to be made early in the design as the data center evolves. Challenges occur such as optimizing the raised floor plenum, floor tile placement, minimizing the data center local hot spots, etc. These adjustments in configuration affect rack inlet air temperatures, which is one of the important keys to effective thermal management. In this paper, a raised floor data center with 12 kW racks is considered. There are four rows of racks with alternating hot and cold aisle arrangement. Each row has six racks installed. Two air-conditioning units supply chilled air to the data center through the pressurized plenum. Effect of plenum depth, floor tile placement, and ceiling height on the rack inlet air temperature is discussed. Plots will be presented over the defined range. A multivariable approach to optimize data center room layout to minimize the rack inlet air temperature is proposed. Significant improvement over the initial model is shown by using a multivariable design optimization approach.

System cooling design for the water-cooled IBM Enterprise System/9000 processors

The high operating speed and corresponding high chip heat fluxes in the IBM Enterprise System/9000™ water-cooled mainframe processors are made possible by improvements in component- and system-level cooling. The heart of the closed-loop water-cooling system is a coolant distribution frame (CDF) common to all water-cooled processors. The CDF provides a controlled water temperature of 21.7°C to the central electronic complex (CEC) at water flow rates up to 245 liters per minute (lpm) and rejects heat loads of up to 63 kW for the largest processor. The water flow provides cooling to multichip thermal conduction modules (TCMs), to power supplies, and to air-to-water heat exchangers that provide preconditioned air to channel and memory cards. As many as 121 chips are mounted on a TCM glass-ceramic substrate, with chip powers reaching 27 W or a heat flux of 64 W/cm 2 . A separate cold plate was developed to cool these modules. The power supplies with high heat densities are primarily cooled by water which flows through a unique separable cold plate designed for ease of serviceability of the power supply. Although water cooling is utilized for components with high heat fluxes, air cooling is employed for elements of the system with lower power densities. For cards cooled by forced air, careful trade-off studies among acoustical power, chip reliability, and high availability were required. The acoustic noise emissions of all the fans and blowers were determined, and a system model was constructed to measure the noise radiated from each frame in the system. The data were used to design top covers and other components to ensure that the system could meet its thermal/acoustical requirements. A closed-loop frame in which all the heat was rejected to water was developed to meet these requirements.