Michael K. Patterson

The effect of data center temperature on energy efficiency

Servers capabilities are increasing at or beyond the rate of performance improvement gains predicted by Moores Law for the silicon itself. The challenge for the information technology (IT) owner is housing and operating all of this computational power in the data center. With more computational power in each unit volume, the industry is experiencing a significant increase in power density and hence a greater cooling challenge. The ability to now tackle computational tasks that were previously unattainable has driven energy costs to new levels. Methods to reduce the energy used in cooling these machines are being studied throughout the industry. One of the areas being considered is increasing the data center server ambient inlet temperature. ASHRAE [1] suggests a recommended limit of 20 - 25degC for the most advanced Data Centers. There is a belief that operating at the high end of this range or above it will reduce the total power use in the data center by making the cooling system more efficient. A thermodynamic analysis clearly indicates that increasing the temperature of the high temperature heat source, while holding the lower temperature heat sink constant, will give an efficiency gain to the heat removal from the system. Unfortunately the simple model does not capture all of the components of the overall system and may lead to an erroneous conclusion. In fact, increasing the ambient temperature can lead to an increase in power usage of some components and systems in the data center as temperature goes up. The overall room energy use may only go down marginally or may even go up at warmer temperatures. In this paper we examine the complete energy picture from the utility connection to the rejection of heat from the facility to the outdoor environment and look at the impact an increased ambient temperature will have on each component in that chain. This analysis indicates that there is an optimum temperature for data center operation that will depend on each data centers individual characteristics, include IT equipment, cooling system architecture, data center location (e.g. outside ambient conditions), as well as other factors. Additional impacts of an increasing ambient inlet temperature, such as reliability issues and operational complexity are also discussed. It is concluded that simply raising the ambient temperature in the data center may not have the desired effect of energy use reduction.

Experimental and Numerical Study of a Stacked Microchannel Heat Sink for Liquid Cooling of Microelectronic Devices

One of the promising liquid cooling techniques for microelectronics is attaching a microchannel heat sink to, or directly fabricating microchannels on, the inactive side of the chip. A stacked microchannel heat sink integrates many layers of microchannels and manifold layers into one stack. Compared with single-layered microchannels, stacked microchannels provide larger flow passages, so that for a fixed heat load the required pressure drop is significantly reduced. Better temperature uniformity can be achieved by arranging counterflow in adjacent microchannel layers. The dedicated manifolds help to distribute coolant uniformly to microchannels. In the present work, a stacked microchannel heat sink is fabricated using silicon micromachining techniques. Thermal performance of the stacked microchannel heat sink is characterized through experimental measurements and numerical simulations. Effects of coolant flow direction, flow rate allocation among layers, and nonuniform heating are studied. Wall temperature profiles are measured using an array of nine platinum thin-film resistive temperature detectors deposited simultaneously with thin-film platinum heaters on the backside of the stacked structure. Excellent overall cooling performance (0.09 ° C/W cm 2 ) for the stacked microchannel heat sink has been shown in the experiments. It has also been identified that over the tested flow rate range, counterflow arrangement provides better temperature uniformity, while parallel flow has the best performance in reducing the peak temperature. Conjugate heat transfer effects for stacked microchannels for different flow conditions are investigated through numerical simulations. Based on the results, some general design guidelines for stacked microchannel heat sinks are provided.

TUE, a New Energy-Efficiency Metric Applied at ORNL’s Jaguar

The metric, Power Usage Effectiveness (PUE), has been successful in improving energy efficiency of data centers, but it is not perfect. One challenge is that PUE does not account for the power distribution and cooling losses inside IT equipment. This is particularly problematic in the HPC (high performance computing) space where system suppliers are moving cooling and power subsystems into or out of the cluster. This paper proposes two new metrics: ITUE (IT-power usage effectiveness), similar to PUE but “inside” the IT and TUE (total-power usage effectiveness), which combines the two for a total efficiency picture. We conclude with a demonstration of the method, and a case study of measurements at ORNL’s Jaguar system. TUE provides a ratio of total energy, (internal and external support energy uses) and the specific energy used in the HPC. TUE can also be a means for comparing HPC site to HPC site.

Numerical study of conjugate heat transfer in stacked microchannels

Microchannel heat sinks feature a high convective heat transfer coefficient, which is particularly beneficial to high-end electronics cooling. There are some issues to be addressed before these can be commercially implemented, among which pressure drop penalty and temperature non-uniformity are critical. Recently, a stacked microchannel heat sink has been proposed to address these two issues. Stacked microchannels provide larger flow passage, so that for a fixed heat load the required pressure drop is significantly reduced. One unique feature of the stacked microchannel heat sink is that individual layers populated with parallel microchannels can be stacked independently. As a beneficial result, flexible control over the flow direction and flow rate can be harnessed to achieve better temperature uniformity and the lowest silicon temperature. The present study conducts numerical study of heat transfer inside stacked microchannels with different flow arrangements including parallel, counter-flow, and serial. For the serial arrangement both top feeding and bottom feeding are considered. The predicted heat removal performance is compared with single layer microchannels that have the same effective flow area. It has been identified that counter-flow arrangement has the best overall performance for temperature uniformity, while parallel flow has the best performance in reducing the peak temperature. This can be explained by the detailed heat transfer information obtained through the conjugate numerical study.

Energy Efficiency Metrics

This presentation addresses the questions: why metrics and what makes a good metric. It then looks at metrics for data centers, infrastructure, sustainability and compute performance.

DWPE, a new data center energy-efficiency metric bridging the gap between infrastructure and workload

To determine whether a High-Performance Computing (HPC) data center is energy efficient, various aspects have to be taken into account: the data centers power distribution and cooling infrastructure, the HPC system itself, the influence of the system management software, and the HPC workloads; all can contribute to the overall energy efficiency of the data center. Currently, two well-established metrics are used to determine energy efficiency for HPC data centers and systems: Power Usage Effectiveness (PUE) and FLOPS per Watt (as defined by the Green500 in their ranking list). PUE evaluates the overhead for running a data center and FLOPS per Watt characterizes the energy efficiency of a system running the High-Performance Linpack (HPL) benchmark, i.e. floating point operations per second achieved with 1 watt of electrical power. Unfortunately, under closer examination even the combination of both metrics does not characterize the overall energy efficiency of a HPC data center. First, HPL does not constitute a representative workload for most of todays HPC applications and the rev 0.9 Green500 run rules for power measurements allows for excluding subsystems (e.g. networking, storage, cooling). Second, even a combination of PUE with FLOPS per Watt metric neglects that the total energy efficiency of a system can vary with the characteristics of the data center in which it is operated. This is due to different cooling technologies implemented in HPC systems and the difference in costs incurred by the data center removing the heat using these technologies. To address these issues, this paper introduces the metrics system PUE (sPUE) and Data center Workload Power Efficiency (DWPE). sPUE calculates the overhead for operating a given system in a certain data center. DWPE is then calculated by determining the energy efficiency of a specific workload and dividing it by the sPUE. DWPE can then be used to define the energy efficiency of running a given workload on a specific HPC system in a specific data center and is currently the only fully-integrated metric suitable for rating an HPC data centers energy efficiency. In addition, DWPE allows for predicting the energy efficiency of different HPC systems in existing HPC data centers, thus making it an ideal approach for guiding HPC system procurement. This paper concludes with a demonstration of the application of DWPE using a set of representative HPC workloads.

A Field Investigation Into the Limits of High-Density Air-Cooling

In this paper we report on a field investigation into airflow management challenges in high density data centers. This field investigation has also served to validate laboratory investigations into high density air cooling issues. In data centers with significant power consumption, and consequently high cooling loads per rack, high volumes of room airflow are required to meet server cooling airflow requirements. These volumes of air can be difficult to deliver in raised floor hot aisle / cold aisle layouts. The velocity of the airflow is such that it creates a negative pressure near the bottom of the rack. This negative pressure entrains air from under and behind the rack, causing recirculation and warmer air being provided to the servers at the base of the rack. This can cause operational problems and server performance impacts. This phenomenon has been demonstrated in previous papers reporting on test data using particle imaging velocimetry (PIV) techniques. The current work validates those studies by looking at airflow, infrared thermography, and actual IT performance while the under rack recirculation flows are occurring. Additionally, we demonstrate significant improvement by employing rigorous airflow management practices. We also discuss the limitations of current CFD modeling, the majority of which does not have sufficient grid-wise resolution to capture the problem. Further we discuss typical operational conditions that have suppressed the problem (or perhaps the awareness of) to date. Finally, the paper recommends best practices to mitigate the problem in high density data centers.Copyright

Use of Computational Fluid Dynamics in the Design and Optimization of Microchannel Heat Exchangers for Microelectronics Cooling

Increasing circuit density and adherence to Moore’s law is driving advanced cooling systems for the next generation microprocessors. One method receiving considerable study is that of microchannel heat exchangers in silicon substrate. These very fine channels in the heat exchanger provide a greatly enhanced convective heat transfer rate and have been shown to be able to meet the demands of the cooling challenge for microprocessors for many generations to come. While the thermal performance has been demonstrated, the design methodology and analysis for fluid structures at this size scale remains difficult. This paper reviews the use of CFD analysis for the design and optimization of microchannel heat exchangers. These results are compared with classical approaches to the same design and demonstrate the need for CFD analysis. Errors using the standard correlations and methods are demonstrated through the results of an optimization study on microchannels. The effects of entrance lengths, spatial variation of the Nusselt number, and temperature dependencies are considered. The literature has widely varying reports on comparisons between experimental results and corresponding theoretical results in microchannel heat exchangers. Experimental validation of the CFD analysis has also been performed and demonstrates that current CFD techniques are actually well suited to heat exchanger designs of this size.Copyright

Stacked Microchannel Heat Sinks for Liquid Cooling of Microelectronics

Stacked microchannels provide larger flow passages, so that for a fixed heat load the required pressure drop is significantly reduced. One unique feature of the stacked microchannel heat sink is that individual layers populated with parallel microchannels or distributing manifolds can be bonded into one stack with independent flow path. As a beneficial result, flexible control over the flow direction and flow rate can be harnessed to achieve better temperature uniformity and the low junction temperature. In the present work, stacked microchannels with different flow arrangement have been fabricated on silicon wafers using micromachining techniques. Platinum thin film heaters are deposited on the backside of the stacked structure to provide heating. In a close-loop setup, water is pumped through the microchannels to carry the heat from the heaters to a remote liquid-liquid heat exchanger rejecting the heat to a recirculating chiller. Wall temperature along the flow direction is measured at nine locations using platinum resistive temperature detectors deposited at the same time as the heaters. Good overall cooling performance (0.09°C/(W/cm2 )) for the stacked microchannel heat sink has been shown in the experiments. It has also been identified that over the tested flow rate range counter-flow arrangement provides better temperature uniformity, while parallel flow has the best performance in reducing the peak temperature.Copyright

Evaluation of a new data center air-cooling architecture: The down-flow Plenum

The rate of innovation in IT system design and especially in High Performance Computing continues to be very high. To keep pace TU Dresden has constructed its new data center using the Plenum concept. The traditional raised floor was substituted by a full building story, creating a highly flexible space to transport power, water, and air. A strict hot-aisle air separation is used and the computer room air-handling (CRAH) units in downflow configuration are positioned directly beneath the hot aisles. This unique arrangement necessitates an unconventional downward flow of hot air from the enclosed hot aisle. Extensive testing has been performed in a cluster of 24 racks (12 per side) equipped with (3+1)×100 kW CRAH unit cooling capacity and 60 test fixtures (air heaters) with 5-15 kW heating power each. Our analysis demonstrates the extremely high efficiency of this air cooling concept even in high-density configurations, up to at least 30 kW per rack. This efficiency is mostly due to the very short airflow paths and wide open cross-sections. We also showcase that no malicious thermal stratification occurs in our hot air downflow configuration. A detailed analysis of the CRAH controls for temperature (through cooling water flow modulation) and airflow (fan speed) highlights the challenges of such control systems in enclosed hot aisle configurations at high power density and short feedback loops. The analysis also considers dynamically changing load patterns including very low partial load scenarios and aspects of operational reliability.