Vinod Kamath

Server liquid cooling with chiller-less data center design to enable significant energy savings

This paper summarizes the concept design and hardware build efforts as part of a US Department of Energy cost shared grant, two year project (2010-2012) that was undertaken to develop highly energy efficient, warm liquid cooled servers for use in chiller-less data centers. Significant savings are expected in data center energy, refrigerant and make up water use. The technologies being developed include liquid cooling hardware for high volume servers, advanced thermal interface materials, and dry air heat exchanger (chiller-less with all year “economizer”) based facility level cooling systems that reject the Information Technology (IT) equipment heat load directly to the outside ambient air. Substantial effort has also been devoted towards exploring the use of high volume manufacturable components and cost optimized cooling designs that address high volume market design points. Demonstration hardware for server liquid cooling and data center economizer based cooling has been built and is operational for a 15 kW rack fully populated with liquid cooled servers. This design allows the use of up to 45 °C liquid coolant to the rack. Data collection has commenced to document the system thermal performance and energy usage using sophisticated instrumentation and data collection software methodologies. The anticipated benefits of such energy-centric configurations are significant energy savings at the data center level of as much as 30% and energy-proportional cooling in real time based on IT load and ambient air temperatures. The objective of this project is to reduce the cooling energy to 5% or less of a comparable typical air cooled chiller based total data center energy. Additional energy savings can be realized by reducing the IT power itself through reduced server fan power and potentially less leakage power due to lower device temperatures on average for most locations. This paper focuses on the server liquid cooling, the rack enclosure with heat exchanger cooling and liquid distribution, and the data center level cooling infrastructure. A sample of recently collected energy-efficiency data is also presented to provide experimental validation of the concept demonstrating cooling energy use to be less than 3.5% of the IT power for a hot summer day in New York.

Experimental investigation of water cooled server microprocessors and memory devices in an energy efficient chiller-less data center

Understanding and improving the thermal management and energy efficiency of data center cooling systems is of growing importance from a cost and sustainability perspective. Toward this goal, warm liquid cooled servers were developed to enable highly energy efficient chiller-less data centers that utilize only “free” ambient environment cooling. This approach greatly reduces cooling energy use, and could reduce data center refrigerant and make up water usage. In one exemplary experiment, a rack having such liquid cooled servers was tested on a hot summer day (~32°C) with CPU exercisers and memory exercisers running on every server to provide steady heat dissipation from the processors and from the DIMMs, respectively. Compared to a typical air cooled rack, significantly lower DIMM temperatures and CPU thermal values were observed.

Extreme energy efficiency using water cooled servers inside a chiller-less data center

The paper summarizes part of a project that was undertaken to develop highly energy efficient warm liquid cooled servers for use in chiller-less data centers that could save significant data center energy use and reduce data center refrigerant and make up water usage. One of the key concepts developed as part of this project is the Dual-Enclosure-Liquid-Cooling (DELC), which comprises of a 100% liquid cooled server cabinet and an outdoor dry cooler unit for heat rejection to the ambient and this configuration is used to reject the Information Technology (IT) equipment heat load directly to the outside ambient air without the use of a chiller. Demonstration hardware for server liquid cooling and a chiller-less data center was built and is operational for a 15 kW rack fully populated with liquid cooled servers which has been designed for use for up to 45°C liquid coolant to the rack. The anticipated benefits of such energy-centric configurations are significant energy savings of as much as 25% at the data center level. This paper builds on recent work that focused on the server liquid cooling, the rack enclosure with heat exchanger cooling and liquid distribution, and the data center level cooling infrastructure and which also presented sample data from experiments in support of the DELC concept. This paper presents experimental data related to the novel data center loop in a new manner, which is used to create a simplified thermodynamic model using curve-fit of surfaces of heat exchanger approach temperatures and power use of cooling devices. The model is validated using experimental data for a 22 hour test that was conducted in August of 2011. Subsequent to model validation, the simplified model is then used to make projections for DELC prototype performance (thermal and energy) under different conditions including different simple control schemes and weather conditions in the US. Weather data from nine different US cities is analyzed for a single day in August and realizable energy and energy cost savings over traditional chiller based data center cooling designs are presented. The results show that the new innovative data center cooling configuration presented could reduce cooling energy use to be less than 3.5% of the IT power for most US locations even in warm summer times of the year.

System-Level Design for Liquid Cooled Chiller-Less Data Center

In 2010, data center power usage amounted to 2% of total US energy consumption [1]. Approximately 25% of the energy consumed is used for IT cooling, which consists of facility refrigeration equipment used to provide computer room air conditioning [2]. As part of a US Department of Energy cost shared grant, a data center test facility was designed and constructed with the goal to demonstrate the potential to reduce IT cooling energy usage to less than 5% of the total IT and facilities energy usage by utilizing warm liquid cooling of the electronics rack. A system thermodynamic model was developed for this liquid cooled chiller-less data center design using theory, numerical simulations and manufacturer’s data sheets. The model was used for the design selection of data center level cooling infrastructure and of server liquid cooling components and for performance estimation of the liquid cooled chiller-less data center design. The system-level model was validated against experimental data and was used to predict the system performance in different geographies.Copyright

Air injection and convection cooling of multi-chip modules: a computational study

In the personal computer and workstation industry, new power-hungry, high-performance microprocessors have been gaining ground. Some proposed commercial heat dissipation schemes use fan-mounted heat sinks to ensure that the chip junction temperature meets technology requirements for reliable performance. This study uses a numerical simulation to predict heat transfer rates for cooling of a typical multi-chip package. Two cooling schemes are evaluated. The first uses a pin-fin heat sink mounted on the package with air forced over the sink. The second scheme uses a fan mounted directly over the heat-sink injecting air on the sink. Increases in heat transfer rates of the injection case compared to standard forced cooling schemes are tabulated. A qualitative description of the flow features is given, along with limitations of the flow computations.<<ETX>>

A Holistic Evaluation of Data Center Water Cooling Total Cost of Ownership

The generation-to-generation information technology (IT) performance and density demands continue to drive innovation in data center cooling technologies. For many applications, the ability to efficiently deliver cooling via traditional chilled air cooling approaches has become inadequate. Water cooling has been used in data centers for more than 50 years to improve heat dissipation, boost performance, and increase efficiency. While water cooling can undoubtedly have a higher initial capital cost, water cooling can be very cost effective when looking at the true life cycle cost of a water-cooled data center. This study aims at addressing how one should evaluate the true total cost of ownership (TCO) for water-cooled data centers by considering the combined capital and operational cost for both the IT systems and the data center facility. It compares several metrics, including return-on-investment for three cooling technologies: traditional air cooling, rack-level cooling using rear door heat exchangers, and direct water cooling (DWC) via cold plates. The results highlight several important variables, namely, IT power, data center location, site electric utility cost, and construction costs and how each of these influences the TCO of water cooling. The study further looks at implementing water cooling as part of a new data center construction project versus a retrofit or upgrade into an existing data center facility.

Understanding the True Total Cost of Ownership of Water Cooling for Data Centers

The generation-to-generation IT performance and density demands continue to drive innovation in data center cooling technologies. For many applications, the ability to efficiently deliver cooling via traditional chilled air cooling approaches has become inadequate. Water cooling has been used in data centers for more than 50 years to improve heat dissipation, boost performance and increase efficiency. While water cooling can undoubtedly have a higher initial capital cost, water cooling can be very cost effective when looking at the true lifecycle cost of a water cooled data center.This study aims at addressing how one should evaluate the true total cost of ownership for water cooled data centers by considering the combined capital and operational cost for both the IT systems and the data center facility. It compares several metrics, including return-on-investment for three cooling technologies: traditional air cooling, rack-level cooling using rear door heat exchangers and direct water cooling via cold plates. The results highlight several important variables, namely, IT power, data center location, site electric utility cost, and construction costs and how each of these influence the total cost of ownership of water cooling. The study further looks at implementing water cooling as part of a new data center construction project versus a retrofit or upgrade into an existing data center facility.Copyright

A Computational Study to Compare the Data Center Cooling Energy Spent Using Traditional Air Cooled and Hybrid Water Cooled Servers

High performance datacenters that are being built and operated to ensure optimized compute density for high performance computing (HPC) workloads are constrained by the requirement to provide adequate cooling for the servers. Traditional methods of cooling dense high power servers using air cooling imposes a large cooling and power burden on datacenters. Airflow optimization of the datacenter is a constraint subject to a high energy penalty when dense power hungry racks each capable of consuming 30 to 40 kW are populated in a dense datacenter environment. The work documented using a simulation model (TileFlow) in this paper demonstrates the challenges associated with a standard air cooled approach in a HPC datacenter. Alternate cooling approaches to traditional air cooling are simulated as a comparison to traditional air cooling. These include models using a heat exchanger assisted rack cooling solution with conventional chilled water and, a direct to node cooling model simulated for the racks.These three distinct data center models are simulated at varying workloads and the resulting data is presented for typical and maximal inlet temperatures to the racks. For each cooling solution an estimate of the energy spend for the servers is determined based on the estimated PUEs of the cooling solutions chosen.Copyright

Thermodynamic Characterization of a Direct Water Cooled Server Rack Running Synthetic and Real High Performance Computing Work Loads

High performance computing server racks are being engineered to contain significantly more processing capability within the same computer room footprint year after year. The processor density within a single rack is becoming high enough that traditional, inefficient air-cooling of servers is inadequate to sustain HPC workloads. Experiments that characterize the performance of a direct water-cooled server rack in an operating HPC facility are described in this paper. Performance of the rack is reported for a range of cooling water inlet temperatures, flow rates and workloads that include actual and worst-case synthetic benchmarks. Power and temperature measurements of all processors and memory components in the rack were made while extended benchmark tests were conducted throughout the range of cooling variables allowed within an operational HPC facility. Synthetic benchmark results were compared with those obtained on a single server of the same design that had been characterized thermodynamically. Neither actual nor synthetic benchmark performances were affected during the course of the experiments, varying less than 0.13 percent. Power consumption change in the rack was minimal for the entire excursion of coolant temperatures and flow rates. Establishing the characteristics of such a highly energy efficient server rack in situ is critical to determine how the technology might be integrated into an existing heterogeneous, hybrid cooled computing facility — i.e., a facility that includes some servers that are air cooled as well as some that are direct water cooled.Copyright

Thermodynamic Characterization of a Server Optimized for High Performance Computing Using Water Cooling

Energy efficiency is an essential element of server design for high performance computers. Traditional HPC servers or nodes that are air cooled enable efficiency by using optimized system design elements that include efficient heat sink design for critical components such as CPUs, Memory, Networking and Disk Subsystems. In addition, airflow optimization is enabled via critical component placement decisions as well as fan and cooling algorithms that have an objective to optimize airflow and maximize system performance. Critical elements that cannot be avoided in traditional air cooled servers are computer center level management of both the airflow requirements and the exhaust heat flux of the servers. An alternative approach shown in this paper uses a novel water cooled design that enables both extreme energy efficiency for heat extraction of the server heat load and allows for lower device operating temperatures for the critical components. Experimental data documented in this paper illustrates the advantages of using non-chilled water to cool the server, allowing 85 to 90 percent of the server heat load to be extracted by water while allowing inlet water temperatures up to 45 degrees Celsius. A comparison is made of the energy consumption needed to cool the server components for both the air cooled and water cooled systems. The base system used for the comparison uses identical system electronics and firmware. The server thermal data shown in the paper include thermal behavior at idle, typical and maximum power consumption states for the server. The data documents the range of boundary conditions that can be tolerated for water cooled server solutions and the comparative advantages of using this technology.Copyright