Milnes P. David

Server liquid cooling with chiller-less data center design to enable significant energy savings

This paper summarizes the concept design and hardware build efforts as part of a US Department of Energy cost shared grant, two year project (2010-2012) that was undertaken to develop highly energy efficient, warm liquid cooled servers for use in chiller-less data centers. Significant savings are expected in data center energy, refrigerant and make up water use. The technologies being developed include liquid cooling hardware for high volume servers, advanced thermal interface materials, and dry air heat exchanger (chiller-less with all year “economizer”) based facility level cooling systems that reject the Information Technology (IT) equipment heat load directly to the outside ambient air. Substantial effort has also been devoted towards exploring the use of high volume manufacturable components and cost optimized cooling designs that address high volume market design points. Demonstration hardware for server liquid cooling and data center economizer based cooling has been built and is operational for a 15 kW rack fully populated with liquid cooled servers. This design allows the use of up to 45 °C liquid coolant to the rack. Data collection has commenced to document the system thermal performance and energy usage using sophisticated instrumentation and data collection software methodologies. The anticipated benefits of such energy-centric configurations are significant energy savings at the data center level of as much as 30% and energy-proportional cooling in real time based on IT load and ambient air temperatures. The objective of this project is to reduce the cooling energy to 5% or less of a comparable typical air cooled chiller based total data center energy. Additional energy savings can be realized by reducing the IT power itself through reduced server fan power and potentially less leakage power due to lower device temperatures on average for most locations. This paper focuses on the server liquid cooling, the rack enclosure with heat exchanger cooling and liquid distribution, and the data center level cooling infrastructure. A sample of recently collected energy-efficiency data is also presented to provide experimental validation of the concept demonstrating cooling energy use to be less than 3.5% of the IT power for a hot summer day in New York.

Experimental characterization of an energy efficient chiller-less data center test facility with warm water cooled servers

Typical data centers utilize approximately 50% of the total IT energy in cooling of the server racks. We present a chillerless data center where server-level cooling is achieved through a combination of warm water cooling hardware and re-circulated air; eventual heat rejection to ambient air is achieved using a closed secondary liquid loop to ambient-air heat exchanger (dry-cooler). Several experiments were carried out to characterize the individual pieces of equipment and data center thermal performance and energy consumption. A 22+ hour experimental run was also carried out with results indicating an average cooling energy use of 3.5% of the total IT energy use, with average ambient air temperatures of 23.8°C and average IT power use of 13.14 kW.

Impact of operating conditions on a chiller-less data center test facility with liquid cooled servers

Data center cooling can constitute a significant portion of the total data center energy usage, with typical cooling energy expenditures approximately 50% of the IT energy use. Much of this energy consumption occurs at the refrigeration/chiller plant and in the Computer Room Air Handlers (CRAH) that cool and condition the air used to cool the electronics racks. To reduce cooling energy use, a data center test facility was designed and constructed to reduce cooling energy use to less than 5% of the total IT energy use through a combination of warm water cooling of the electronics and liquid-side economization. Several data center operating conditions, such as changes in liquid and air flow rates, heat exchanger arrangements and addition of propylene glycol were investigated to determine their impact on the energy consumption and thermal performance of the key cooling equipment. Day long runs collected from summer and fall days are also reported to illustrate the impact of external weather conditions and loop operating conditions on the thermal performance and energy consumption of the dual-loop data center test facility. The work presented highlights the impact of various operating conditions in influencing the cooling energy use and improving data center energy efficiency in chiller-less, ambient-air cooled data center designs using water cooled servers.

Experimental investigation of water cooled server microprocessors and memory devices in an energy efficient chiller-less data center

Understanding and improving the thermal management and energy efficiency of data center cooling systems is of growing importance from a cost and sustainability perspective. Toward this goal, warm liquid cooled servers were developed to enable highly energy efficient chiller-less data centers that utilize only “free” ambient environment cooling. This approach greatly reduces cooling energy use, and could reduce data center refrigerant and make up water usage. In one exemplary experiment, a rack having such liquid cooled servers was tested on a hot summer day (~32°C) with CPU exercisers and memory exercisers running on every server to provide steady heat dissipation from the processors and from the DIMMs, respectively. Compared to a typical air cooled rack, significantly lower DIMM temperatures and CPU thermal values were observed.

Extreme energy efficiency using water cooled servers inside a chiller-less data center

The paper summarizes part of a project that was undertaken to develop highly energy efficient warm liquid cooled servers for use in chiller-less data centers that could save significant data center energy use and reduce data center refrigerant and make up water usage. One of the key concepts developed as part of this project is the Dual-Enclosure-Liquid-Cooling (DELC), which comprises of a 100% liquid cooled server cabinet and an outdoor dry cooler unit for heat rejection to the ambient and this configuration is used to reject the Information Technology (IT) equipment heat load directly to the outside ambient air without the use of a chiller. Demonstration hardware for server liquid cooling and a chiller-less data center was built and is operational for a 15 kW rack fully populated with liquid cooled servers which has been designed for use for up to 45°C liquid coolant to the rack. The anticipated benefits of such energy-centric configurations are significant energy savings of as much as 25% at the data center level. This paper builds on recent work that focused on the server liquid cooling, the rack enclosure with heat exchanger cooling and liquid distribution, and the data center level cooling infrastructure and which also presented sample data from experiments in support of the DELC concept. This paper presents experimental data related to the novel data center loop in a new manner, which is used to create a simplified thermodynamic model using curve-fit of surfaces of heat exchanger approach temperatures and power use of cooling devices. The model is validated using experimental data for a 22 hour test that was conducted in August of 2011. Subsequent to model validation, the simplified model is then used to make projections for DELC prototype performance (thermal and energy) under different conditions including different simple control schemes and weather conditions in the US. Weather data from nine different US cities is analyzed for a single day in August and realizable energy and energy cost savings over traditional chiller based data center cooling designs are presented. The results show that the new innovative data center cooling configuration presented could reduce cooling energy use to be less than 3.5% of the IT power for most US locations even in warm summer times of the year.

Impact of ASHRAE environmental classes on data centers

Data centers consume a significant amount of energy in the US and worldwide, much of which is consumed by the cooling infrastructure, particularly the chiller plant and computer room air conditioners and air handlers. To enable energy efficient data center designs, ASHRAE added two new IT environmental classes, A3 and A4, with associated allowable inlet air temperatures of 40C and 45C respectively. IT equipment that meet these new allowable environmental envelopes can operate in data centers with minimal refrigeration cooling and instead rely on ambient free cooling. In this paper we investigate the impact of allowing a data center to operate up to the A3 limit of 40C on total data center energy use for 3 different types of servers in a chiller-less data center located in a variety of locations. The study finds that though facility power reduces as the demand for cold air reduces, the increase in IT power consumption, due to fan speed-up, can offset these savings and in some cases result in an overall increase in data center power. Thus the most energy efficient operating point is dependant on the specific energy use profiles of the infrastructure and the IT equipment. The higher allowable temperature can also result in higher failure rates and an increased risk of equipment or service loss due to data center cooling failures. This paper also presents a study on the potential for chiller elimination and chiller use reduction across the US, Europe and in India by operating in the various ASHRAE envelopes. For wet, water side economized data centers, A2 and A3 equipment is sufficient to almost completely remove the need for chillers in many geographic locations.

Experimentally Verified Transient Models of Data Center Crossflow Heat Exchangers

Heat exchangers are key components that are commonly used in data center cooling systems. Rear door heat exchangers, in-row coolers, overhead coolers and fully contained cabinets are some examples of liquid and hybrid cooling systems used in data centers. A liquid to liquid heat exchanger is one of the main components of the Coolant Distribution Unit (CDU), which supplies chilled water to the heat exchangers mentioned above. Computer Room Air Conditioner (CRAC) units also consist of liquid to air cross flow heat exchangers. Optimizing the energy use and the reliability of IT equipment in data centers requires Computational Fluid Dynamics (CFD) tools that can accurately model data centers for both the steady state and dynamic operations. Typically, data centers operate in dynamic conditions due to workload allocations that change both spatially and temporally. Additional dynamic situations may also arise due to failures in the thermal management and electrical distribution systems. In the computational simulation, individual component models, such as transient heat exchanger models, are therefore needed. It is also important to develop simple, yet accurate, compact models for components, such as heat exchangers, to reduce the computational time without decreasing simulation accuracy.In this study, a method for modeling compact transient heat exchangers using CFD code is presented. The method describes an approach for installing thermal dynamic heat exchanger models in CFD codes. The transient effectiveness concept and model are used in the development of the methodology. Heat exchanger CFD compact models are developed and tested by comparing them with full thermal dynamic models, and also with experimental measurements. The transient responses of the CFD model are presented for step and ramp change in flow rates of the hot and cold fluids, as well as step, ramp, and exponential variation in the inlet temperature. Finally, some practical dynamic scenarios involving IBM buffer liquid to liquid heat exchanger, rear door heat exchanger, and CRAC unit, are parametrically modeled to test the developed methodology. It is shown that the compact heat exchanger model can be used to successfully predict dynamic scenarios in typical data centers.Copyright

A dynamic model of failure scenarios of the dry cooler in a liquid cooled chiller-less data center

Data centers take up a significant portion of the national energy consumption in the U.S., accounting for approximately 2% of the total electricity used in 2010. Around the world that percentage is growing at a rapid rate. Within data centers, cooling systems constitute a large portion of the energy consumed, accounting for roughly 25-35% of total expenditures. Many new data center cooling technologies have been developed to improve cooling energy efficiency. A data center cooling facility proposed by IBM was constructed to reduce cooling energy use to less than 5% of the total Information Technology (IT) energy use, which was accomplished through a combination of warm water cooling servers and liquid-side economization. The heat load of IT equipment is rejected out to the ambient air without using a refrigeration/chiller plant or a Computer Room Air Handlers unit (CRAH), which consume a large amount of energy. This work develops simulation tools for the dry cooler, which is an air-to-liquid cross flow heat exchanger used in this cooling technology. Steady state test data is incorporated into the models, and is then used to validate them. The dynamic prediction accuracy of the simulation tool is also compared with real time measurement results of the dry cooler. Results of the simulation and testing are then used to obtain a more complete understanding of the dry cooler performance data and analyze the performance of the dry cooler in the cooling infrastructure. Failure analyses of the dry cooler are also performed in this work. One failure category of the dry cooler is parametrically modeled. The dynamic effects of failures of the dry cooler are reported and analyzed. The effects of the thermal mass of liquid and ambient air as well as the effects of natural convection and thermal, hydraulic characteristics of the external loop on the dynamic behavior of the dry cooler in this failure scenario are presented.

A Numerical Steady State and Dynamic Study in a Data Center Using Calibrated Fan Curves for CRACs and Servers

This study presents the results of a detailed parametric study for a data center that is air cooled using a set of four CRAC units in a cold/hot aisle raised floor configuration. The fans of the CRAC units and the servers are calibrated using their practical characteristics fan curves. A commercial CFD code is utilized for this purpose in which the buoyancy forces are taken into account. The k-epsilon model and the Boussinesq approximation are used to model the turbulent airflow and the buoyancy effect, respectively. A dynamic model is developed to take into account the changes in flow rates and power dissipation in the data center environment. The current dynamic model does not take into account the thermal mass of the CRAC units or the servers. The effect of the CRAC fan speed, instantaneous change in power dissipation, tiles perforation ratio, and servers fan speeds on the total flow rate in the room and the inlet temperatures of the racks are investigated. In the transient model, we investigate the effect of different CRAC failure scenarios on the time history of the temperatures and the flow pattern in the data center. Time constants and safe time are estimated from this study. A fundamental understanding of the effect of different data center entities on the flow and the temperatures is developed. Interesting flow patterns are observed in the case of different CRAC failures that could be used to recommend general design guidelines.Copyright

System-Level Design for Liquid Cooled Chiller-Less Data Center

In 2010, data center power usage amounted to 2% of total US energy consumption [1]. Approximately 25% of the energy consumed is used for IT cooling, which consists of facility refrigeration equipment used to provide computer room air conditioning [2]. As part of a US Department of Energy cost shared grant, a data center test facility was designed and constructed with the goal to demonstrate the potential to reduce IT cooling energy usage to less than 5% of the total IT and facilities energy usage by utilizing warm liquid cooling of the electronics rack. A system thermodynamic model was developed for this liquid cooled chiller-less data center design using theory, numerical simulations and manufacturer’s data sheets. The model was used for the design selection of data center level cooling infrastructure and of server liquid cooling components and for performance estimation of the liquid cooled chiller-less data center design. The system-level model was validated against experimental data and was used to predict the system performance in different geographies.Copyright