Mark D. Schultz

Server liquid cooling with chiller-less data center design to enable significant energy savings

This paper summarizes the concept design and hardware build efforts as part of a US Department of Energy cost shared grant, two year project (2010-2012) that was undertaken to develop highly energy efficient, warm liquid cooled servers for use in chiller-less data centers. Significant savings are expected in data center energy, refrigerant and make up water use. The technologies being developed include liquid cooling hardware for high volume servers, advanced thermal interface materials, and dry air heat exchanger (chiller-less with all year “economizer”) based facility level cooling systems that reject the Information Technology (IT) equipment heat load directly to the outside ambient air. Substantial effort has also been devoted towards exploring the use of high volume manufacturable components and cost optimized cooling designs that address high volume market design points. Demonstration hardware for server liquid cooling and data center economizer based cooling has been built and is operational for a 15 kW rack fully populated with liquid cooled servers. This design allows the use of up to 45 °C liquid coolant to the rack. Data collection has commenced to document the system thermal performance and energy usage using sophisticated instrumentation and data collection software methodologies. The anticipated benefits of such energy-centric configurations are significant energy savings at the data center level of as much as 30% and energy-proportional cooling in real time based on IT load and ambient air temperatures. The objective of this project is to reduce the cooling energy to 5% or less of a comparable typical air cooled chiller based total data center energy. Additional energy savings can be realized by reducing the IT power itself through reduced server fan power and potentially less leakage power due to lower device temperatures on average for most locations. This paper focuses on the server liquid cooling, the rack enclosure with heat exchanger cooling and liquid distribution, and the data center level cooling infrastructure. A sample of recently collected energy-efficiency data is also presented to provide experimental validation of the concept demonstrating cooling energy use to be less than 3.5% of the IT power for a hot summer day in New York.

Experimental characterization of an energy efficient chiller-less data center test facility with warm water cooled servers

Typical data centers utilize approximately 50% of the total IT energy in cooling of the server racks. We present a chillerless data center where server-level cooling is achieved through a combination of warm water cooling hardware and re-circulated air; eventual heat rejection to ambient air is achieved using a closed secondary liquid loop to ambient-air heat exchanger (dry-cooler). Several experiments were carried out to characterize the individual pieces of equipment and data center thermal performance and energy consumption. A 22+ hour experimental run was also carried out with results indicating an average cooling energy use of 3.5% of the total IT energy use, with average ambient air temperatures of 23.8°C and average IT power use of 13.14 kW.

A self-servowrite clocking process

A high-speed process for servowriting hard-disk assemblies (HDAs) without an external clock head has been developed. This robust process achieves servo-pattern alignment accuracy comparable to or better than that achieved with conventional clock-head based servowriters at a substantially reduced capital and process cost.

Impact of operating conditions on a chiller-less data center test facility with liquid cooled servers

Data center cooling can constitute a significant portion of the total data center energy usage, with typical cooling energy expenditures approximately 50% of the IT energy use. Much of this energy consumption occurs at the refrigeration/chiller plant and in the Computer Room Air Handlers (CRAH) that cool and condition the air used to cool the electronics racks. To reduce cooling energy use, a data center test facility was designed and constructed to reduce cooling energy use to less than 5% of the total IT energy use through a combination of warm water cooling of the electronics and liquid-side economization. Several data center operating conditions, such as changes in liquid and air flow rates, heat exchanger arrangements and addition of propylene glycol were investigated to determine their impact on the energy consumption and thermal performance of the key cooling equipment. Day long runs collected from summer and fall days are also reported to illustrate the impact of external weather conditions and loop operating conditions on the thermal performance and energy consumption of the dual-loop data center test facility. The work presented highlights the impact of various operating conditions in influencing the cooling energy use and improving data center energy efficiency in chiller-less, ambient-air cooled data center designs using water cooled servers.

Embedded Two-Phase Cooling of Large 3D Compatible Chips With Radial Channels

Thermal performance for embedded two phase cooling using dielectric coolant (R1234ze) is evaluated on a ∼20 mm × 20 mm large die. The test vehicles incorporate radial expanding channels with embedded pin fields suitable for through-silicon-via (TSV) interconnects of multi-die stacks. Power generating features mimicking those anticipated in future generations of processor chips with 8 cores are included. Initial results show that for the types of power maps anticipated, critical heat fluxes in “core” areas of at least 350 W/cm2 with at least 20 W/cm2 “background” heating in rest of the chip area can be achieved with less than 30 °C temperature rise over the inlet coolant temperature. These heat fluxes are significantly higher than those seen for relatively long parallel channel devices of similar base channel dimensions. Experimental results of flow rate, pressure drop, “device,” and coolant temperature are also provided for these test vehicles along with details of the test facility developed to properly characterize the test vehicles.Copyright

Experimental investigation of water cooled server microprocessors and memory devices in an energy efficient chiller-less data center

Understanding and improving the thermal management and energy efficiency of data center cooling systems is of growing importance from a cost and sustainability perspective. Toward this goal, warm liquid cooled servers were developed to enable highly energy efficient chiller-less data centers that utilize only “free” ambient environment cooling. This approach greatly reduces cooling energy use, and could reduce data center refrigerant and make up water usage. In one exemplary experiment, a rack having such liquid cooled servers was tested on a hot summer day (~32°C) with CPU exercisers and memory exercisers running on every server to provide steady heat dissipation from the processors and from the DIMMs, respectively. Compared to a typical air cooled rack, significantly lower DIMM temperatures and CPU thermal values were observed.

Local Measurements of Flow Boiling Heat Transfer on Hot Spots in 3D Compatible Radial Microchannels

Hot spots and temperature non-uniformities are critical thermal characteristics of current high power electronics and future three dimensional (3D) integrated circuits (ICs). Experimental investigation to understand flow boiling heat transfer on hot spots is required for any two-phase cooling configuration targeting these applications. This work investigates hot spot cooling utilizing novel radial microchannels with embedded pin arrays representing through-silicon-via (TSV) interconnects. Inlet orifices were designed to distribute flow in radial channels in a manner that supplies appropriate amounts of coolant to high-power-density cores. Specially designed test vehicles and systems were used to produce non-uniform heat flux profiles with nominally 20 W/cm2 background heating, 200 W/cm2 core heating and up to 21 W/mm2 hot spot (0.2 mm × 0.2 mm) heating to mimic a stackable eight core processor die (20 mm × 20 mm) with two hot spots on each core. The temperatures associated with flow boiling heat transfer at the hot spots were locally measured by resistance temperature detectors (RTDs) integrated between the heat source and sink. At nominal pressure and flow conditions, use of R1234ze in these devices resulted in a maximum hot spot temperature (Ths) of under 63 °C and average Ths of 57 °C at a hot spot power density of 21 W/mm2. A semi-empirical model was used to calculate the equivalent heat transfer rate around the hot spots which can provide a baseline for future studies on local thermal management of hot spots.Copyright

Extreme energy efficiency using water cooled servers inside a chiller-less data center

The paper summarizes part of a project that was undertaken to develop highly energy efficient warm liquid cooled servers for use in chiller-less data centers that could save significant data center energy use and reduce data center refrigerant and make up water usage. One of the key concepts developed as part of this project is the Dual-Enclosure-Liquid-Cooling (DELC), which comprises of a 100% liquid cooled server cabinet and an outdoor dry cooler unit for heat rejection to the ambient and this configuration is used to reject the Information Technology (IT) equipment heat load directly to the outside ambient air without the use of a chiller. Demonstration hardware for server liquid cooling and a chiller-less data center was built and is operational for a 15 kW rack fully populated with liquid cooled servers which has been designed for use for up to 45°C liquid coolant to the rack. The anticipated benefits of such energy-centric configurations are significant energy savings of as much as 25% at the data center level. This paper builds on recent work that focused on the server liquid cooling, the rack enclosure with heat exchanger cooling and liquid distribution, and the data center level cooling infrastructure and which also presented sample data from experiments in support of the DELC concept. This paper presents experimental data related to the novel data center loop in a new manner, which is used to create a simplified thermodynamic model using curve-fit of surfaces of heat exchanger approach temperatures and power use of cooling devices. The model is validated using experimental data for a 22 hour test that was conducted in August of 2011. Subsequent to model validation, the simplified model is then used to make projections for DELC prototype performance (thermal and energy) under different conditions including different simple control schemes and weather conditions in the US. Weather data from nine different US cities is analyzed for a single day in August and realizable energy and energy cost savings over traditional chiller based data center cooling designs are presented. The results show that the new innovative data center cooling configuration presented could reduce cooling energy use to be less than 3.5% of the IT power for most US locations even in warm summer times of the year.

Multicore fiber 4 TX + 4 RX optical transceiver based on holey SiGe IC

A novel optical transceiver with 4 transmitter plus 4 receiver channels designed for coupling to multicore multimode fiber has been fabricated and characterized. The transceiver is based on the holey Optochip concept where 4-channel VCSEL and photodiode arrays are flip-chip attached to a single-chip SiGe IC using AuSn solder. Optical vias (holes) are fabricated into the SiGe IC to enable optical access to the conventional topside emitting 850-nm optoelectronic arrays. The optoelectronic arrays are arranged in a quad-VCSEL and quad-photodiode configuration where the 4 devices are on a 2 × 2 array on a dense 50-μm pitch. The transceiver module is completed by flip-chip soldering the Optochip onto a 8 mm × 8 mm high-speed high-density organic carrier. Optical access through the backside of the IC is provided through 2 optical vias. Electrical I/O is supplied through BGA pads on 0.8 mm pitch at the bottom of the module. High-speed characterization was carried out between 2 modules soldered to test cards, a transmitter (TX) and a receiver (RX) module. Each of the 4 optical outputs from the TX Optochip was coupled into a MMF and directed to individual photodiodes in the RX module. Eye-diagrams were measured for TX outputs as well as TX-to-RX links at data rates 20 Gb/s to 42 Gb/s. The 4 optical links operate error free up to 40 Gb/s, achieving a record data rate for multimode parallel optical transceivers.

Experimental investigation of direct attach microprocessors in a Liquid-Cooled chiller-less Data Center

As part of a US Department of Energy cost shared grant, a liquid cooled chiller-less data center test facility was designed and constructed with the goal of reducing total cooling energy use to less than 5% of the total IT and facilities energy usage by utilizing warm water cooling of the electronic rack. A server compatible Liquid Metal Thermal Interface (LMTI) [1] was developed and integrated to improve the thermal conduction path of the hot server components to the ambient cooling of the data center. This LMTI has a thermal resistance an order of magnitude better than that achieved with most commercially utilized thermal interface materials (TIMs). When integrated directly between a bare die and a water cooled heat sink, this technology achieved a significant improvement in thermal conduction and enabled the computer devices to operate in a much higher ambient temperature environment. Initial studies on single modules showed substantial improvement in operating temperature when utilizing LMTI. Based upon this result, a detailed study was completed using two liquid cooled System X 3550 servers, comparing the thermal performance between the commercial thermal solution of a standard lidded module interfaced with a thermal grease to a cold plate, and the solution where the lid was removed and LMTI was used between the bare die and the same cold plate. The servers were first characterized using bench top investigation and then in a Data Center Liquid Cooled System with the standard lidded module and subsequently reassembled with a direct die attach LMTI. The servers CPU core temperatures showed a 5 to 6 °C advantage in CPU core temperature for the direct attach LMTI compared to the standard lidded module with thermal grease.