Dustin W. Demetriou

Chip to Chiller Experimental Cooling Failure Analysis of Data Centers: The Interaction Between IT and Facility

Cooling failure in data centers (DCs) is a complex phenomenon due to the many interactions between the cooling infrastructure and the information technology equipment (IT). To fully understand it, a system integration philosophy is vital to the testing and design of experiment. In this paper, a facility-level DC cooling failure experiment is run and analyzed. An airside cooling failure is introduced to the facility during two different cooling set points as well as in open and contained environments. Quantitative instrumentation includes pressure differentials, tile airflow, external contour and discrete air inlet temperature, intelligent platform management interface (IPMI), and cooling system data during failure recovery. Qualitative measurements include infrared imaging and airflow visualization via smoke trace. To our knowledge of current literature, this is the first experimental study in which an actual multi-aisle facility cooling failure is run with real IT (compute, network, and storage) load in the white space. This will establish a link between variations from the facility to the central processing unit (CPU). The results show that using the external IT inlet temperature sensors, the containment configuration shows a longer available uptime (AU) during failure. However, the IPMI data show the opposite. In fact, the available uptime is reduced significantly when the external sensors are compared to internal IT analytics. The response of the IT power, CPU temperature, and fan speed shows higher values during the containment failure. This occurs because of the instantaneous formation of external impedances in the containment during failure, which renders the contained aisle to be less resilient than the open aisle. The tradeoffs between PUE, OPEX, and AU are also explained.

A Holistic Evaluation of Data Center Water Cooling Total Cost of Ownership

The generation-to-generation information technology (IT) performance and density demands continue to drive innovation in data center cooling technologies. For many applications, the ability to efficiently deliver cooling via traditional chilled air cooling approaches has become inadequate. Water cooling has been used in data centers for more than 50 years to improve heat dissipation, boost performance, and increase efficiency. While water cooling can undoubtedly have a higher initial capital cost, water cooling can be very cost effective when looking at the true life cycle cost of a water-cooled data center. This study aims at addressing how one should evaluate the true total cost of ownership (TCO) for water-cooled data centers by considering the combined capital and operational cost for both the IT systems and the data center facility. It compares several metrics, including return-on-investment for three cooling technologies: traditional air cooling, rack-level cooling using rear door heat exchangers, and direct water cooling (DWC) via cold plates. The results highlight several important variables, namely, IT power, data center location, site electric utility cost, and construction costs and how each of these influences the TCO of water cooling. The study further looks at implementing water cooling as part of a new data center construction project versus a retrofit or upgrade into an existing data center facility.

Understanding the True Total Cost of Ownership of Water Cooling for Data Centers

The generation-to-generation IT performance and density demands continue to drive innovation in data center cooling technologies. For many applications, the ability to efficiently deliver cooling via traditional chilled air cooling approaches has become inadequate. Water cooling has been used in data centers for more than 50 years to improve heat dissipation, boost performance and increase efficiency. While water cooling can undoubtedly have a higher initial capital cost, water cooling can be very cost effective when looking at the true lifecycle cost of a water cooled data center.This study aims at addressing how one should evaluate the true total cost of ownership for water cooled data centers by considering the combined capital and operational cost for both the IT systems and the data center facility. It compares several metrics, including return-on-investment for three cooling technologies: traditional air cooling, rack-level cooling using rear door heat exchangers and direct water cooling via cold plates. The results highlight several important variables, namely, IT power, data center location, site electric utility cost, and construction costs and how each of these influence the total cost of ownership of water cooling. The study further looks at implementing water cooling as part of a new data center construction project versus a retrofit or upgrade into an existing data center facility.Copyright

Development of an IT equipment lumped capacitance parameter database for transient data center simulations

The transient behavior of IT equipment can be represented by a lumped thermal capacitance for use in data center level transient thermal simulations. Previous work has proposed a mathematical methodology to extract the necessary lumped capacitance parameters, the servers time constant and heat transfer effectiveness, via air temperature measurements. This work describes and experimentally tests that proposed methodology to introduce a possible approach for developing an easy-to-use database of lumped capacitance characteristics for use in transient thermal simulations. The database will allow the user to select the appropriate transient parameters based on characteristics that do not require an “autopsy” of each-and-every server in the data center. Experiments are conducted to provide representative transient parameters which classify the servers by mass density and operating air flow rate. The paper describes the experimental methodology in detail to allow for the easy addition of other IT equipment or future server generations.

Combining cooling technology and facility design to improve HPC data center energy efficiency

As the science and engineering demand for high performance computing (HPC) grows beyond leading edge research institutions and communities to encompass routine activities of many disciplines, computing center infrastructures expand in their size, density and power demands. Long-practiced HPC center enhancements are increasingly demanding cooling methods that extend well beyond the capabilities of implementations that have become the staple of facility designs, even beyond those of the now-dominant architecture - commodity-processor-based, air-cooled rack clusters. As compute capacity aggressively increases in HPC centers, the power and cooling requirements, and thus cost of operation of the facilities, continues to rise correspondingly. This growth increasingly taxes the abilities of the hosting organizations to accommodate these demands. To address the challenge of meeting such pervasive demands, this paper examines energy efficiency in existing data centers from a two-pronged approach: employing direct water cooling and optimizing the facility infrastructure with as little capital investment to the building as possible.

Energy efficiency and reliability transformation at the IBM India Software Lab data center

This paper describes a case study on the adoption of innovative technologies that enabled energy efficiency, reliability and extended the life of the India Software Labs data center at IBM Bangalore. Essential to the transformation was the adoption of water cooling in nearly 50% of the data center including both at IT rack level and direct water cooling at the chip. The overall cooling architecture uses a high evaporator temperature, free-cooling chiller design to enable significant energy savings. The strategically designed ring plumbing architecture enables redundancy while allowing scalability with existing chilled water infrastructure reducing the dependency on perimeter cooling units which were reaching end of life conditions. The approach enabled a 48% increase in added IT equipment power capacity resulting in nearly 35,000 square feet of recovered data center foot print. The solution did this while increasing the data center infrastructure efficiency by over 15% and reducing the maximum data center temperature by nearly 10 degrees Celsius. To reduce the dependency on un-reliable power from the utility grid and expensive diesel generators, a photovoltaic-based high voltage direct current solar power system was implemented. The system used a unique multi-source power supply to enable intelligent, simultaneous access to grid power, solar power and backup battery power to supply 380 VDC power to IBM high-end servers. Using measurements of the solar power system output, and with government incentives, the solar power system shows a return-on-investment of less than 4 years.