Mark Seymour

Experimental and Numerical Characterization of a Raised Floor Data Center Using Rapid Operational Flow Curves Model

As the number of data centers is exponentially growing globally, pragmatic characterization schemes are considered to be a necessity for measuring and modeling the load capacity and flow pattern of the facility. This paper contains experimental and numerical characterization of a new data center laboratory using practical measurements methods, including tiles and CRAH flow measurements. Then a full physics based CFD model is built to simulate/predict the measured data. A rapid flow curve method is used showing high accuracy and low computational expense.Detailed descriptions of the data center structure, dimensions, layout (Appendix, A-1) and flow devices are given. Also, modeling parameters are mentioned in details to provide a baseline for any investigative parametric or sensitivity studies. Four experimental room level flow constraint scenarios are applied at which measurements were taken, (Appendix, A-2). The model is then built and calibrated then used to predict measurements.Measurements of the cooling unit were performed using hot wire anemometry with a traverse duct installed at the top of the CRAH. The tiles measurements were carried out using a flow hood with back pressure compensation. A detailed CFD model is constructed to predict the four experimental cases. For modeling the interdependency between the flow and pressure in flow devices flow curve approach is used. This is a rapid modeling technique that relies on experimentally measured (for IT) or approximated (for CRAH) flow curves. Applying the operational flow curves boundary conditions at the vents of the flow device results in a very accurate simulation model. It is also shown that the flow curves can be used to predict the real-time flow rate of servers at known RPM. This greatly simplifies flow rate measurements of IT in the data center.Copyright

Experimentally Validated Numerical Model of a Fully-Enclosed Hybrid Cooled Server Cabinet

Because of the rapid growth in the number of data centers combined with the high density heat dissipation in the IT and telecommunications equipment, energy efficient thermal management of data centers has become a key research focus in the electronics packaging community. Traditional legacy data centers still rely largely on chilled air flow delivered to the IT equipment racks through perforated tiles from the raised floor plenum. When there is large variation in the amount of heat dissipated by the racks in a given aisle, the standard air cooling approach requires over-provisioning.Localized hybrid air-water cooling is one approach to more effectively control the cooling when there is wide variation in the amount of dissipation in neighboring racks. In a closed hybrid air-water cooled server cabinet, the generated heat is removed by a self-contained system that does not interact with the room level air cooling system. In this study, a comprehensive procedure for CFD validation in a close coupled hybrid cooled enclosed cabinet is described. The commercial enclosure has been characterized experimentally in an earlier study, where the effectiveness values were applied as boundary conditions to the compact heat exchanger model.Here, the previously obtained experimental data are used to validate the results from computational modeling. Two cases with different air flow rates are compared. Very good agreement is achieved, with the maximum overall average error less than 4%. Due to relatively high pressure inside the cabinet, it is possible that air leakage from the cabinet may be responsible for the discrepancy between the model and experimental results. A sensitivity study was applied to the validated model to investigate the effect leakage had on the cabinet’s performance.Copyright

Chip to Chiller Experimental Cooling Failure Analysis of Data Centers: The Interaction Between IT and Facility

Cooling failure in data centers (DCs) is a complex phenomenon due to the many interactions between the cooling infrastructure and the information technology equipment (IT). To fully understand it, a system integration philosophy is vital to the testing and design of experiment. In this paper, a facility-level DC cooling failure experiment is run and analyzed. An airside cooling failure is introduced to the facility during two different cooling set points as well as in open and contained environments. Quantitative instrumentation includes pressure differentials, tile airflow, external contour and discrete air inlet temperature, intelligent platform management interface (IPMI), and cooling system data during failure recovery. Qualitative measurements include infrared imaging and airflow visualization via smoke trace. To our knowledge of current literature, this is the first experimental study in which an actual multi-aisle facility cooling failure is run with real IT (compute, network, and storage) load in the white space. This will establish a link between variations from the facility to the central processing unit (CPU). The results show that using the external IT inlet temperature sensors, the containment configuration shows a longer available uptime (AU) during failure. However, the IPMI data show the opposite. In fact, the available uptime is reduced significantly when the external sensors are compared to internal IT analytics. The response of the IT power, CPU temperature, and fan speed shows higher values during the containment failure. This occurs because of the instantaneous formation of external impedances in the containment during failure, which renders the contained aisle to be less resilient than the open aisle. The tradeoffs between PUE, OPEX, and AU are also explained.

Ranking and Optimization of CAC and HAC Leakage Using Pressure Controlled Models

In cold aisle containment (CAC) the supply of cold air is separated within the contained volume. The hot air exhaust leaves the IT and increases the room’s temperature before returning to the cooling unit. On the other hand, hot aisle containment (HAC) generates a cooler environment in the data center room as a whole by segregating hot air within the containment. Hot air is routed back to the cooling unit return by a drop ceiling or a chimney. Each system has different characteristics and airflow paths. For instance, leakage introduces different effects for CACs and HACs since the hot and cold aisles are switched.This article utilizes data center measurements and containment characterization carried out circa April 2015 in the ES2 Data Center lab at Binghamton University. Details on the containment model include leakages at below racks, above racks, below CAC doors, between doors, and above doors. The model deploys the experimentally obtained flow curves approach for flow-pressure correlation.Data center operators rely on the pressure differential to measure how much the IT is provided. Hence, in this study the level of provisioning was expressed in terms of pressure differentials between the hot and cold aisles. In this manner the model reflected real-life DC thermal management practices. This was done by integrating a pressure differential based controller to the cooling unit model. Leakages in each system are quantified and ranked based on a proposed LIF (Leakage Impact Factor) metric.The LIF describes the transport contribution each leakage location has. This metric can be used by containment designers and data center operators to prioritize their sealing efforts or consider deploying the containment solution differently. Finally, a systematic approach is shown in which containment models can be used to optimize operations at the real-life site.Copyright

Steady State and Transient Comparison of Perimeter and Row-Based Cooling Employing Controlled Cooling Curves

The perpetual increase of data processing has led to an ever increasing need for power and in turn to greater cooling challenges. High density (HD) IT loads have necessitated more aggressive and direct approaches of cooling as opposed to the legacy approach by the utilization of row-based cooling. In-row cooler systems are placed between the racks aligned with row orientation; they offer cool air to the IT equipment more directly and effectively. Following a horizontal airflow pattern and typically occupying 50% of a rack’s width; in-row cooling can be the main source of cooling in the data center or can work jointly with perimeter cooling. Another important development is the use of containment systems since they reduce mixing of hot and cold air in the facility. Both in-row technology and containment can be combined to form a very effective cooling solution for HD data centers.This current study numerically investigates the behavior of in-row coolers in cold aisle containment (CAC) vs. perimeter cooling scheme. Also, we address the steady state performance for both systems, this includes manufacturer’s specifications such as heat exchanger performance and cooling coil capacity.A brief failure scenario is then run, and duration of ride through time in the case of row-based cooling system failure is compared to raised floor perimeter cooling with containment. Non-raised floor cooling schemes will reduce the air volumetric storage of the whole facility (in this small data center cell it is about a 20% reduction). Also, the varying thermal inertia between the typical in-row and perimeter cooling units is of decisive importance.The CFD model is validated using a new data center laboratory at Binghamton University with perimeter cooling. This data center consists of one main Liebert cooling unit, 46 perforated tiles with 22% open area, 40 racks distributed on three main cold aisles C and D. A computational slice is taken of the data center to generalize results. Cold aisle C consists of 16 rack and 18 perforated tiles with containment installed. In-row coolers are then added to the CFD model. Fixed IT load is maintained throughout the simulation and steady state comparisons are built between the legacy and row-based cooling schemes. An empirically obtained flow curve method is used to capture the flow-pressure correlation for flow devices.Performance scenarios were parametrically analyzed for the following cases: (a) Perimeter cooling in CAC, (b) In-row cooling in CAC. Results showed that in-row coolers increased the efficiency of supply air flow utilization since the floor leakage was eliminated, and higher pressure build up in CAC were observed. This reduced the rack recirculation when compared to the perimeter cooled case. However, the heat exchanger size demonstrated the limitation of the in-row to maintain controlled set point at increased air flow conditions. For the pump failure scenario, experimental data provided by Emerson labs were used to capture the thermal inertia effect of the cooling coils for in-row and perimeter unit, perimeter cooled system proved to have longer ride through time.Copyright

Management and predictions of operational changes and growth in mission critical facilities

The main challenge in understanding the cooling performance in a legacy data center is the invisible transport medium (air). This emphasizes the need for smart and meticulous measurement techniques. However, the nature of measurements is finite (e.g. top, middle and bottom) and the variation between often non-linear, therefore to capture the gradients between measured points a validated CFD simulation is needed. In this paper a brief description of an experimental characterization of a new data center lab is discussed. Airflow and temperature measurements are utilized to understand the facilitys performance at different operational stages, until reaching the designed capacity. Since the facility houses a wide range of different IT equipment (servers, switches, storage and blades), it is important to understand the airflow demand of each. To do that, each type of IT was tested separately and flow characteristics were obtained (i.e. free delivery, critical pressure and flow curves). In the second part, all the characterization data is integrated via compact models into a full CFD simulation. The measured points are used for validation and the full field of air temperature and flow is resolved. Both the measurements and simulation data will now be used to answer important design, deployment and operational change questions.

Experimental characterization of a cold plate used in warm water cooling of data centers

Recent advancements in microelectronics packaging and fabrication have resulted in high heat flux densities in data center server components. Liquid cooling is increasingly replacing air cooling in data centers because of its high heat carrying capacity. It also provides an energy efficient way to transport heat from processor as compared to air cooling using Computer Room Air Conditioning (CRAC). This study presents the results of a bench level experiment to characterize a commercially available cold plate. The cold plate under consideration is used in Direct Liquid Cooling (DLC) application in data center cooling. Thermal characterization of cold-plate is necessary in order to develop a fundamental understanding of its energy transport which would enable researchers to improve the overall energy-efficiency; reliability and usability of warm water cooling in data centers. The temperature rise (ΔT) across the cold plate and the cold plate surface temperature are measured for various coolant flow rate and chip power. The results are presented in the form of thermal resistance curve. A close estimation of heat transfer coefficient values is then obtained from the resistance values using well-established relations.

Is energy efficiency enough? Filling the engineering gap in data center design and operation

The data center industry is focused on improving the energy efficiency of modern data centers due to their increasing energy costs and consumption, and the consequent carbon emissions. However, there is a lack of adopted process or metrics for data center cooling performance; this potentially puts facilities at risk from the unseen consequences of focusing on energy efficiency alone. The challenge for a data center operator is how to assess cooling performance. The facilitys design likely makes assumptions about what IT equipment will be installed and how it will be configured that are not reflected in the operational configuration. This paper uses engineering simulation based on computational fluid dynamics (CFD) to show the relationship between the quantity of IT equipment that may be safely installed and the efficiency of the cooling system. Both are affected by changes to the cooling system settings and any IT equipment/applications deployed. The IT heat load and airflow requirement varies in legacy/enterprise style data centers and virtualized/cloud data centers alike, and this affects the cooling requirement. Using engineering simulation to predict the consequences of change provides a valuable complementary tool to the operator: it gives insight on how to avoid lost capacity, and helps them to better configure their facility when making decisions on infrastructure, IT deployment, or - in a virtualized environment - application deployment.

On reliability and uptime of IT in contained solution

In this investigation, we utilize a new method of characterization in which IT airflow systems are ranked from a 1RU ToR switches, xRU servers to 9RU Blade Center. The free delivery (FD) and the critical pressure (Pc) are unique properties of each IT airflow system, satisfying its active flow curve (AFC). Those curves are used in previously validated models to investigate the interaction of IT with different air systems in a confined space. A validated model of containment is studied of a CAC aisle from Binghamton University data center lab. It is shown that reverse flow can take place in an operating piece of IT with weaker air system; this can endanger IT reliability and up time at different data center events. This can also change previous conclusions about the ride through time in well-sealed fully contained systems. Another important finding is that design containment systems based on the assumption of uniform load in the aisle/cabinet is a misconception that can endanger facility operation. In general, the reduction in airflow rate upon events in the data center (DC) is a function of each IT air system. To our knowledge, this is the first time in literature where both recent models of IT active flow curves and black box thermal inertia are combined to yield realistic failure analysis of contained solutions.

Impact of elevated temperature on data center operation based on internal and external IT instrumentation

Todays data centers increasingly rely on environmental data collection and analysis to operate the cooling infrastructure as efficiently as possible and to maintain the reliability of IT equipment. This in turn emphasizes the importance of the quality of data collected and its relevance to the overall operation of the data center. This study presents an experiment based analysis and comparison of environmental and power data collection using two different approaches; one uses a discrete sensor network and smart PDUs, and another uses available data from the installed IT equipment (IPMI data). The comparison looks deeply into the effect of both approaches that are adopted to control data center cooling. In addition, the effect that the Supply Air Temperature (SAT) from the Computer Room Air Handler (CRAH) unit had on the IT equipment was investigated in fully sealed Cold Aisle Containment (CAC) with 100% CPU utilization. It can be observed that the difference between the discrete and IPMI inlet temperature of the IT equipment increased as SAT increased due to the IT fans increasing speed in an attempt to get more cooling and the resulting in negative pressure differential build up inside the containment. Furthermore, the authors identified a value of the supply air temperature at which IT equipment started to ramp up for both approaches of data center cooling and control. The novelty of this study may aid data center operators when making the decision of what monitoring or control scheme to use.