Husam A. Alissa

Experimental and Numerical Characterization of a Raised Floor Data Center Using Rapid Operational Flow Curves Model

As the number of data centers is exponentially growing globally, pragmatic characterization schemes are considered to be a necessity for measuring and modeling the load capacity and flow pattern of the facility. This paper contains experimental and numerical characterization of a new data center laboratory using practical measurements methods, including tiles and CRAH flow measurements. Then a full physics based CFD model is built to simulate/predict the measured data. A rapid flow curve method is used showing high accuracy and low computational expense.Detailed descriptions of the data center structure, dimensions, layout (Appendix, A-1) and flow devices are given. Also, modeling parameters are mentioned in details to provide a baseline for any investigative parametric or sensitivity studies. Four experimental room level flow constraint scenarios are applied at which measurements were taken, (Appendix, A-2). The model is then built and calibrated then used to predict measurements.Measurements of the cooling unit were performed using hot wire anemometry with a traverse duct installed at the top of the CRAH. The tiles measurements were carried out using a flow hood with back pressure compensation. A detailed CFD model is constructed to predict the four experimental cases. For modeling the interdependency between the flow and pressure in flow devices flow curve approach is used. This is a rapid modeling technique that relies on experimentally measured (for IT) or approximated (for CRAH) flow curves. Applying the operational flow curves boundary conditions at the vents of the flow device results in a very accurate simulation model. It is also shown that the flow curves can be used to predict the real-time flow rate of servers at known RPM. This greatly simplifies flow rate measurements of IT in the data center.Copyright

Experimentally Validated Numerical Model of a Fully-Enclosed Hybrid Cooled Server Cabinet

Because of the rapid growth in the number of data centers combined with the high density heat dissipation in the IT and telecommunications equipment, energy efficient thermal management of data centers has become a key research focus in the electronics packaging community. Traditional legacy data centers still rely largely on chilled air flow delivered to the IT equipment racks through perforated tiles from the raised floor plenum. When there is large variation in the amount of heat dissipated by the racks in a given aisle, the standard air cooling approach requires over-provisioning.Localized hybrid air-water cooling is one approach to more effectively control the cooling when there is wide variation in the amount of dissipation in neighboring racks. In a closed hybrid air-water cooled server cabinet, the generated heat is removed by a self-contained system that does not interact with the room level air cooling system. In this study, a comprehensive procedure for CFD validation in a close coupled hybrid cooled enclosed cabinet is described. The commercial enclosure has been characterized experimentally in an earlier study, where the effectiveness values were applied as boundary conditions to the compact heat exchanger model.Here, the previously obtained experimental data are used to validate the results from computational modeling. Two cases with different air flow rates are compared. Very good agreement is achieved, with the maximum overall average error less than 4%. Due to relatively high pressure inside the cabinet, it is possible that air leakage from the cabinet may be responsible for the discrepancy between the model and experimental results. A sensitivity study was applied to the validated model to investigate the effect leakage had on the cabinet’s performance.Copyright

Experimental characterization of a Rear Door Heat exchanger with localized containment

In the current study, the thermal performance of a rear door heat exchanger is investigated experimentally. The experiments were conducted at the ES2 research data center lab at Binghamton University. A rear door heat exchanger was attached to an isolated equipment cabinet so that the operating conditions could be carefully controlled. On the water side of the heat exchanger, a control system provides for variation of the water flowrate by changing the pump speed. The water supply temperature is controllable by adjusting a three way valve that alters the mixture of fresh cold water with the heat exchanger return water. Both the water supply and return temperatures are monitored remotely. On the air side, a grid of 36 air velocity/temperature sensors installed on a portable frame enabled remote monitoring of the air stream properties. A combination of commercial servers and a 10kW server simulator were used to generate a controllable heat load. A single perforated tile with 65% openness delivers sufficient airflow from the raised floor plenum to the IT equipment. A localized containment system was added to the front of the cabinet in order to control the air inlet conditions to the equipment. The rear door heat exchanger was characterized for a range of different air and water flowrates along with different power generation levels. In addition, the impact of air leakage within the cabinet was studied in relation to the air flow provisioning level.

Chip to Chiller Experimental Cooling Failure Analysis of Data Centers: The Interaction Between IT and Facility

Cooling failure in data centers (DCs) is a complex phenomenon due to the many interactions between the cooling infrastructure and the information technology equipment (IT). To fully understand it, a system integration philosophy is vital to the testing and design of experiment. In this paper, a facility-level DC cooling failure experiment is run and analyzed. An airside cooling failure is introduced to the facility during two different cooling set points as well as in open and contained environments. Quantitative instrumentation includes pressure differentials, tile airflow, external contour and discrete air inlet temperature, intelligent platform management interface (IPMI), and cooling system data during failure recovery. Qualitative measurements include infrared imaging and airflow visualization via smoke trace. To our knowledge of current literature, this is the first experimental study in which an actual multi-aisle facility cooling failure is run with real IT (compute, network, and storage) load in the white space. This will establish a link between variations from the facility to the central processing unit (CPU). The results show that using the external IT inlet temperature sensors, the containment configuration shows a longer available uptime (AU) during failure. However, the IPMI data show the opposite. In fact, the available uptime is reduced significantly when the external sensors are compared to internal IT analytics. The response of the IT power, CPU temperature, and fan speed shows higher values during the containment failure. This occurs because of the instantaneous formation of external impedances in the containment during failure, which renders the contained aisle to be less resilient than the open aisle. The tradeoffs between PUE, OPEX, and AU are also explained.

Performance of Temperature Controlled Perimeter and Row-Based Cooling Systems in Open and Containment Environment

The continuous increase of data center usage is leading the industry to increase the load density per square foot of existing facilities. High density (HD) IT load per rack demands bringing the cooling source closer to the heat load in contrast to room level air cooling. For high density racks, the use of in-row cooling systems is becoming increasingly popular. In-row cooling can be the main source of cooling for a data center or work jointly with perimeter cooling in what is called a hybrid cooled room level system. Also, hot or cold aisle containment can be integrated with perimeter cooling and used throughout the data center to reduce the mixing of hot and cold air. Currently, there has not been much work comparing the performance of in-row cooling in open versus contained environments.The present work builds on a previous study where the interaction of perimeter and row-based cooling was evaluated for a cold-aisle containment (CAC) environment. Previously, the benefit of using row-based cooled in an aisle has not been compared with an aisle in open conditions. Here, we numerically investigate the performance of in-row coolers in both opened and cold-aisle contained environments. Groups of IT equipment that differ in air flow strength are used to provide the heat load. Empirically measured flow curves for common IT equipment are employed to provide simplified models of the IT equipment in the CFD software used. The steady state analysis includes information provided in the manufacturer’s specifications such as heat exchanger performance characteristics. The model was validated using a new data center laboratory with perimeter cooling. A single aisle of the data center is modeled to reduce the computational time, and the results are generalized. The cold aisle contains 16 racks of IT equipment distributed on both sides. In addition, the aisle contains 2 power distribution units. Full details are incorporated in the computational model. A single Liebert® CW114 CRAC unit provides the perimeter cooling in the data center. The model captures the particular air flow behavior in the cold aisle when row-based cooling is utilized. Correlations are derived to predict the ability of air cooling units to maintain set points at different air flow rates. The effect of leakage is also considered.Copyright

Ranking and Optimization of CAC and HAC Leakage Using Pressure Controlled Models

In cold aisle containment (CAC) the supply of cold air is separated within the contained volume. The hot air exhaust leaves the IT and increases the room’s temperature before returning to the cooling unit. On the other hand, hot aisle containment (HAC) generates a cooler environment in the data center room as a whole by segregating hot air within the containment. Hot air is routed back to the cooling unit return by a drop ceiling or a chimney. Each system has different characteristics and airflow paths. For instance, leakage introduces different effects for CACs and HACs since the hot and cold aisles are switched.This article utilizes data center measurements and containment characterization carried out circa April 2015 in the ES2 Data Center lab at Binghamton University. Details on the containment model include leakages at below racks, above racks, below CAC doors, between doors, and above doors. The model deploys the experimentally obtained flow curves approach for flow-pressure correlation.Data center operators rely on the pressure differential to measure how much the IT is provided. Hence, in this study the level of provisioning was expressed in terms of pressure differentials between the hot and cold aisles. In this manner the model reflected real-life DC thermal management practices. This was done by integrating a pressure differential based controller to the cooling unit model. Leakages in each system are quantified and ranked based on a proposed LIF (Leakage Impact Factor) metric.The LIF describes the transport contribution each leakage location has. This metric can be used by containment designers and data center operators to prioritize their sealing efforts or consider deploying the containment solution differently. Finally, a systematic approach is shown in which containment models can be used to optimize operations at the real-life site.Copyright

Steady State and Transient Comparison of Perimeter and Row-Based Cooling Employing Controlled Cooling Curves

The perpetual increase of data processing has led to an ever increasing need for power and in turn to greater cooling challenges. High density (HD) IT loads have necessitated more aggressive and direct approaches of cooling as opposed to the legacy approach by the utilization of row-based cooling. In-row cooler systems are placed between the racks aligned with row orientation; they offer cool air to the IT equipment more directly and effectively. Following a horizontal airflow pattern and typically occupying 50% of a rack’s width; in-row cooling can be the main source of cooling in the data center or can work jointly with perimeter cooling. Another important development is the use of containment systems since they reduce mixing of hot and cold air in the facility. Both in-row technology and containment can be combined to form a very effective cooling solution for HD data centers.This current study numerically investigates the behavior of in-row coolers in cold aisle containment (CAC) vs. perimeter cooling scheme. Also, we address the steady state performance for both systems, this includes manufacturer’s specifications such as heat exchanger performance and cooling coil capacity.A brief failure scenario is then run, and duration of ride through time in the case of row-based cooling system failure is compared to raised floor perimeter cooling with containment. Non-raised floor cooling schemes will reduce the air volumetric storage of the whole facility (in this small data center cell it is about a 20% reduction). Also, the varying thermal inertia between the typical in-row and perimeter cooling units is of decisive importance.The CFD model is validated using a new data center laboratory at Binghamton University with perimeter cooling. This data center consists of one main Liebert cooling unit, 46 perforated tiles with 22% open area, 40 racks distributed on three main cold aisles C and D. A computational slice is taken of the data center to generalize results. Cold aisle C consists of 16 rack and 18 perforated tiles with containment installed. In-row coolers are then added to the CFD model. Fixed IT load is maintained throughout the simulation and steady state comparisons are built between the legacy and row-based cooling schemes. An empirically obtained flow curve method is used to capture the flow-pressure correlation for flow devices.Performance scenarios were parametrically analyzed for the following cases: (a) Perimeter cooling in CAC, (b) In-row cooling in CAC. Results showed that in-row coolers increased the efficiency of supply air flow utilization since the floor leakage was eliminated, and higher pressure build up in CAC were observed. This reduced the rack recirculation when compared to the perimeter cooled case. However, the heat exchanger size demonstrated the limitation of the in-row to maintain controlled set point at increased air flow conditions. For the pump failure scenario, experimental data provided by Emerson labs were used to capture the thermal inertia effect of the cooling coils for in-row and perimeter unit, perimeter cooled system proved to have longer ride through time.Copyright

Numerical investigation of underfloor obstructions in open-contained data center with fan curves

The purpose of under floor plenum in a typical raised floor data center is to route the supply of cold air to perforated tiles in the cold aisles, and hence, to the racks. However, the presence of under floor chiller piping and various wiring may have an adverse effect on flow rates if not placed based on physical considerations; the pressure drop caused by chiller piping and under floor blockages has not been investigated thoroughly in modeling of a fully representative real life data center, and in particular, in applications where some or all of the cold aisles may be contained. This effect on flow rate is expected to be even more profound in contained systems; this study aims to address the effect of under floor obstructions on data center performance. It was shown that when having blockages in critical locations, containment can act as a solution for the inlet temperatures at the racks, however, the blockage effect can still be seen on the racks outlet and the CRAC return temperatures. It was also observed that inadequate distribution of those blockages led to a change in the operating point of both the CRAC and severs fan curve reducing the flow being fed to the IT equipment, and hence the chiller cooling load is expected to increase, which results in a thermal deficiency of the data center.

Steady-state and transient comparison of cold and hot aisle containment and chimney

Aisle containment is increasingly being used in data center cooling solutions because of the benefits of segregating the cold and hot air streams which reduces mixing and enhances energy savings. Containment can be integrated into legacy raised floor or non-raised floor local cooling systems such as in-row coolers. However, containment is not always entirely effective due to leakage. Although containment systems providing better sealing are becoming available, significant leakage exists in currently operational sites. The impact of leakage can be significant leading to backflow, the Venturi effect or entrainment. Comparisons between Cold Aisle Containment (CAC) and Hot Aisle Containment (HAC) indicate that there is a slight advantage of the latter in terms of ambient room temperature, failure scenarios and heat exchanger efficiency, although CAC systems can be easier to install. Detailed comparisons between both containment solutions are not widely available. Each containment approach has different behavior characteristics and air flow paths. For instance, leakage can introduce different flow and temperature conditions at server inlets depending on whether CAC or HAC is being used. An experimental characterization of containment was carried out at the ES2 research data center lab at Binghamton University [1] along with detailed numerical modeling. The numerical model was validated based on the experimentally measured flow rate of the perforated tiles in the cold aisles. Also, for the different types of IT equipment modeled in the simulations, experimentally measured flow impedance and fan curves were utilized. The details of the containment including leakage at the bottom and sides of the racks, the mounting rails, blanking panels and door seams were characterized experimentally and then incorporated in the computational model. Fan curves for the three CRAH blowers were used to accurately represent the targeted flow rates. This study focuses on three containment configurations: cold aisle containment and hot aisle containment and chimney (with a drop ceiling). In first two cases, the CRAH flow rate is controlled based on the pressure difference that was measured by three sensors inside and outside of the containment and for the third case, the CRAH flow rate was controlled based on IT required air flow. The water flow rate of the CRAH unit is also controlled based on the air return temperature and total heat load in the room so as to maintain the supply air temperature set point. The temperature and pressure fields in the aisles and at the server inlets and outlets are monitored for all the cases with leakage.

Management and predictions of operational changes and growth in mission critical facilities

The main challenge in understanding the cooling performance in a legacy data center is the invisible transport medium (air). This emphasizes the need for smart and meticulous measurement techniques. However, the nature of measurements is finite (e.g. top, middle and bottom) and the variation between often non-linear, therefore to capture the gradients between measured points a validated CFD simulation is needed. In this paper a brief description of an experimental characterization of a new data center lab is discussed. Airflow and temperature measurements are utilized to understand the facilitys performance at different operational stages, until reaching the designed capacity. Since the facility houses a wide range of different IT equipment (servers, switches, storage and blades), it is important to understand the airflow demand of each. To do that, each type of IT was tested separately and flow characteristics were obtained (i.e. free delivery, critical pressure and flow curves). In the second part, all the characterization data is integrated via compact models into a full CFD simulation. The measured points are used for validation and the full field of air temperature and flow is resolved. Both the measurements and simulation data will now be used to answer important design, deployment and operational change questions.