Donald W. Porter

Hybrid cooling with cycle steering in the IBM eServer z990

The IBM eServerTM z990 introduces a new mode for cooling multichip processor modules that enables significantly more processors to be refrigerant-cooled than previously. In recent IBM zSeries® offerings, including G4, G5, G6, and z900, chip junctions in a single muhichip module (MCM) located in a central electronic complex (CEC) frame were cooled for reliability and performance benefits, using refrigerant technology, to temperatures lower than those achievable with air cooling. In the z990 system, a hybrid cooling approach is used, allowing refrigeration to be extended to four MCMs in a single CEC, which makes possible denser systems and greater power efficiency compared with prior modular refrigeration technologies used. In the event of a malfunction of the primary refrigeration cooling system, a backup air-cooling System is automatically engaged until the refrigeration problem is fixed. System sensors monitor the cooling state at all times. When air cooling is required, the chip circuit temperatures increase and the logic clocks are optimally adjusted to match the new junction temperatures. When refrigeration cooling is restored, the clocks are adjusted back to their fast speed. This technique allows the z990 system to match the processor density of direct-air-cooled systems while retaining a system performance and reliability benefit from refrigeration.

Packaging design of the IBM system z10 enterprise class platform central electronic complex

The IBM System z10™ Enterprise Class mainframe addresses the modern data center requirements for minimizing floor space while increasing computing power efficiency. These objectives placed challenges on the z10™ packaging design as a result of significantly increased demand on system packaging density, power delivery, and logic and power cooling efficiency compared with the recent IBM System z9® and z990 mainframe generations. Several innovations were implemented to successfully meet these challenges: a more powerful multichip module (MCM) that delivers denser computing capability and a 64-way system; a vertically mated processor unit (PU) book structure that achieves a more efficient thermal implementation and a higher signal bandwidth between processors; and a PU book-centric dc-dc power delivery design that is more efficient. This paper presents the key elements to achieve this design: the novel mechanical load transmission paths and the connector technologies for the MCM, PU book, I/O, and power regulation components; an innovative cooling and thermal design that includes component-level tolerance of failures; and improved power delivery and power code developments to maximize the overall z10 compute efficiency.

Mechanical packaging, power, and cooling design for the IBM z13

The system-level packaging of the IBM z13™ supports the implementation of a new drawer-based Central Processor Complex (CPC). Departing from previous IBM z Systems™ designs, the introduction of distributed land-grid-array (LGA) attached single-chip modules (SCMs) required new mechanical, power, and cooling designs to address specified performance requirements and to provide enhanced reliability, availability, and serviceability (RAS) attributes. Building upon the designs created for the IBM zEnterprise® BC12 (zBC12), new CPC drawer and frame mechanical designs were created to significantly increase overall packaging density. Similar to its predecessor, the IBM zEnterprise EC12 (zEC12), the z13 utilizes water-cooling of the processors, but in contrast to the single input and return flow used to cool the multi-chip module (MCM) in the zEC12, the z13 accomplishes its processor cooling using a flexible hose internal manifold design that provides parallel input and return fluid flow to each SCM. The use of flexible hose also enabled SCM field replacement, new to high-end IBM z Systems. A new internal cooling loop unit and an updated external (building-chilled) modular water-conditioning unit were designed utilizing customized water delivery manifold systems to feed the common CPC drawer design. Revised power delivery and service control structures were also created to address the distributed nature of the z13 system design.

An overview of the IBM zEnterprise EC12 processor cooling system

On September 19, 2012 IBM announced its latest System z Enterprise Class zServer, the IBM zEnterprise EC12 (zEC12). This server uses a 96 mm glass ceramic substrate to interconnect processors and related cache chips on a multi-chip module (MCM). In rare applications, the power in these MCMs can exceed 2000W, well beyond air cooling capability. This paper describes a new cooling methodology IBM employs in zEC12 to cool its processor MCMs. From the IBM S/390 G4, which first shipped in 1997, through z196 which is EC12s enterprise class predecessor, IBMs high end System z servers have utilized vapor compression refrigeration to cool its processor MCMs. In zEC12, the thermal solution employs an air to water heat exchanger to provide this function. This paper discusses the technical details of this cooling system. Thermal performance of each component of the cooling path from processor core to ambient, as well as comparison to prior cooling approaches in terms of temperatures, reliability, and energy efficiency will be reviewed. In summary, this technology shows considerable promise for cooling this class of server.