Frank E. Bosco

Packaging design of the IBM system z10 enterprise class platform central electronic complex

The IBM System z10™ Enterprise Class mainframe addresses the modern data center requirements for minimizing floor space while increasing computing power efficiency. These objectives placed challenges on the z10™ packaging design as a result of significantly increased demand on system packaging density, power delivery, and logic and power cooling efficiency compared with the recent IBM System z9® and z990 mainframe generations. Several innovations were implemented to successfully meet these challenges: a more powerful multichip module (MCM) that delivers denser computing capability and a 64-way system; a vertically mated processor unit (PU) book structure that achieves a more efficient thermal implementation and a higher signal bandwidth between processors; and a PU book-centric dc-dc power delivery design that is more efficient. This paper presents the key elements to achieve this design: the novel mechanical load transmission paths and the connector technologies for the MCM, PU book, I/O, and power regulation components; an innovative cooling and thermal design that includes component-level tolerance of failures; and improved power delivery and power code developments to maximize the overall z10 compute efficiency.

A power, packaging, and cooling overview of the IBM eServer z900

This paper provides an overview of the power, packaging, and cooling aspects of the IBM eServer z900 design. The semiconductor processor chips must be supported and protected in a mechanical structure that has to provide electrical interconnects while maintaining the chip junction temperature within specified limits. The mechanical structure should be able to withstand shock and vibrations during transportation or events such as earthquakes. The processor chips require electrical power at well-regulated voltages, unaffected by the ac-line voltage and load current fluctuations. The acoustical and electromagnetic noise produced by the hardware must be within the limits set by national regulatory agencies, and the electronic operations must be adequately protected from disruption caused by electromagnetic radiation. For high availability, the power, packaging, and cooling hardware must have redundancy and the ability to be maintained while the system is operating. This paper first overviews the packaging hardware, followed by a description of the first- and second-level packaging, which includes the mother board and the multichip module. Thermal management is discussed from the point of view of both the multichip module and the overall system. Power conversion, management, and distribution are presented next. Finally, the design aspects involved with meeting the requirements of electromagnetic compatibility, acoustics, and immunity to shock, vibration, and earthquakes are discussed.

Overview of IBM zEnterprise 196 I/O subsystem with focus on new PCI express infrastructure

IBM zEnterprise® 196 introduces a new input/output (I/O) s5ubsystem, including a new I/O drawer that is largely based on a greatly expanded exploitation of industry-standard high-volume PCI Express® (PCIe®) links and switches. The System z® qualities of reliability, availability, and serviceability (RAS) are preserved and enhanced by combining the PCIe RAS capabilities with new System z capabilities. PCIe ports connecting the processor book to the I/O drawer are provided by a new IBM-designed PCIe fan-out card. This fan-out card and its firmware (Licensed Internal Code) support both traditional System z I/O and new I/O paradigms. In the new PCIe I/O drawer, PCIe switches provide fan-out and the well-established System z I/O failover function referred to as redundant I/O interconnect. This is the third generation of the I/O drawer/cage to be used in System z platforms. The PCIe I/O drawer design is extremely compact and provides enhanced I/O port granularity and density. It has been designed to provide performance extendibility for future I/O advancements. Traditional I/O such as FICONA, Fibre Channel Protocol, and Ethernet are provided with enhanced functionality and are packaged in this new PCIe I/O drawer. The advent of this new infrastructure opens up the possibility of attaching native PCIe adapters while allowing them to be controlled by system firmware or by the operating systems directly.

Electronic packaging of the IBM System z196 enterprise-class server processor cage

In this paper, we describe the first- and second-level system packaging structure of the IBM zEnterprise® 196 (z196) enterprise-class server. The design point required a more than 50% overall increase in system performance (in millions of instructions per second) in comparison to its predecessor. This resulted in a new system design that includes, among other things, increased input/output bandwidth, more processors with higher frequencies, and increased current demand of more than 2,000 A for the six processor chips and two cache chips per multichip module. To achieve these targets, we implemented several new packaging technologies. The z196 enterprise-class server uses a new differential memory interface between the processor chips and custom-designed server memory modules. The electrical power delivery system design follows a substantially new approach using Vicor Factor Power® blocks, which results in higher packaging integration density and minimized package electrical losses. The power noise decoupling strategy was changed because of the availability of deep-trench technology on the new processor chip generation.

Packaging the IBM eServer z990 central electronic complex

The z990 eServerTM central electronic complex (CEC) houses four multichip-module-based processor units instead of one, as in the previous-generation z900 eServer. The multichip module (MCM) input/output pin density in z990 processor units is more than twice that of the MCMs in z900 processor units. This increase in packaging density and the consequent tripling of the current drawn by the processor units were accommodated by the first-time use of land grid array (LGA) MCM-to-board interconnections in an IBM zSeries® eServer. This was done by using innovative refrigeration cooling of the MCM with air cooling as backup, and by a new mechanical packaging and power distribution scheme. This paper describes the mechanical engineering of the CEC cage, the LGA MCM-to-board interconnection scheme, and the mechanical isolation of the MCM evaporator-heat-sink mass from the LGA contacts. The paper also describes the electrical power and the cooling solutions implemented to meet the more demanding requirements of the denser CEC package.

IBM zEnterprise energy management

Data centers are facing serious energy challenges. Increasing energy costs make the operation and cooling of servers more significant cost factors. Furthermore, improvements in technology have led to processor chips and systems with rapidly increasing power density. The resulting power consumption and cooling requirements of these systems are pushing many existing data centers to the limits of their power distribution capability and cooling capacity. Improvements in energy efficiency and management are needed at the chip and the system level to counteract this trend. This paper provides a comprehensive description of the hardware and firmware improvements implemented with IBM zEnterprise® 196 to stop the growth of and even reduce its energy footprint compared with previous IBM System z® servers. These include more power-efficient chips; power conversion and distribution; new sensors; cooling control firmware; new energy management functions; integrated hybrid energy management for power saving and power capping across the whole hybrid system; and data center energy-efficiency improvements resulting from options for water cooling, high-voltage DC (HVDC) power, and overhead cabling.

Mechanical packaging, power, and cooling design for the IBM z13

The system-level packaging of the IBM z13™ supports the implementation of a new drawer-based Central Processor Complex (CPC). Departing from previous IBM z Systems™ designs, the introduction of distributed land-grid-array (LGA) attached single-chip modules (SCMs) required new mechanical, power, and cooling designs to address specified performance requirements and to provide enhanced reliability, availability, and serviceability (RAS) attributes. Building upon the designs created for the IBM zEnterprise® BC12 (zBC12), new CPC drawer and frame mechanical designs were created to significantly increase overall packaging density. Similar to its predecessor, the IBM zEnterprise EC12 (zEC12), the z13 utilizes water-cooling of the processors, but in contrast to the single input and return flow used to cool the multi-chip module (MCM) in the zEC12, the z13 accomplishes its processor cooling using a flexible hose internal manifold design that provides parallel input and return fluid flow to each SCM. The use of flexible hose also enabled SCM field replacement, new to high-end IBM z Systems. A new internal cooling loop unit and an updated external (building-chilled) modular water-conditioning unit were designed utilizing customized water delivery manifold systems to feed the common CPC drawer design. Revised power delivery and service control structures were also created to address the distributed nature of the z13 system design.