David Craddock

Overview of IBM zEnterprise 196 I/O subsystem with focus on new PCI express infrastructure

IBM zEnterprise® 196 introduces a new input/output (I/O) s5ubsystem, including a new I/O drawer that is largely based on a greatly expanded exploitation of industry-standard high-volume PCI Express® (PCIe®) links and switches. The System z® qualities of reliability, availability, and serviceability (RAS) are preserved and enhanced by combining the PCIe RAS capabilities with new System z capabilities. PCIe ports connecting the processor book to the I/O drawer are provided by a new IBM-designed PCIe fan-out card. This fan-out card and its firmware (Licensed Internal Code) support both traditional System z I/O and new I/O paradigms. In the new PCIe I/O drawer, PCIe switches provide fan-out and the well-established System z I/O failover function referred to as redundant I/O interconnect. This is the third generation of the I/O drawer/cage to be used in System z platforms. The PCIe I/O drawer design is extremely compact and provides enhanced I/O port granularity and density. It has been designed to provide performance extendibility for future I/O advancements. Traditional I/O such as FICONA, Fibre Channel Protocol, and Ethernet are provided with enhanced functionality and are packaged in this new PCIe I/O drawer. The advent of this new infrastructure opens up the possibility of attaching native PCIe adapters while allowing them to be controlled by system firmware or by the operating systems directly.

Advances in the IBM z13 I/O function and capability

The IBM z13™ introduces a number of significant improvements in the I/O subsystem relative to the IBM zEnterprise™ EC12. This includes advancing the I/O drawer infrastructure to implement generation 3 (Gen 3) of the Peripheral Component Interconnect Express® (PCIe®) links and switches. For the first time on this platform, the PCIe ports are placed directly within the IBM z13 processor itself, reducing latency by avoiding the necessity of using hub chips that have traditionally been implemented for routing and expansion of the PCIe infrastructure. Portions of the processor chip have been reserved to implement the functionality that would previously have resided on these separate hub chips. Traditional I/O features such as Fiber Connection (FICON®), Fibre Channel Protocol (FCP), Open System Adapter (OSA), Crypto Express, and Flash Express are provided with enhanced functionality in the PCIe I/O drawer. The PCIe infrastructure has created the opportunity to integrate additional native PCI adapters. This is further exploited with the use of Single Root I/O virtualization (SR-IOV) capability on a Remote Direct Memory Access over Converged Ethernet (RoCE) Express adapter, which now provides shared exploitation of the facilities within this adapter. An adapter utilizing field programmable gate arrays has been developed to provide compression acceleration in the z13 and forms a foundation for additional varieties of acceleration in the future.

Flash storage integration in the IBM System z EC12 I/O drawer

Flash storage is integrated for the first time on System z® as a card in the EC12 I/O drawer. This provides a number of functions and benefits in the immediate product, in addition to laying a foundation for further system benefits in future generations of System z systems. Enabling flash MLC (multilevel cell) technology as SCM (storage class memory) in an enterprise-class product required myriad diverse individual technological advances, together with a series of system design features. Extreme care and attention were paid to ensure that the required level of System z reliability was maintained. As with legacy I/O, the programming interface is subchannel-based. The subchannel programming interface is expanded with new architecture via the extended asynchronous-data-move facility. Operating system changes were required to enable exploitation of the features that this new system technology offers. These individual hardware, firmware, and software design aspects are described in this paper, along with the overall functionality and system-level value of this new technology.