Joseph M. Hoke

IBM eServer z900 I/O subsystem

The IBM eServer z900 is the first in a generation of future eServers that continues its leadership via a new I/O subsystem with enhancements in capability, performance, configuration management, and qualities of service. Significant features of the I/O subsystem are included to support the 64-bit z/Architecture™ and configuration-management enhancements [e.g., assignable channel path identifiers (CHPIDs) and dynamic channel path management (DCM)]. A 1GB/s self-timed interface (STI), I/O infrastructure, and I/O card cage are described which support the high-bandwidth I/O (e.g., FICON™, Ethernet). These improvements yield enhanced configuration flexibility and connectivity, as well as reliability, availability, and serviceability (RAS), while providing for future bandwidth growth. The various types of I/O ports supported by the IBM eServer z900 platform are also discussed. A common I/O platform is discussed which has been used to provide a uniform, high-bandwidth attachment of industry-standard peripheral computer interface (PCI) cards, while maintaining the leadership functionality and RAS of the eServer zSeries™. A high-density (16-port) ESCON® I/O card has been designed by exploiting IBM advanced CMOS and state-of-the-art fiber optic technologies. Finally, a high-performance, high-density intersystem channel (ISC-3) coupling link I/O card has been developed for the IBM eServer z900 by leveraging advanced technology and packaging techniques.

Self-timed interface for S/390 I/O subsystem interconnection

A high-speed interface has been designed for interconnection of the S/390® I/O subsystem to the IBM S/390 G5 processor. The self-timed interface (STI) provides high bandwidth, greater communication distances, and simpler timing within the S/390 servers than traditional interfaces. The STI communicates between the memory bus adapter and the expanded S/390 I/O subsystem, which now includes the new S/390 fiber channel offering (FICON™) and other network-based protocols such as ATM, Fast Ethernet, and Gigabit Ethernet. Also new for G5 is the use of STI for the integrated cluster bus (ICB), providing direct links among multiple G5 servers. The STI communicates over cables up to ten meters in length at a clock frequency of 167 MHz. Data is sent at twice the clock frequency (333 MB/s). The hardware implementation comprises specially designed logic macros, differential drivers and receivers, and the cables and connectors. The receive macro accommodates up to three bittimes of skew by retiming each data bit of the interface to the transmitted clock. This paper describes the STI logic, the characteristics of the link, and the transmission and reception of the data (and clock).

The structure of chips and links comprising the IBM eServer z990 I/O subsystem

The performance of large servers is to a high degree determined by their I/O subsystems. In the z990 server, nearly all of the components in the I/O path have been considerably improved in performance, capability, and cost. A 2-GB/s enhanced self-timed interface (eSTI) was introduced which is capable of absorbing the ever-increasing data rates of modern high-speed adapters. The I/O bandwidth available from a single node (three memory bus adapter, or MBA, chips, each with four eSTI ports) now equals 48 GB/s. As a consequence, both the MBA chip and the STI multiplexer switch (STI switch) chip had to be completely redesigned. In addition to these two chips, this paper describes the eSTI design itself and the Sweep chip, which integrates the function of four bidirectional adapter chips, one switch chip, and a clock chip.

Advances in the IBM z13 I/O function and capability

The IBM z13™ introduces a number of significant improvements in the I/O subsystem relative to the IBM zEnterprise™ EC12. This includes advancing the I/O drawer infrastructure to implement generation 3 (Gen 3) of the Peripheral Component Interconnect Express® (PCIe®) links and switches. For the first time on this platform, the PCIe ports are placed directly within the IBM z13 processor itself, reducing latency by avoiding the necessity of using hub chips that have traditionally been implemented for routing and expansion of the PCIe infrastructure. Portions of the processor chip have been reserved to implement the functionality that would previously have resided on these separate hub chips. Traditional I/O features such as Fiber Connection (FICON®), Fibre Channel Protocol (FCP), Open System Adapter (OSA), Crypto Express, and Flash Express are provided with enhanced functionality in the PCIe I/O drawer. The PCIe infrastructure has created the opportunity to integrate additional native PCI adapters. This is further exploited with the use of Single Root I/O virtualization (SR-IOV) capability on a Remote Direct Memory Access over Converged Ethernet (RoCE) Express adapter, which now provides shared exploitation of the facilities within this adapter. An adapter utilizing field programmable gate arrays has been developed to provide compression acceleration in the z13 and forms a foundation for additional varieties of acceleration in the future.