Cyriel Minkenberg

Optical-packet-switched interconnect for supercomputer applications [Invited]

Feature Issue on Optical Interconnection Networks (OIN). We describe a low-latency, high-throughput scalable optical interconnect switch for high-performance computer systems that features a broadcast-and-select architecture based on wavelength- and space-division multiplexing. Its electronic control architecture is optimized for low latency and high use. Our demonstration system will support 64 nodes with a line rate of 40 Gbit/s per node and operate on fixed-length packets with a duration of 51.2 ns using burst-mode receivers. We address the key system-level requirements and challenges for such applications.

Current issues in packet switch design

Addressing the ever growing capacity demand for packet switches, current research focuses on scheduling algorithms or buffer bandwidth reductions. Although these topics remain relevant, our position is that the primary design focus for systems beyond 1 Tb/s must be shifted to aspects resulting from packaging disruptions. Based on trends such as increased link rates and improved CMOS technologies, we derive new design factors for such switch fabrics. For instance, we argue that the packet round-trip transmission time within the fabric has become a major design parameter. Furthermore, we observe that high-speed fabrics have become extremely dependent on serial I/O technology that is both high speed and high density. Finally, we conclude that in developing the architecture, packaging constraints must be put first and not as an afterthought, which also applies to solving the tremendous power consumption challenges.

A combined input and output queued packet switched system based on PRIZMA switch on a chip technology

A packet-switched system architecture based on the combination of a single-chip output-buffered switch element and input queues that sort arriving packets on a per-output-port basis is proposed. Scheduling is performed in a distributed two-stage approach. Independent arbiters at each of the inputs resolve input contention. Whereas the output-buffered switch element resolves output contention. As a result of this distribution of functionality, complexity of the input arbiters is only linearly proportional to the number of output ports N, thus offering better scalability than purely input-buffered approaches that require complex centralized schedulers. Since the input queues are used as the main buffering mechanism, only a relatively small amount of memory (on the order of N/sup 2/ packet locations) is required in the shared-memory switch, allowing high-throughput implementations. We present simulation results to demonstrate the high performance and robustness under bursty traffic achieved with the proposed system architecture. A practical implementation in the form of the PRIZMA family of switch chips is outlined, with emphasis on its versatility in scaling in terms of both port speed and number of ports, and its support for quality-of-service mechanisms.

A four-terabit packet switch supporting long round-trip times

This 4-TBPS packet switch uses a combined input- and crosspoint-queued (CICQ) structure with virtual output queuing at the ingress to achieve the scalability of input-buffered switches, the performance of output-buffered switches, and low latency.

Designing a Crossbar Scheduler for HPC Applications

A crucial part of any high-performance computing (HPC) system is its interconnection network. Corning and IBM are jointly developing a demonstration interconnect based on optical cell switching with electronic control. The Corning-IBM joint optical shared memory supercomputer interconnect system (Osmosis) project explores the opportunity to advance the role of optical-switching technologies in such systems. Key innovations in the scheduler architecture directly address the main HPC requirements: low latency, high throughput, efficient multicast support, and high reliability

Viable opto-electronic HPC interconnect fabrics

We address the problem of how to exploit optics for ultrascale High Performance Computing interconnect fabrics. We show that for high port counts these fabrics require multistage topologies regardless of whether electronic or optical switch components are used. Also, per stage electronic buffers remain indispensable for maintaining throughput, lossless-ness and packet sequence. Although the notion of true all-optical packet switching is not yet viable, we show that appropriate use of optical switching technology offers power and scaling advantages that can be leveraged economically, and propose a hybrid opto-electronic HPC interconnect fabric architecture that combines the strength of electronics in processing and storing information with the strength of optics in switching and transporting high bandwidths. Using Semiconductor Optical Amplifier technology, we are building a prototype demonstrator switch that we believe solves all the technical challenges. Having reached this threshold now enables commercialization of this technology, which we are currently pursuing.

Low-latency pipelined crossbar arbitration

Heuristic, parallel, iterative matching algorithms for input-queued cell switches with virtual output queuing require O(log N) iterations to achieve good performance. If the hardware implementation of the number of iterations required is not feasible within the cell duration, the matching process can be pipelined to obtain a matching in every cell time slot. However, existing approaches incur a substantial latency penalty due to the way the pipelining is performed, which renders them unattractive in latency-sensitive applications such as parallel computer interconnects. We introduce a new class of pipelined matching algorithms that can be based on any existing iterative matching algorithm, makes the minimum latency independent of the pipeline depth, and is highly amenable to distributed implementation. Our simulation results confirm that specific instances of this class achieve significantly lower average latency throughout the load range than existing schemes do. We also propose an instantiation of the scheme that, in addition, significantly improves the performance with nonuniform traffic.

On-the-Fly Adaptive Routing in High-Radix Hierarchical Networks

Dragonfly networks have been recently proposed for the interconnection network of forthcoming exascale supercomputers. Relying on large-radix routers, they build a topology with low diameter and high throughput, divided into multiple groups of routers. While minimal routing is appropriate for uniform traffic patterns, adversarial traffic patterns can saturate inter-group links and degrade the obtained performance. Such traffic patterns occur in typical communication patterns used by many HPC applications, such as neighbor data exchanges in multi-dimensional space decompositions. Non-minimal traffic routing is employed to handle such cases. Adaptive policies have been designed to select between minimal and nonminimal routing to handle variable traffic patterns. However, previous papers have not taken into account the effect of saturation of intra-group (local) links. This paper studies how local link saturation can be common in these networks, and shows that it can largely reduce the performance. The solution to this problem is to use nonminimal paths that avoid those saturated local links. However, this extends the maximum path length, and since all previous routing proposals prevent deadlock by relying on an ascending order of virtual channels, it would imply unaffordable cost and complexity in the network routers. In this paper we introduce a novel routing/flow-control scheme that decouples the routing and the deadlock avoidance mechanisms. Our model does not impose any dependencies between virtual channels, allowing for on-the-fly (in-transit) adaptive routing of packets. To prevent deadlock we employ a deadlock-free escape sub network based on injection restriction. Simulations show that our model obtains lower latency, higher throughput, and faster adaptation to transient traffic, because it dynamically exploits a higher path diversity to avoid saturated links. Notably, our proposal consumes traffic bursts 43% faster than previous ones.

Design issues in next-generation merchant switch fabrics

Packet-switch fabrics with widely varying characteristics are currently deployed in the domains of both communications and computer interconnection networks. For economical reasons, it would be highly desirable that a single switch fabric could accommodate the needs of a variety of heterogeneous services and applications from both domains. In this paper, we consider the current requirements, technological trends, and their implications on the design of an ASIC chipset for a merchant switch fabric. We then identify the architecture upon which such a suitable and generic switch fabric could be based, and we present the general characteristics of an implementation of this switching fabric within the bounds of current state-of-the-art technology. To our knowledge, this is the first attempt to design a chipset that can be used for both communications and computer interconnection networks.

Oblivious routing schemes in extended generalized Fat Tree networks

A family of oblivious routing schemes for Fat Trees and their slimmed versions is presented in this work. First, two popular oblivious routing algorithms, which we refer to as S-mod-k and D-mod-k, are analyzed in detail. S-mod-k is the default routing algorithm given as an example in the first works formally describing Fat Tree networks. D-mod-k has been independently proposed and investigated by several authors, who conclude in their evaluations that it achieves better performance than a random or adaptive routing approach. First, we identify the reasons why these algorithms perform well. Using this insight we extend these algorithms, originally intended for full bisection networks, to slimmed networks. Based on the lessons learned we propose a new generalized family of algorithms that provides a better oblivious solution than the existing ones for this class of networks. Moreover, this family extends the previous work from k-ary n-trees to the more general class of extended generalized fat trees.