Claude Basso

A wire-speed power TM processor: 2.3GHz 45nm SOI with 16 cores and 64 threads

An emerging data-center market merges network and server attributes into a single wire-speed processor SoC. These processors are not network endpoints that consume data, but inline processors that filter or modify data and send it on. Wire-speed processors merge attributes from 1) network processors: many threaded low power cores, accelerators, integrated network and memory I/O, smaller memory line sizes and low total power, and from 2) server processors: full ISA cores, standard programming models, OS and hypervisor support, full virtualization and server RAS & infrastructure. Typical applications are edge-of-network processing, intelligent I/O devices in servers, network attached appliances, distributed computing, and streaming applications.

SCOC: High-radix switches made of bufferless clos networks

In todays datacenters handling big data and for exascale computers of tomorrow, there is a pressing need for high-radix switches to economically and efficiently unify the computing and storage resources that are dispersed across multiple racks. In this paper, we present SCOC, a switch architecture suitable for economical IC implementation that can efficiently replace crossbars for high-radix switch nodes. SCOC is a multi-stage bufferless network with O(N2/m) cost, where m is a design parameter, practically ranging between 4-16. We identify and resolve more than five fairness violations that are pertinent to hierarchical scheduling. Effectively, from a performance perspective, SCOC is indistinguishable from efficient flat crossbars. Computer simulations show that it competes well or even outperforms flat crossbars and hierarchical switches. We report data from our ASIC implementation at 32 nm of a SCOC 136×136 switch, with shallow buffers, connecting 25 Gb/s links. In this first incarnation, SCOC is used at the spines of a server-rack, fat-tree network. Internally, it runs at 9.9 Tb/s, thus offering a speedup of 1.45 ×, and provides a fall-through latency of just 61 ns.

Occupancy Sampling for Terabit CEE Switches

One consequential feature of Converged Enhanced Ethernet (CEE) is loss lessness, achieved through L2 Priority Flow Control (PFC) and Quantized Congestion Notification (QCN). We focus on QCN and its effectiveness in identifying congestive flows in input-buffered CEE switches. QCN assumes an idealized, output-queued switch, however, as future switches scale to higher port counts and link speeds, purely output-queued or shared-memory architectures lead to excessive memory bandwidth requirements, moreover, PFC typically requires dedicated buffers per input. Our objective is to complement PFCs coarse per-port/priority granularity with QCNs per-flow control. By detecting buffer overload early, QCN can drastically reduce PFCs side effects. We install QCN congestion points (CPs) at input buffers with virtual output queues and demonstrate that arrival-based marking cannot correctly discriminate between culprits and victims. Our main contribution is occupancy sampling (QCN-OS), a novel, QCN-compatible marking scheme. We focus on random occupancy sampling, a practical method not requiring any per-flow state. For CPs with arbitrarily scheduled buffers, QCN-OSis shown to correctly identify congestive flows, improving buffer utilization, switch efficiency, and fairness.

ATM: paving the information superhighway

A new generation of networking requirements is fueling the growth of a cell-based communications technology known as the asynchronous transfer mode (ATM). ATM technology allows the integration of voice, video, traditional data, and other traffic types on a single network. ATM offers a unique opportunity to deploy the same standardized networking technology in both the wide-area and local-area environments. IBM has introduced a family of products that provides a complete ATM solution for customers. The products support ATM standards, allowing the products to interwork with devices from other vendors. This paper provides a tutorial on ATM technology and an overview of the IBM ATM product family. The IBM 8260 Intelligent Switching Hub is described as a representative ATM product.

All routes to efficient datacenter fabrics

Performance optimized datacenters (PoDs) require efficient PoD interconnects to deal with the increasing volumes of inter-server (east-west) traffic. To cope with these stringent traffic patterns, datacenter networks are abandoning the oversubscribed topologies of the past, and move towards full-bisection fat-tree fabrics. However, these fabrics typically employ either single-path or coarse-grained (flow-level) multi-path routing. In this paper, we use computer simulations and analysis to characterize the waste of bandwidth that is due to routing inefficiencies. Our analysis suggests that, under a randomly selected permutation, the expected throughputs of d-mod-k routing and of flow-level multi-path routing are close to 63% and 47%, respectively. Furthermore, nearly 30% of the flows are expected to undergo an unnecessary 3-fold slowdown. By contrast, packet-level multi-path routing consistently delivers full throughput to all flows, and proactively avoids internal hotspots, thus serving better the growing demands of inter-server (east-west) traffic.

Integration and QoS of multicast traffic in a server-rack fabric with 640 100g ports

Flexible datacenters rely on high-bandwidth server-rack fabrics to allocate their distributed computing and storage resources anywhere, anyhow, and anytime demanded. We describe the multicast architecture of a distributed server-rack fabric, which is arranged around a spine-leaf topology and connects 640 Ethernet ports running at 100G. To cope with the immense fabric speed, we resort to hierarchical, tree-based replication, facilitated by specially commissioned fabric-end ports. At each (port-to-port) leg of the tree, a frame copy is forwarded after a request-grant admission phase and is ACKed by the receiver. To save on bandwidth, we use a packet cache in our input-queued switching-nodes, which replicates asynchronously forwarded frames thus tolerating the variable-delay in the admission phase. Because the cache has limited size, we loosely synchronize the multicast subflows to protect the cache from thrashing. We describe our policies for lossy classes, which segregate and provide fair treatment to multicast subflows. Finally, we show that industry-standard Level2 congestion control does not adapt well to one-to-many flows, and demonstrate that the methods that we implement achieve the best performance.

Arbitration of many thousand flows at 100G and beyond

Network devices supporting above-100G links are needed today in order to scale communication bandwidth along with the processing capabilities of computing nodes in data centers and warehouse computers. In this paper, we propose a light-weight, fair scheduler for such ultra high-speed links, and an arbitrarily large number of requestors. We show that, in practice, our first algorithm, as well its predecessor, DRR, may result in bursty service even in the common case, where flow weights are approximately equal, and we identify applications where this can damage performance. Our second contribution is an enhancement that improves short-term fairness to deliver very smooth service when flow weights are approximately equal, whilst allocating bandwidth in a weighted fair manner.

Large switches or blocking multi-stage networks? An evaluation of routing strategies for datacenter fabrics

Cloud computing clusters require efficient interconnects to deal with the increasing volume of inter-server (east-west) traffic. To cope with these new traffic patterns, datacenter networks are abandoning the oversubscribed topologies of the past, and adopt fat-tree fabrics with high bisection bandwidth. However, these fabrics typically employ either single-path or coarse-grained (flow-level) multipath routing.In this paper, we characterize the waste of bandwidth due to routing inefficiencies. Our analysis, confirmed by computer simulations, demonstrates that under a randomly selected permutation the expected throughputs of d-mod-k routing and of Equal-Cost-Multi-Pathing (ECMP) (or flow-level multipath routing) (Thaler and Hopps, 2000) 1 are close to 63% and 47%, respectively. Furthermore, nearly 30% of the flows are expected to undergo an unnecessary 3-fold slowdown. In contrast, packet-level multipath routing consistently delivers full throughput to all flows, thus serving the growing demands of inter-server (east-west) traffic better.Using unmodified TCP stacks, we also demonstrate that under typical traffic conditions and system configurations flow-level multi-path routing can abruptly increase the completion time of latency-critical flows by more than one order of magnitude. In contrast, packet-level multipath routing, proactively avoids in-fabric backlogs, and minimizes the flow completion time across the full range of configurations that we examine. Finally, we present the design of a cost-efficient switch node performing adaptive packet-level spraying.

Tandem queue weighted fair smooth scheduling

Network devices supporting 100G links are in demand to meet the communication requirements of computing nodes in datacenters and warehouse computers. In this paper, we propose TQ and TQ-Smooth, two light-weight, fair schedulers that accommodate an arbitrarily large number of requestors and are suitable for ultra high-speed links. We show that our first algorithm, TQ, as well its predecessor, DRR, may result in bursty service even in the common case where flow weights are approximately equal, and we find that this can damage the performance of buffer-credit allocation schemes. Our second algorithm, TQ-Smooth, improves short-term fairness to deliver very smooth service when flow weights are approximately equal, while allocating bandwidth in a weighted fair manner. In many practical situations, a scheduler is asked to allocate resources in fixed-size chunks (e.g. buffer units), whose size may exceed that of (small) network packets. In such cases, byte-level fairness will typically be compromised when small-packet flows compete with large-packet ones. We describe and evaluate a scheme that dynamically adjusts the service rates of request/grant buffer reservation to achieve byte-level fairness based on received packet sizes.

Intelligent LAN Concentrators Evolution to ATM

Disclaimer The information presented hereunder is a sequence of what is expected to be generally available in the industry. The devices as shown and concepts developed are generic in nature and are not necessarily going to be mapped into IBM products Abstract The following pages consider the main drivers of the evolution of Local Area Networks (LAN) and try to project the future of current Intelligent Concentrators (I-HUBs). The trend toward ATM is rationalized and explained. Then some possible ways to implement ATM as the LAN technology of the late 90s are explored.