Abdul Kabbani

Data center transport mechanisms: Congestion control theory and IEEE standardization

Data Center Networks present a novel, unique and rich environment for algorithm development and deployment. Projects are underway in the IEEE 802.1 standards body, especially in the Data Center Bridging Task Group, to define new switched Ethernet functions for data center use. One such project is IEEE 802.1Qau, the Congestion Notification project, whose aim is to develop an Ethernet congestion control algorithm for hardware implementation. A major contribution of this paper is the description and analysis of the congestion control algorithm - QCN, for Quantized Congestion Notification- which has been developed for this purpose. A second contribution of the paper is an articulation of the Averaging Principle: a simple method for making congestion control loops stable in the face of increasing lags. This contrasts with two well-known methods of stabilizing control loops as lags increase; namely, (i) increasing the order of the system by sensing and feeding back higher-order derivatives of the state, and (ii) determining the lag and then choosing appropriate loop gains. Both methods have been applied in the congestion control literature to obtain stable algorithms for high bandwidth-delay product paths in the Internet. However, these methods are either undesirable or infeasible in the Ethernet context. The Averaging Principle provides a simple alternative, one which we are able to theoretically characterize.

AF-QCN: Approximate Fairness with Quantized Congestion Notification for Multi-tenanted Data Centers

Data Center Networks represent the convergence of computing and networking, of data and storage networks, and of packet transport mechanisms in Layers 2 and 3. Congestion control algorithms are a key component of data transport in this type of network. Recently, a Layer 2 congestion management algorithm, called QCN (Quantized Congestion Notification), has been adopted for the IEEE 802.1 Data Center Bridging standard: IEEE 802.1Qau. The QCN algorithm has been designed to be stable, responsive, and simple to implement. However, it does not provide weighted fairness, where the weights can be set by the operator on a per-flow or per-class basis. Such a feature can be very useful in multi-tenanted Cloud Computing and Data Center environments. This paper addresses this issue. Specifically, we develop an algorithm, called AF-QCN (for Approximately Fair QCN), which ensures a faster convergence to fairness than QCN, maintains this fairness at fine-grained time scales, and provides programmable weighted fair bandwidth shares to flows/flow-classes. It combines the QCN algorithm developed by some of the authors of this paper, and the AFD algorithm previously developed by Pan et. al. AF-QCN requires no modifications to a QCN source (Reaction Point) and introduces a very light-weight addition to a QCNcapable switch (Congestion Point). The results obtained through simulations and an FPGA implementation on a 1Gbps platform show that AF-QCN retains the good congestion management performance of QCN while achieving rapid and programmable (approximate) weighted fairness.

WCMP: weighted cost multipathing for improved fairness in data centers

Data Center topologies employ multiple paths among servers to deliver scalable, cost-effective network capacity. The simplest and the most widely deployed approach for load balancing among these paths, Equal Cost Multipath (ECMP), hashes flows among the shortest paths toward a destination. ECMP leverages uniform hashing of balanced flow sizes to achieve fairness and good load balancing in data centers. However, we show that ECMP further assumes a balanced, regular, and fault-free topology, which are invalid assumptions in practice that can lead to substantial performance degradation and, worse, variation in flow bandwidths even for same size flows. We present a set of simple algorithms that achieve Weighted Cost Multipath (WCMP) to balance traffic in the data center based on the changing network topology. The state required for WCMP is already disseminated as part of standard routing protocols and it can be readily implemented in the current switch silicon without any hardware modifications. We show how to deploy WCMP in a production OpenFlow network environment and present experimental and simulation results to show that variation in flow bandwidths can be reduced by as much as 25X by employing WCMP relative to ECMP.

FlowBender: Flow-level Adaptive Routing for Improved Latency and Throughput in Datacenter Networks

Datacenter networks provide high path diversity for traffic between machines. Load balancing traffic across these paths is important for both, latency- and throughput-sensitive applications. The standard load balancing techniques used today obliviously hash a flow to a random path. When long flows collide on the same path, this might lead to long lasting congestion while other paths could be underutilized, degrading performance of other flows as well. Recent proposals to address this shortcoming incur significant implementation complexity at the host that would actually slow down short flows (MPTCP), depend on relatively slow centralized controllers for rerouting large congesting flows (Hedera), or require custom switch hardware, hindering near-term deployment (DeTail). We propose FlowBender, a novel technique that: (1) Load balances distributively at the granularity of flows instead of packets, avoiding excessive packet reordering. (2) Uses end-host-driven rehashing to trigger dynamic flow-to-path assignment. (3) Recovers from link failures within a Retransmit Timeout (RTO). (4) Amounts to less than 50 lines of critical kernel code and is readily deployable in commodity data centers today. (5) Is very robust and simple to tune. We evaluate FlowBender using both simulations and a real testbed implementation, and show that it improves average and tail latencies significantly compared to state of the art techniques without incurring the significant overhead and complexity of other load balancing schemes.

Stability analysis of QCN: the averaging principle

Data Center Networks have recently caused much excitement in the industry and in the research community. They represent the convergence of networking, storage, computing and virtualization. This paper is concerned with the Quantized Congestion Notification (QCN) algorithm, developed for Layer 2 congestion management. QCN has recently been standardized as the IEEE 802.1Qau Ethernet Congestion Notification standard. We provide a stability analysis of QCN, especially in terms of its ability to utilize high capacity links in the shallow-buffered data center network environment. After a brief description of the QCN algorithm, we develop a delay-differential equation model for mathematically characterizing it. We analyze the model using a linearized approximation, obtaining stability margins as a function of algorithm parameters and network operating conditions. A second contribution of the paper is the articulation and analysis of the Averaging Principle (AP)---a new method for stabilizing control loops when lags increase. The AP is distinct from other well-known methods of feedback stabilization such as higher-order state feedback and lag-dependent gain adjustment. It turns out that the QCN and the BIC-TCP (and CUBIC) algorithms use the AP; we show that this enables them to be stable under large lags. The AP is also of independent interest since it applies to general control systems, not just congestion control systems.

Distributed Low-Complexity Maximum-Throughput Scheduling for Wireless Backhaul Networks

We introduce a low-complexity distributed slotted MAC protocol that can support all feasible arrival rates in a wireless backhaul network (WBN). For arbitrary wireless networks, such a maximum throughput protocol has been notoriously hard to realize because even if global topology information is available, the problem of computing the optimal link transmission set at each slot is NP-complete. For the logical tree structures induced by WBN traffic matrices, we first introduce a centralized algorithm that solves the optimal scheduling problem in a number of steps at most linear in the number of nodes in the network. This is achieved by discovering and exploiting a novel set of graph-theoretical properties of WBN contention graph. Guided by the centralized algorithm, we design a distributed protocol where, at the beginning of each slot, nodes coordinate and incrementally compute the optimal link transmission set. We then introduce an algorithm to compute the minimum number of steps to complete this computation, thus minimizing the per-slot overhead. Using both analysis and simulations, we show that in practice our protocol yields low overhead when implemented over existing wireless technologies and significantly outperforms existing suboptimal distributed slotted scheduling mechanisms.

J uggler : a practical reordering resilient network stack for datacenters

We present Juggler, a practical reordering resilient network stack for datacenters that enables any packet to be sent on any path at any level of priority. Juggler adds functionality to the Generic Receive Offload layer at the entry of the network stack to put packets in order in a best-effort fashion. Jugglers design exploits the small packet delays in datacenter networks and the inherent burstiness of traffic to eliminate the negative effects of packet reordering almost entirely while keeping state for only a small number of flows at any given time. Extensive testbed experiments at 10Gb/s and 40Gb/s speeds show that Juggler is effective and lightweight: it prevents performance loss even with severe packet reordering while imposing low CPU overhead. We demonstrate the use of Juggler for per-packet multi-path load balancing and a novel system that provides bandwidth guarantees by dynamically prioritizing packets.

Exploiting physical layer detection techniques to mitigate starvation in CSMA/CA wireless networks

Many proposed and implemented wireless MAC protocols are based on collision-avoidance handshakes between senders and receivers. Unfortunately, information asymmetry problems can potentially occur with such protocols and severely reduce the network capacity. In this paper, we introduce a receiver-initiated mechanism, called carrier sense multiple access with collision avoidance by receiver detection (CSMA/CARD), that makes use of collisions sensed at the physical layer of a receiver to mitigate the effect of such problems. More specifically, and depending on the exact nature of the handshake mechanism, collisions can be used by the receiver to predict whether some sender attempted to initiate a transmission towards this receiver. The receiver, in its turn, would initiate an action on its own to help expedite the handshake mechanism. Such a simple cooperation mechanism can be coupled with any appropriate MAC protocol to improve its performance. We evaluate the effectiveness of this approach with the 802.11 DCF MAC in particular, and we show that starvation and unfairness results can be highly mitigated

Flier: Flow-level congestion-aware routing for direct-connect data centers

Various topologies have been proposed in the context of high-performance computing and data center networking. Direct-connect topologies generally offer large capacity with high path diversity and are highly cost effective for general data center traffic patterns. However, the lack of simple yet efficient load balancing techniques for direct-connect fabrics has hindered these networks from gaining traction in data centers. This paper presents the design, implementation, and evaluation of Flicr, a light-weight host-based load balancing mechanism for direct-connect data centers. Flicr dynamically reroutes traffic through minimal and non-minimal routes to avoid congesting the fabric. This enables Flicr to efficiently minimize networking resource consumption while exploiting high path diversity in direct-connect fabrics to balance the network and gracefully handle link failures. Flicr requires only a simple kernel modification and is readily deployable in commodity data centers today. Our evaluations show that Flicr consistently outperforms other state-of-the-art load balancing designs, achieving 25–60% lower average flow completion time compared to adaptive routing. Flicr is also more robust against link failures and has 5–8 χ better performance relative to other schemes in the presence of link failures.