Sriram Ramabhadran

Cloud control with distributed rate limiting

Todays cloud-based services integrate globally distributed resources into seamless computing platforms. Provisioning and accounting for the resource usage of these Internet-scale applications presents a challenging technical problem. This paper presents the design and implementation of distributed rate limiters, which work together to enforce a global rate limit across traffic aggregates at multiple sites, enabling the coordinated policing of a cloud-based services network traffic. Our abstraction not only enforces a global limit, but also ensures that congestion-responsive transport-layer flows behave as if they traversed a single, shared limiter. We present two designs - one general purpose, and one optimized for TCP - that allow service operators to explicitly trade off between communication costs and system accuracy, efficiency, and scalability. Both designs are capable of rate limiting thousands of flows with negligible overhead (less than 3% in the tested configuration). We demonstrate that our TCP-centric design is scalable to hundreds of nodes while robust to both loss and communication delay, making it practical for deployment in nationwide service providers.

Brief announcement: prefix hash tree

This paper describes the Prefix Hash Tree, a distributed data structure that enables range queries over Distributed Hash Tables.

Stratified round Robin: a low complexity packet scheduler with bandwidth fairness and bounded delay

Fair queuing is a well-studied problem in modern computer networks. However, there remains a gap between scheduling algorithms that have provably good performance, and those that are feasible and practical to implement in high speed routers. In this paper, we propose a novel packet scheduler called Stratified Round Robin, which has low complexity, and is amenable to a simple hardware implementation. Stratified Robin Robin exhibits good fairness and delay properties that are demonstrated through both analytical results and simulations. In particular, it provides a single packet delay bound that is independent of the number of flows. This property is unique to Stratified Round Robin among all other schedulers of comparable complexity.

Analysis of Long-Running Replicated Systems

We address the problem of using replication to reliably maintain state in a distributed system for time spans that far exceed the lifetimes of individual replicas. This scenario is relevant for any system comprised of a potentially large and selectable number of replicated components, each of which may be highly unreliable, where the goal is to have enough replicas to keep the system “alive” (meaning at least one replica is working or available) for a certain expected period of time, i.e., the system’s lifetime. In particular, this applies to recent efforts to build highly available storage systems based on the peer-to-peer paradigm. We model notions of replica loss and replica repair in such systems by a simple Markov chain model, and derive an expression for the lifetime of the replicated state. We then apply this model to study the impact of practical considerations like storage and bandwidth limits on the system, and describe methods to optimally choose system parameters so as to maximize lifetime. Our analysis sheds light on the efficacy of various replication strategies.

Efficient implementation of a statistics counter architecture

Internet routers and switches need to maintain millions of (e.g., per prefix) counters at up to OC-768 speeds that are essential for traffic engineering. Unfortunately, the speed requirements require the use of large amounts of expensive SRAM memory. Shah et al [1]introduced a cheaper statistics counter architecture that uses a much smaller amount of SRAM by using the SRAM as a cache together with a (cheap) backing DRAM that stores the complete counters. Counters in SRAM are periodically updated to the DRAM before they overflow under the control of a counter management algorithm. Shah et al [1] also devised a counter management algorithm called LCF that they prove uses an optimal amount of SRAM. Unfortunately, it is difficult to implement LCF at high speeds because it requires sorting to evict the largest counter in the SRAM. This paper removes this bottleneck in [1] by proposing a counter management algorithm called LR(T) (Largest Recent with thresh-old T) that avoids sorting by only keeping a bitmap that tracks counters that are larger than threshold T. This allows LR(T) to be practically realizable using only at most 2 bits extra per counter and a simple pipelined data structure. Despite this, we show through a formal analysis, that for a particular value of the threshold T, the LR(T) requires an optimal amount of SRAM, matching LCF. Further,we also describe an implementation, based on a novel data structure called aggregated bitmap, that allows the LR(T) algorithm to be realized at line rates.

The Stratified Round Robin scheduler: design, analysis and implementation

Stratified Round Robin is a fair-queueing packet scheduler which has good fairness and delay properties, and low quasi-O(1) complexity. It is unique among all other schedulers of comparable complexity in that it provides a single packet delay bound that is independent of the number of flows. Importantly, it is also amenable to a simple hardware implementation, and thus fills a current gap between scheduling algorithms that have provably good performance and those that are feasible and practical to implement in high-speed routers. We present both analytical results and simulations to demonstrate its performance properties

A study of end-to-end web access failures

We present a study of end-to-end web access failures in the Internet. Part of our characterization of failures is based on directly observable end-to-end information. We also present novel analyses that reveal aspects of end-to-end failures that would be hard to discern otherwise. First, we combine end-to-end failure observations across a large number of clients to classify failures as server-related or client-related. Second, we correlate failures attributed to a client or server with BGP churn for the corresponding IP address prefix(es), to shed light on the end-to-end impact of BGP instability. Our study is based on failure observations during a month-long experiment involving 134 client hosts (across Planet-Lab, commercial dialup and broadband ISPs, and a corporate network) repeatedly accessing 80 websites. We find that the median failure rate of web accesses is about 1.5%, which is non-negligible. About 34--42% of the web access failures are due to DNS problems, primarily due to the inability of the client to connect to its local DNS server. The majority of the remaining failures are due to TCP connection establishment failures. Also, by correlating failure observations across clients and servers, we find that server-side problems are the dominant cause of TCP connection failures.

Cooperative packet scheduling via pipelining in 802.11 wireless networks

The proliferation of 802.11a/b/g based wireless devices has fueled their adoption in many domains -- some of which are unforseen. Yet, these devices lack native support for some of the advanced features (such as service differentiation, etc.) required in specific application domains. A subset of these features relies on cooperative scheduling whereby nodes cooperate among each other to effectively manage resources such as power, throughput and interference in wireless networks. The trajectory of evolution in these devices has been primarily through new extension standards (such as 802.11e/s etc.) that offer support for these features. Plagued with long design cycles and cost overhead to upgrade, this process of upgrading creates an uphill task to users who want to use their wireless devices for different applications. In this paper, we argue that such cooperative scheduling extensions can be supported using a new layer on top of the existing MAC layer. We propose a 2½- pipeline architecture as a generic mechanism to create such domain specific extensions and propose two such protocols, SPARTA (power conservation) and ARGOS (throughput guarantees) over the native 802.11/b/g MAC layer.

Durability of replicated distributed storage systems

We study the problem of guaranteeing data durability [2] in distributed storage systems based on replication. Our work is motivated by several several recent efforts [3, 5, 1] to build such systems in a peer-to-peer environment. The key features of this environment which make achieving durability difficult are (1) data lifetimes may be several orders of magnitude larger than the lifetimes of individual storage units, and (2) the system may have little or no control over the participation of these storage units in the system. We use a model-based approach to develop engineering principles for designing automated replication and repair mechanisms to implement durability in such systems.

Analysis of durability in replicated distributed storage systems

In this paper, we investigate the roles of replication vs. repair to achieve durability in large-scale distributed storage systems. Specifically, we address the fundamental questions: How does the lifetime of an object depend on the degree of replication and rate of repair, and how is lifetime maximized when there is a constraint on resources? In addition, in real systems, when a node becomes unavailable, there is uncertainty whether this is temporary or permanent; we analyze the use of timeouts as a mechanism to make this determination. Finally, we explore the importance of memory in repair mechanisms, and show that under certain cost conditions, memoryless systems, which are inherently less complex, perform just as well.