Caitlin Bestler

Scalable Object Storage with Resource Reservations and Dynamic Load Balancing

Scale-out distributed storage clusters require efficient selection of the storage targets holding replicas. This requires coordinated allocation of target IOPS, target persistent storage capacity and network bandwidth. The current generation of distributed storage solutions mostly select storage targets using consistent hashing. Implementations have become significantly more sophisticated than the algorithm originally presented for Amazon Dynamo. They (consistently) yield consistent locations for a given combination of content and clustered topology. But none of them consider dynamic factors, such as current work load or remaining capacity. Moreover, non-dynamic target selection leads to uneven distribution of network traffic. Uneven resource distribution underutilizes available capacity. We present an alternative solution, the Replicast protocol, which combines dynamic load balanced scheduling, multicast messaging and reservation-based payload delivery. Replicast has been implemented and is being deployed in a commercial solution called NexentaEdge. To evaluate Replicast for large and super-large clusters, we use a discrete event simulation framework called SURGE. In this paper, we compare the results of simulated object put benchmarks with models for conventional storage clusters that use consistent hashing and reliable unicast connections to transfer content.