Adaptive Network Coded Clouds: High Speed Downloads and Cost-Effective Version Control

Abstract

Although cloud systems provide a reliable and flexible storage solution, the use of a single cloud service constitutes a single point of failure, which can compromise data availability, download speed, and security. To address these challenges, we advocate for the use of multiple cloud storage providers simultaneously using network coding as the key enabling technology. Our goal is to study two challenges of network coded storage systems. First, the efficient update of the number of coded fragments per cloud in a system aggregating multiple clouds in order to boost the download speed of files. We developed a novel scheme using recoding with limited packets to trade-off storage space, reliability, and data retrieval speed. Implementation and measurements with commercial cloud providers show that up to 9x less network use is needed compared to other network coding schemes, while maintaining similar download speeds and reliability. Second, the ability to update coded fragments from a linear erasure code when the original file is modified. We exploit code structure to provide efficient representations of the evolution of the file. Evaluations using file changes on software library repositories show that a five-order of magnitude reduction in network and storage use is possible compared to state-of-the-art.