Lluis Pamies-Juarez

Locally repairable codes with multiple repair alternatives

Distributed storage systems need to store data redundantly in order to provide some fault-tolerance and guarantee system reliability. Different coding techniques have been proposed to provide the required redundancy more efficiently than traditional replication schemes. However, compared to replication, coding techniques are less efficient for repairing lost redundancy, as they require retrieval of larger amounts of data from larger subsets of storage nodes. To mitigate these problems, several recent works have presented locally repairable codes designed to minimize the repair traffic and the number of nodes involved per repair. Unfortunately, existing methods often lead to codes where there is only one subset of nodes able to repair a piece of lost data, limiting the local repairability to the availability of the nodes in this subset. In this paper, we present a new family of locally repairable codes that allows different trade-offs between the number of contacted nodes per repair, and the number of different subsets of nodes that enable this repair. We show that slightly increasing the number of contacted nodes per repair allows to have repair alternatives, which in turn increases the probability of being able to perform efficient repairs. Finally, we present pg-BLRC, an explicit construction of locally repairable codes with multiple repair alternatives, constructed from partial geometries, in particular from Generalized Quadrangles. We show how these codes can achieve practical lengths and high rates, while requiring a small number of nodes per repair, and providing multiple repair alternatives.

RapidRAID: Pipelined erasure codes for fast data archival in distributed storage systems

To achieve reliability in distributed storage systems, data has usually been replicated across different nodes. However the increasing volume of data to be stored has motivated the introduction of erasure codes, a storage efficient alternative to replication, particularly suited for archival in data centers, where old datasets (rarely accessed) can be erasure encoded, while replicas are maintained only for the latest data. Many recent works consider the design of new storage-centric erasure codes for improved repairability. In contrast, this paper addresses the migration from replication to encoding: traditionally erasure coding is an atomic operation in that a single node with the whole object encodes and uploads all the encoded pieces. Although large datasets can be concurrently archived by distributing individual object encodings among different nodes, the network and computing capacity of individual nodes constrain the archival process due to such atomicity. We propose a new pipelined coding strategy that distributes the network and computing load of single-object encodings among different nodes, which also speeds up multiple object archival. We further present RapidRAID codes, an explicit family of pipelined erasure codes which provides fast archival without compromising either data reliability or storage overheads. Finally, we provide a real implementation of RapidRAID codes and benchmark its performance using both a cluster of 50 nodes and a set of Amazon EC2 instances. Experiments show that RapidRAID codes reduce a single objects coding time by up to 90%, while when multiple objects are encoded concurrently, the reduction is up to 20%.

Decentralized Erasure Coding for Efficient Data Archival in Distributed Storage Systems

Distributed storage systems usually achieve fault tolerance by replicating data across different nodes. However, redundancy schemes based on erasure codes can provide a storage-efficient alternative to replication. This is particularly suited for data archival since archived data is rarely accessed. Typically, the migration to erasure-encoded storage does not leverage on the existing replication based redundancy, and simply discards (garbage collects) the excessive replicas. In this paper we propose a new decentralized erasure coding process that achieves the migration in a network-efficient manner in contrast to the traditional coding processes. The proposed approach exploits the presence of data that is already replicated across the system and distributes the redundancy generation among those nodes that store part of this replicated data, which in turn reduces the overall amount of data transferred during the encoding process. By storing additional replicated blocks at nodes executing the distributed encoding tasks, the necessary network traffic for archiving can be further reduced. We analyze the problem using symbolic computation and show that the proposed decentralized encoding process can reduce the traffic by up to 56% for typical system configurations.

CORE: Cross-object redundancy for efficient data repair in storage systems

Erasure codes are an integral part of many distributed storage systems aimed at Big Data, since they provide high fault-tolerance for low overheads. However, traditional erasure codes are inefficient on replenishing lost data (vital for long term resilience) and on reading stored data in degraded environments (when nodes might be unavailable). Consequently, novel codes optimized to cope with distributed storage system nuances are vigorously being researched. In this paper, we take an engineering alternative, exploring the use of simple and mature techniques - juxtaposing a standard erasure code with RAID-4 like parity to realize cross object redundancy (CORE), and integrate it with HDFS. We benchmark the implementation in a proprietary cluster and in EC2. Our experiments show that for an extra 20% storage overhead (compared to traditional erasure codes) CORE yields up to 58% saving in bandwidth and is up to 76% faster while recovering a single failed node. The gains are respectively 16% and 64% for double node failures.

CloudSNAP: A transparent infrastructure for decentralized web deployment using distributed interception

Abstract Over the last years we have seen the proliferation of many new popular web applications, which are commonly used on a daily basis by most of us. The challenges that have to be overcome by web application designers include how to make these applications support as much concurrent users as possible, without degrading application’s performance, and without single points of failure. Such complex task would be much easier to achieve if designers could concentrate on the application functionalities without worrying about its wide-area scope and derived problems. In this article, we introduce CloudSNAP, a decentralized web deployment platform. CloudSNAP allows transforming any actual web application into a globally-enabled and scalable one. By using a distributed Peer-to-Peer (P2P) Cloud interception middleware, all necessary functionalities are injected into existent web infrastructures in a transparent way. Therefore, CloudSNAP provides many benefits from P2P Cloud computing, like a decentralized deployment environment as well as a set of distributed mechanisms, like load balancing, fault tolerance, dynamic activation, persistence and replication. Moreover, our solution offers important advantages: (i) a high degree of transparency and decoupling in all provided services by means of distributed interception techniques, and (ii) the direct deployment of existent Java Enterprise Edition (Java EE) applications and services with practically no changes on them. In summary, CloudSNAP makes it easy to deploy any Java EE web application into a P2P Cloud infrastructure, and immediately benefit from all of its inherent services at a minimal development and infrastructure cost.

Data Insertion and Archiving in Erasure-Coding Based Large-Scale Storage Systems

Given the vast volume of data that needs to be stored reliably, many data-centers and large-scale file systems have started using erasure codes to achieve reliable storage while keeping the storage overhead low. This has invigorated the research on erasure codes tailor made to achieve different desirable storage system properties such as efficient redundancy replenishment mechanisms, resilience against data corruption, degraded reads, to name a few prominent ones. A problem that has mainly been overlooked until recently is that of how the storage system can be efficiently populated with erasure coded data to start with. In this paper, we will look at two distinct but related scenarios: (i) migration to archival - leveraging on existing replicated data to create an erasure encoded archive, and (ii) data insertion - new data being inserted in the system directly in erasure coded format. We will elaborate on coding techniques to achieve better throughput for data insertion and migration, and in doing so, explore the connection of these techniques with recently proposed locally repairable codes such as self-repairing codes.

A study of the performance of novel storage-centric repairable codes

Erasure coding has become an integral part of the storage infrastructure in data-centers and cloud backends—since it provides significantly higher fault tolerance for substantially lower storage overhead compared to a naive approach like n-way replication. Fault tolerance refers to the ability to achieve very high availability despite (temporary) failures, but for long term data durability, the redundancy provided by erasure coding needs to be replenished as storage nodes fail or are retired. Traditional erasure codes are not easily amenable to repairs, and their repair process is usually both expensive and slow. Consequently, in recent years, numerous novel codes tailor-made for distributed storage have been proposed to optimize the repair process. Broadly, most of these codes belong to either of the two following families: network coding inspired regenerating codes that aim at minimizing the per repair traffic, and locally repairable codes (LRC) which minimize the number of nodes contacted per repair (which in turn leads to the reduction of repair traffic and latency). Existing studies of these codes however restrict themselves to the repair of individual data objects in isolation. They ignore many practical issues that a real system storing multiple objects needs to take into account. Our goal is to explore a subset of such issues, particularly pertaining to the scenario where multiple objects are stored in the system. We use a simulation based approach, which models the network bottlenecks at the edges of a distributed storage system, and the nodes’ load and (un)availability. Specifically, we abstract the key features of both regenerating and LRC, and examine the effect of data placement and the corresponding de/correlation of failures, and the competition for limited network resources when multiple objects need to be repaired simultaneously by exploring the interplay of code parameters and trade-offs of bandwidth usage and speed of repairs.

Spider Codes: Practical erasure codes for distributed storage systems

Distributed storage systems use erasure codes to reliably store data with a small storage overhead. To further improve system performance, some novel erasure codes introduce new features such as the regenerating property or symbol locality, enabling these codes to have optimal repair times and optimal degraded read performance. Unfortunately, the introduction of these new features often exacerbates the performance of other system metrics such as encoding throughput, data reliability, and storage overhead, among others. In this paper we describe the intricate relationships between erasure code properties and system-level performance metrics, showing the different tradeoffs distributed storage designers need to face. We also present Spider Codes, a new erasure code achieving a practical trade-off between the different system-level performance metrics.