Joseph Moore

CoARC: Co-operative, Aggressive Recovery and Caching for Failures in Erasure Coded Hadoop

Cloud file systems like Hadoop have become a norm for handling big data because of the easy scaling and distributed storage layout. However, these systems are susceptible to failures and data needs to be recovered when a failure is detected. During temporary failures, MapReduce jobs or file system clients perform degraded reads and satisfy the read request. We argue that lack of sharing of the recovered data during degraded reads and recovery of only the requested data block places a heavy strain on the systems network resources and increases the job execution time. To this end, we propose CoARC (Co-operative, Aggressive Recovery and Caching), which is a new data-recovery mechanism for unavailable data during degraded reads in distributed file systems. The main idea is to recover not only the data block that was requested but also other temporarily unavailable blocks in the same strip and cache them in a separate data node. We also propose an LRF (Least Recently Failed) cache replacement algorithm for such a kind of recovery caches. We also show that CoARC significantly reduces the network usage and job runtime in erasure coded Hadoop.

P3: Priority based proactive prediction for soon-to-fail disks

Predicting soon-to-fail (STF) disks is fundamental to keeping disk data safe and enforcing quality of service. Most current proactive prediction approaches achieve high prediction rate at the cost of high false alarm rate, labeling healthy disks as STF, because of the imbalanced fraction of failed disks in the training dataset and the characteristics of the machine learning (ML) techniques used. Given the known fact that healthy disks far outnumber STF disks, high false alarm rate means that more healthy disks than the actual STF disks maybe labeled as STF and results in undue waste of resources such as network bandwidth and new disks. The cumulative number of false alarms can be even larger considering that the prediction is taken periodically. This paper presents a priority based proactive prediction algorithm for STF disks (or P3), which leverages a combination of ML models. The predictor takes the attributes of the self monitoring facility (SMART) of all disks as input and outputs predicted STF disks. Compared to existing approaches, P3 can achieve lower false alarm rate, which is important to efficiently schedule resources for disk data and service migration. The tradeoff is slight decrease in prediction rate, which is negligible because the prediction is made periodically and reactive prediction can be employed as backup. In an evaluation on a population of 7,018 disks with Weka, the predictor can predict 112 out of the 130 failed disks with 36 false alarms. Compared with the state-of-art ML models that predict 122 failed disks with 34 false alarms, P3 is able to predict 113 failed disks with 7 false alarms.