A unified storage layer for supporting distributed workflows in kubernetes

Abstract

The distributed computing landscape has been undergoing radical changes: High-Performance Computing (HPC) applications are moving to the Cloud, as a way of simplifying development, deployment, and migration across computing systems. Meanwhile, cloud applications are becoming increasingly complex and computationally intensive, with the advent of High-Performance Data Analytics (HPDA) pipelines --- highly distributed workflows dealing with enormous and diverse datasets, heavily relying on virtualization and containers, that would benefit from technologies used in traditional HPC. The foreseeable convergence, demands new abstractions to cope with the increased heterogeneity, so that differing workload classes can coexist seamlessly on the same infrastructure. In this paper, we propose a Unified Storage Layer (USL), to enable cloud-native applications to transparently access a wide spectrum of storage solutions, ranging from high-performance filesystems to cloud-based object stores and key-value databases. Allowing software to exploit the best out of the storage services available with no need for manual intervention from users or programmers, simplifies development and execution of workflows, and boosts overall productivity. This work has been motivated by the requirements of real-world, industry-driven applications running on Kubernetes, the industry standard for cloud infrastructure orchestration.