Christos Karamanolis

Sinfonia: a new paradigm for building scalable distributed systems

We propose a new paradigm for building scalable distributed systems. Our approach does not require dealing with message-passing protocols -- a major complication in existing distributed systems. Instead, developers just design and manipulate data structures within our service called Sinfonia. Sinfonia keeps data for applications on a set of memory nodes, each exporting a linear address space. At the core of Sinfonia is a novel minitransaction primitive that enables efficient and consistent access to data, while hiding the complexities that arise from concurrency and failures. Using Sinfonia, we implemented two very different and complex applications in a few months: a cluster file system and a group communication service. Our implementations perform well and scale to hundreds of machines.

Hybrid Cloud Storage: Bridging the Gap between Compute Clusters and Cloud Storage

Thanks to the compelling economics of public cloud storage, the trend in the IT industry is to move the bulk of analytics and application data to services such as AWS S3 and Google Cloud Storage. At the same time, customers want to continue accessing and analyzing much of that data using applications that run on compute clusters that may reside either on public clouds or on-premise. For VMware customers, those clusters run vSphere (sometimes with vSAN) on-premise and in the future may utilize SDDCaaS. Cloud storage exhibits high latencies and it is not appropriate for direct use by applications. A key challenge for these use cases is determining the subset of the typically huge data sets that need to be moved into the primary storage tier of the compute clusters. This paper introduces a novel approach for creating a hybrid cloud storage that allows customers to utilize the fast primary storage of their compute clusters as a caching tier in front of a slow secondary storage tier. This approach can be completely transparent requiring no changes to the application. To achieve this, we extended VDFS [16], a POSIX-compliant scale-out filesystem, with the concept of caching-tier volumes. VDFS caching-tier volumes resemble regular file system volumes, but they fault-in data from a cloud storage back-end on first access. Cached data are persisted on fast primary storage, close to the compute cluster, like VMwares vSAN. Caching-tier volumes use a write-back approach. The enterprise features of the primary storage ensure the persistence and fault tolerance of new or updated data. Write-back from the primary to cloud storage is managed using an efficient change-tracking mechanism built into VDFS called exo-clones [18]. This paper outlines the architecture and implementation of caching tier volumes on VDFS and reports on an initial evaluation of the current prototype.