Michael Schoettner

The Architecture of the XtreemOS Grid Checkpointing Service

The EU-funded XtreemOS project implements a grid operating system (OS) transparently exploiting distributed resources through the SAGA and POSIX interfaces. XtreemOS uses an integrated grid checkpointing service (XtreemGCP) for implementing migration and fault tolerance for grid applications. Checkpointing and restarting applications in a grid requires saving and restoring distributed/parallel applications in distributed heterogeneous environments. In this paper we present the architecture of the XtreemGCP service integrating existing system-specific checkpointer solutions. We propose to bridge the gap between grid semantics and system-specific checkpointers by introducing a common kernel checkpointer API that allows using different checkpointers in a uniform way. Our architecture is open to support different checkpointing strategies that can be adapted according to evolving failure situations or changing application requirements. Finally, we discuss measurements numbers showing that the XtreemGGP architecture introduces only minimal overhead.

Independent checkpointing in a heterogeneous grid environment

The EU-funded XtreemOS project implements an open-source grid operating system based on Linux. In order to provide fault tolerance and migration for grid applications, it integrates a distributed grid-checkpointing service called XtreemGCP. This service is designed to support various checkpointing protocols and different checkpointer packages (e.g. BLCR, LinuxSSI, OpenVZ, etc.) in a transparent manner through a uniform checkpointer interface. In this paper, we present the integration of a backward error recovery protocol based on independent checkpointing into the XtreemGCP service. The solution we propose is not checkpointer bound and thus can be transparently used on top of any checkpointer package. To evaluate the prototype we run it within a heterogeneous environment composed of single-PC nodes and a Single System Image (SSI) cluster. The experimental results demonstrate the capability of the XtreemGCP service to integrate different checkpointing protocols and independently checkpoint a distributed application within a heterogeneous grid environment. Moreover, the performance evaluation also shows that our solution outperforms the existing coordinated checkpointing protocol in terms of scalability.

An In-Memory Framework for Extended MapReduce

The MapReduce programming model simplifies the design and implementation of certain parallel algorithms. Recently, several work-groups have extended MapReduces application domain to iterative and on-line data processing. Despite having different data access characteristics, these extensions rely on the same storage facility as the original model, but propagate data updates using additional techniques. In order to benefit from large main memories, fast data access and stronger data consistency, we propose to employ in-memory storage for extended MapReduce. In this paper, we describe the design and implementation of EMR, an in-memory framework for extended MapReduce. To illustrate the usage and performance of our framework, we present measurements of typical MapReduce applications.

Checkpointing and migration of communication channels in heterogeneous grid environments

A grid checkpointing service providing migration and transparent fault tolerance is important for distributed and parallel applications executed in heterogeneous grids In this paper we address the challenges of checkpointing and migrating communication channels of grid applications executed on nodes equipped with different checkpointer packages We present a solution that is transparent for the applications and the underlying checkpointers It also allows using single node checkpointers for distributed applications The measurement numbers show only a small overhead especially with respect to large grid-applications where checkpointing may consume many minutes.

Distributed Architecture for a Peer-to-Peer-Based Virtual Microscope

Virtual microscopes are commonly used in medical education. They provide a platform for distributing whole slide images (WSI) with several GB size to exploring students. Even in courses with a few hundred students and dozens of WSI the network traffic may be high, but it will vastly increase, when the system is opened to access from the Internet. The same applies to user-generated content like interactive annotations (each student generates approx. 200 labels per term). In a collection that consists of several thousand WSI, which need to be annotated for training or quiz-based purposes, there will be millions of user contributions. In an abstract view users navigate through a universe of WSI and annotations and may meet other users watching the same or related WSI. This paper presents a distributed architecture build on PathFinder for Internet-based virtual microscopy addressing the challenges of distributing tightly connected data chunks on an overlay network consisting of random graphs.

Applications and Evaluation of In-memory MapReduce

In-memory storage techniques provide cloud applications with cheap, fast and large-scale RAM-based storage. By replicating data and providing adequate consistency control mechanisms, in-memory storage can simplify the design and implementation of highly scalable distributed applications. We argue that in-memory storage can increase the flexibility of the MapReduce parallel programming model without requiring additional communication facilities to propagate data updates. In this paper, we present several applications for our in-memory MapReduce framework from diverse problem domains including iterative and on-line data processing.

Parallelized Recovery of Hundreds of Millions Small Data Objects

Social media networks as well as online graph analytics operate on large-scale graphs with millions of vertices, even billions in some cases. Low-latency access is essential, but caching suffers from the mostly irregular access patterns of the aforementioned application domains. Hence, distributed in-memory systems are proposed keeping all data always in memory. But, the sheer amount of small data objects demands new concepts regarding the local and global data management as well as for the fault-tolerance mechanisms to mask server failures and power outages. We propose a backup distribution mechanism and a parallel recovery concept allowing to recover a failed server storing hundreds of millions of small objects within 1 to 2 seconds. All proposed concepts have been implemented within the open source system DXRAM and have been evaluated in the Microsoft Azure cloud with up to 72 virtual machines. The experiments show that DXRAM can recover a server storing 500,000,000 small objects from SSDs within 2 seconds.

A Portable and Platform Independent File System for Large Scale Peer-to-Peer Systems and Distributed Applications

Virtual microscopy is an evolving medical application used for teaching and learning at universities. We have developed a peer-to-peer based solution called Omentum, aiming at bringing virtual microscopy to an Internet-scale community. Omentum has to manage more than 10,000 large proprietary microscopic images that are converted to easily dividable JPEG-trees, each consisting of millions of very small-scaled image parts. In this paper we propose a portable and platform independent user space file system (USPFS) for addressing the application-specific access patterns, security concerns and data integrity. USPFS is able to efficiently manage huge capacities (roughly 9 × 1018 slices with 9,000 Petabytes each) with a theoretically infinite number of storable objects while providing highly important platform independency, data integrity checks as well as an easily extendable API. The evident metadata overhead is only 0.3 % and the performance evaluation shows promising results for both read and write operations.

Omentum - A Peer-to-Peer Approach for Internet-Scale Virtual Microscopy

Virtual microscopy is increasingly used for e-learning and medical online exams at universities. Traditional client-server systems support upi¾?to a few hundred of users accessing more than 10.000 large microscopic images each several Gigabyte and each being able to make interactive annotations. We have developed the first peer-to-peer based solution bringing virtual microscopy to an Internet-scale community. We address data distribution and replication by a novel overlay called Omentum, which is based on a random-graph architecture. Omentum uses a lightweight messaging service for peer communication and supports traffic-free routing-path calculation. Based on the directed random graph the system achieves path compression by walking along inbound links during the actual routing phase. The evaluation shows the efficiency and scalability of the Omentum overlay network, its replication strategy and an administrative communication overhead for creating new replicas around