Peter M. Musial

Long-lived Rambo: Trading knowledge for communication

Shareable data services providing consistency guarantees, such as atomicity (linearizability), make building distributed systems easier. However, combining linearizability with efficiency in practical algorithms is difficult. A reconfigurable linearizable data service, called Rambo, was developed by Lynch and Shvartsman. This service guarantees consistency under dynamic conditions involving asynchrony, message loss, node crashes, and new node arrivals. The specification of the original algorithm is given at an abstract level aimed at concise presentation and formal reasoning about correctness. The algorithm propagates information by means of gossip messages. If the service is in use for a long time, the size and the number of gossip messages may grow without bound. This paper presents a consistent data service for long-lived objects that improves on Rambo in two ways: it includes an incremental communication protocol and a leave service. The new protocol takes advantage of the local knowledge, and carefully manages the size of messages by removing redundant information, while the leave service allows the nodes to leave the system gracefully. The new algorithm is formally proved correct by forward simulation using levels of abstraction. An experimental implementation of the system was developed for networks-of-workstations. The paper also includes selected analytical and preliminary empirical results that illustrate the advantages of the new algorithm.

On the Efficiency of Atomic Multi-reader, Multi-writer Distributed Memory

This paper considers quorum-replicated, multi-writer, multi-reader (MWMR) implementations of survivable atomic registers in a distributed message-passing system with processors prone to failures. Previous implementations in such settings invariably required two rounds of communication between readers/writers and replica owners. Hence the question arises whether it is possible to have single round read and/or write operations in this setting. We thus devise an algorithm, called Sfw , that exploits a new technique called server side ordering ( SSO ), which ---unlike previous approaches--- places partial responsibility for the ordering of write operations on the replica owners (the servers). With SSO, fast write operations are introduced for the very first time in the MWMR setting. We prove that our algorithm preserves atomicity in all permissible executions. While algorithm SFW shows that in principle fast writes are possible, we also show that under certain conditions the MWMR model imposes inherent limitations on any quorum-based fast write implementation of a safe read/write register and potentially even restricts the number of writer participants in the system. In this case our algorithm achieves near optimal efficiency.

Implementing a reconfigurable atomic memory service for dynamic networks

Summary form only given. Transforming abstract algorithm specifications into executable code is an error-prone process in the absence of sophisticated compilers that can automatically translate such specifications into the target distributed system. We present a framework that was developed for translating algorithms specified as Input/Output Automata (IOA) to distributed programs. The framework consists of a methodology that guides the software development process and a core set of functions needed in target implementations that reduce unnecessary software development. The systems developed using this methodology preserve the modularity of the original specifications, making it easier to track refinements and effect optimizations. As a proof of concept, this work also presents a distributed implementation of a reconfigurable atomic memory service for dynamic networks (RAMBO). This service emulates atomic read/write shared objects in the dynamic setting where processors can arbitrarily crash, or join and leave the computation. The algorithm tolerates processor crashes and message loss and guarantees atomicity for arbitrary patterns of asynchrony and failure. The algorithm implementing the service is given in terms of IOA. An important consideration in formulating RAMBO was that it could be employed as a building block in real systems. Following a formal presentation of RAMBO algorithm, this work describes an optimized implementation that was developed using the methodology presented here. The system is implemented in Java and runs on a network of workstations. Empirical data illustrates the behavior of the system.

Explicit Combinatorial Structures for Cooperative Distributed Algorithms

Cooperation in distributed settings often involves activities that must be performed at least once by the participating processors. When processor failures or delays occur, it becomes unavoidable that some tasks are done redundantly. To make efficient use of the available processors, several distributed algorithms schedule the activities of the processors in terms of permutations of tasks that need to be performed at least once. This paper presents the first explicit practical deterministic construction of sets of permutations with certain combinatorial properties that immediately make practical several deterministic distributed algorithms. These algorithms solve a variety of problems, for example, cooperation in shared-memory and message-passing settings, and the gossip problem. Prior to this work, the most efficient algorithms for some of these problems were primarily of theoretical interest - they relied on permutations that are known to exist, but very expensive to construct, with the cost of construction being at least exponential in the size of the permutations. In this paper, the explicitly constructed permutations are ultimately used directly to produce practical instances of several classes of efficient deterministic algorithms. Most importantly, for all of these algorithms, the schedule construction cost is reduced from exponential to polynomial, at the expense of slight detuning, at most polylogarithmic, of the efficiency of these algorithms

Developing a Consistent Domain-Oriented Distributed Object Service

This paper presents a new algorithm for a reconfigurable distributed domain-oriented atomic object service, called DO-RAMBO, which stands for Domain-Oriented Reconfigurable Atomic Memory for Basic Objects. This service is suitable for inclusion as a middleware system service for distributed applications requiring atomic read/write data. The implementation substantially extends and refines the abstract RAMBO algorithm of Lynch and Shvartsman that supports individual atomic objects. In this paper, domains are introduced to allow the users to group related atomic objects. The new implementation manages configurations on the basis of domains, significantly improving the utility and the performance of the resulting service. DO-RAMBO guarantees consistency under asynchrony, message loss, node crashes, new node arrivals, and node departures. We present the formal algorithm development for DO-RAMBO and give analytical and empirical results that illustrate the benefit of the new approach.

Long-Lived Rambo: Trading Knowledge for Communication

Shareable data services providing consistency guarantees, such as atomicity (linearizability), make building distributed systems easier. However, combining linearizability with efficiency in practical algorithms is difficult. A reconfigurable linearizable data service, called Rambo, was developed by Lynch and Shvartsman. This service guarantees consistency under dynamic conditions involving asynchrony, message loss, node crashes, and new node arrivals. The specification of the original algorithm is given at an abstract level aimed at concise presentation and formal reasoning about correctness. The algorithm propagates information by means of gossip messages. If the service is in use for a long time, the size and the number of gossip messages may grow without bound. This paper presents a consistent data service for long-lived objects that improves on Rambo in two ways: it includes an incremental communication protocol and a leave service. The new protocol takes advantage of the local knowledge, and carefully manages the size of messages by removing redundant information, while the leave service allows the nodes to leave the system gracefully. The new algorithm is formally proved correct by forward simulation using levels of abstraction. An experimental implementation of the system was developed for networks-of-workstations. The paper also includes analytical and preliminary empirical results that illustrate the advantages of the new algorithm.

An Abstract Channel Specification and an Algorithm Implementing It Using Java Sockets

Abstract models and specifications can be used in the design of distributed applications to formally reason about their safety properties. However, the benefits of using formal methods are often negated by the ad hoc process of mapping the semantics of an abstract specification to algorithms designed to be executed on target distributed platforms. The challenge of formally specifying communication channels and correctly implementing them as algorithms that use realistic distributed system services is the focus of this paper. This work provides an original formal specification of an abstract asynchronous communication channel with support for dynamic creation and tear down of links between participating network nodes, and its implementation as an algorithm using Java sockets. The specification and the algorithm are expressed using the Input/Output Automata formalism, and it is proved that the algorithm correctly implements the specification, viz. that any externally observable behavior (trace) of the algorithm has a corresponding behavior of the specification. The approach presented here can be used to implement algorithms for dynamic systems, where communicating nodes may join, leave, and experience delays. The result is also of direct benefit to automated code generation, such as that implemented within the Input/Output Automata Toolkit at MIT.

Implementing Atomic Data through Indirect Learning in Dynamic Networks

Developing middleware services for dynamic distributed systems, e.g., ad-hoc networks, is a challenging task given that such services deal with dynamically changing membership and asynchronous communication. Algorithms developed for static settings are often not usable in such settings because they rely on (logical) all-to-all node connectivity through routing protocols, which may be unfeasible or prohibitively expensive to implement in highly dynamic settings. This paper explores the indirect learning, via periodic gossip, approach to information dissemination within a dynamic, distributed data service implementing atomic read/write memory service. The indirect learning scheme is used to improve the liveness of the service in the settings with uncertain connectivity. The service is formally proved to guarantee atomicity in all executions. Conditional performance analysis of the new service is presented, where this analysis has the potential of being generalized to other similar dynamic algorithms. Under the assumption that the network is connected, and assuming reasonable timing conditions, the bounds on the duration of read/write operations of the new service are calculated. Finally, the paper proposes a deployment strategy where indirect learning leads to an improvement in communication costs relative to a previous solution that assumes all-to-all connectivity.

On the automated implementation of time-based paxos using the IOA compiler

Paxos is a well known algorithm for achieving consensus in distributed environments with uncertain processing and communication timing. Implementations of its variants have been successfully used in the industry (eg., Chubby by Google, Autopilot cluster management in Bing by Microsoft, and many others). This paper addresses the challenge of the manual coding of complex distributed algorithms, such as Paxos, where this is an error prone process. Our approach in ensuring correct implementation is to use a verified automated translator to compile a source specification that has been proven to be itself correct. We use specification of the Paxos algorithm in the General Timed Automata (GTA) model, an extension of I/O Automata, as input to an augmented compiler for the Input/Output Automata notation (a.k.a., the IOA compiler) in order to generate executable Java code. The resulting code is interfaced with MPI for communication needs. We have extended the IOA compiler to support a version of the GTA model, which uses time-passage actions such as ν(t), to model the passage of time by t time units. A time-based version of Paxos is used to demonstrate the capabilities of our extension. In this paper we describe the process to be followed in order to compile time-based Paxos, or similar algorithms. The utility of our approach is supported by an experimental evaluation of our Paxos implementation on a collection of workstations. To the best of our knowledge, our case study constitutes the first example of a time-dependent distributed algorithm that has been specified, verified and implemented in an automated way, using a common formal methodology.

Evaluating a Dependable Sharable Atomic Data Service on a Planetary-Scale Network

Practical implementations of atomically consistent read/write memory service are important building blocks for higher level applications. This is especially true when data accessibility and survivability are provided by a distributed platform consisting of networked nodes, where both nodes and connections are subject to failure. This work presents an experimental evaluation of the practicality of an atomic memory service implementation, called RA M B O , which is the first to support multiple reader, multiple writer access to the atomic data with an integrated reconfiguration protocol to replace the underlying set of replicas without any interruption of the ongoing operations. Theoretical guarantees of this service are well understood; however, only rudimentary analytical performance along with limited LAN testing were performed on the implementation of RA M B O --- neither representing any realistic deployment setting. In order to assess true practicality of the RA M B O service, we devised a series of experiments tested on PlanetLab --- a planetary-scale research WAN network. Our experiments show that RA M B O s performance is reasonable (under the tested scenarios) and under the somewhat extreme conditions of PlanetLab. This demonstrates the feasibility of developing dependable reconfigurable sharable data services with provable consistency guarantees on unreliable distributed systems.