Scott Dawson

Testing of fault-tolerant and real-time distributed systems via protocol fault injection

As software for distributed systems becomes more complex, ensuring that a system meets its prescribed specification is a growing challenge that confronts software developers. This is particularly important for distributed applications with strict dependability and timeliness constraints. This paper reports on ORCHESTRA, a portable fault injection environment for testing implementations of distributed protocols. This tool is based on a simple yet powerful framework called script-driven probing and fault injection, for the evaluation and validation of the fault-tolerance and timing characteristics of distributed protocols. The tool, which was initially developed on the Real-Time Mach operating system and later ported to other platforms including Solaris and SunOS, has been used to conduct extensive experiments on several protocol implementations. This paper describes the design and implementation of the fault injection tool focusing on architectural features to support portability, minimizing intrusiveness on target protocols, and explicit support for testing real-time systems. The paper also describes the experimental evaluation of two protocol implementations: a real-time audio-conferencing application on Real-Time Mach, and a distributed group membership service on the Sun Solaris operating system.

Experiments on six commercial TCP implementations using a software fault injection tool

TCP, the de facto standard transport protocol in todays operating systems, is a very robust protocol that adapts to various network characteristics, packet loss, link congestion, and even significant differences in vendor implementations. This paper describes a set of experiments performed on six different vendor TCP implementations using ORCHESTRA, a tool for testing and fault injection of communication protocols. These experiments uncovered violations of the TCP protocol specification, and illustrated differences in the philosophies of various vendors in their implementations of TCP. The paper summarizes several lessons learned about the TCP implementations through these experiments.

ARMADA Middleware and Communication Services

Real-time embedded systems have evolved during the past several decades from small custom-designed digital hardware to large distributed processing systems. As these systems become more complex, their interoperability, evolvability and cost-effectiveness requirements motivate the use of commercial-off-the-shelf components. This raises the challenge of constructing dependable and predictable real-time services for application developers on top of the inexpensive hardware and software components which has minimal support for timeliness and dependability guarantees. We are addressing this challenge in the ARMADA project.ARMADA is set of communication and middleware services that provide support for fault-tolerance and end-to-end guarantees for embedded real-time distributed applications. Since real-time performance of such applications depends heavily on the communication subsystem, the first thrust of the project is to develop a predictable communication service and architecture to ensure QoS-sensitive message delivery. Fault-tolerance is of paramount importance to embedded safety-critical systems. In its second thrust, ARMADA aims to offload the complexity of developing fault-tolerant applications from the application programmer by focusing on a collection of modular, composable middleware for fault-tolerant group communication and replication under timing constraints. Finally, we develop tools for testing and validating the behavior of our services. We give an overview of the ARMADA project, describing the architecture and presenting its implementation status.

Probing and Fault Injection of Dependable Distributed Protocols

Ensuring that a distributed system with strict dependability constraints meets its prescribed specification is a growing challenge that confronts software developers and system engineers. This paper presents a technique for probing and fault injection of fault-tolerant distributed protocols. The proposed technique, called script-driven probing and fault injection, can be used for studying the behaviour of distributed systems and for detecting design and implementation errors of fault-tolerant protocols. The focus of this work is on fault injection techniques that can be used to demonstrate three aspects of a target protocol : detection of design or implementation errors ; identification of violations of protocol specifications ; and insight into design decisions made by the implementers. The emphasis of our approach is on experimental techniques intended to identify specific problems in a protocol or its implementation rather than the evaluation of system dependability through statistical metrics such as fault coverage. To demonstrate the capabilities of this technique, the paper describes a probing and fault injection tool, called the PFI tool (probe/fault injection tool), and a summary of several extensive experiments that studied the behaviour of two protocols : the Transmission Control Protocol (TCP) and a group membership protocol (GMP).

A software fault injection tool on real-time Mach

Ensuring that a distributed real-time system with strict dependability constraints meets its prescribed specification is a growing challenge that confronts software developers and system engineers. This paper reports on a software fault injection tool, called SockPFI, for testing the fault tolerance and timing behavior of distributed real-time applications. SockPFI, developed on Real-Time Mach, can be used to test socket-based distributed real-time applications on this platform without modifying the source code of the target protocol. The SockPFI tool is based on the concept of script-driven probing and fault injection. It is explicitly designed to address some of the intrusiveness associated with fault injection of distributed systems, and in particular, with real-time protocols. The paper describes the design and implementation of SockPFI on Real Time Mach, and a demonstration of the tool on a real-time primary-backup replication protocol.

Fault injection experiments on real-time protocols using ORCHESTRA

The paper presents a set of experiments on real time protocols using the ORCHESTRA software fault injection environment. The paper first describes a specific implementation of ORCHESTRA for testing applications that use Unix sockets for communication. A novel feature of this tool is that it utilizes operating system support provided by Real Time Mach to quantify and compensate for the intrusiveness of the fault injection mechanism. The fault injection experiments which are described, tested fault tolerance and timing behavior of two target real time protocols. In addition, timing intrusiveness of ORCHESTRA was measured during the course of these experiments.