Henrik Thane

Using deterministic replay for debugging of distributed real-time systems

Cyclic debugging is one of the most important and most commonly used activities in program development. During cyclic debugging, a program is repeatedly re-executed to track down errors when a failure has been observed. This process necessitates reproducible program executions. Applying classical debugging techniques, such as using breakpoints or single stepping, in real-time systems changes the temporal behaviour and makes reproduction of the observed failure during debugging less likely, if not impossible. Consequently, these techniques are not directly applicable to the cyclic debugging of real-time systems. In this paper, we present a novel software-based approach for the cyclic debugging of distributed real-time systems. By the online recording of significant system events, and then deterministically replaying them off-line, we can inspect a real-time system in great detail, while still preserving its real-time behaviour.

Replay debugging of real-time systems using time machines

In this paper we present a new approach to deterministic replay using standard components. Our method facilitates cyclic debugging of real-time systems with industry standard real-time operating systems using industry standard debuggers. The method is based on a number of new techniques: A new marker for deterministic differentiation between e.g., loop iterations for deterministic reproduction of interrupts and task preemptions, an algorithm for finding well-defined starting points of replay sessions, as well as a technique for using conditional breakpoints in standard debuggers to replay the target system. We also propose and discuss different methods for deterministic monitoring, and provide benchmarking results from an industrial strength case study demonstrating the feasibility of our method. Previously published solutions to the problem of debugging real-time systems have been based on the concept of deterministic replay: where significant system events like task-switches of multitasking software and external inputs are recorded during run-time, and later replayed (re-executed) off-line. Previous works have been based on either non-standard hardware, specially designed compilers or modified real-time operating systems. The reliance on non-standard components has limited the success of the approach. Even though this idea has been around for 20 years, no industrial application for debugging of real-time systems of the method has been presented.

Towards systematic testing of distributed real-time systems

Reproducible and deterministic testing of sequential programs can in most cases be achieved by controlling the sequence of inputs to the program. The behavior of a distributed real-time system, on the other hand not only depends on the inputs but also on the order and timing of the concurrent tasks that execute and communicate with each other and the environment. Hence, sequential test techniques are not directly applicable, since they disregard the significance of order and timing of the tasks. In this paper we present a method for identifying all possible orderings of task starts, preemptions and completions for tasks executing in a distributed real-time system. Together with an accompanying testing strategy, this method allows test methods for sequential programs to be applied, since each identified ordering can be regarded as a sequential program. In the presented analysis and testing strategy, we consider task sets with recurring release patterns, and take into account the effects of clock synchronization and variations in start and execution times of the involved tasks.

Using measurements to derive the worst-case execution time

Execution time analysis is used in the development of real time and embedded systems to derive the timing estimates required for schedulability analysis. The execution time of the analyzed program is typically obtained by combining results from program flow analysis (such as number of iterations in loops) with low level timing information. The paper proposes a method for low level timing analysis based on measurements of execution times of programs executing on the actual target architecture. The essence of the method is to derive a system of linear equations from a limited number of timing measurements of an instrumented version of the considered program. The solution to these equations give execution times for program fragments, from which execution time measures for the entire program can be derived. The main advantage with this approach is that architectural modeling is not needed, hence the risk of a discrepancy between model and real system is avoided. Also, compared to the non-exhaustive measurements performed in industry today, our approach is more structured and gives complete coverage in terms of the program paths considered. We present our method in the context of a simple, but reasonably realistic processor model and show how it can be extended to architectures with pipelines.

Testing distributed real-time systems

Abstract For testing of sequential software it is usually sufficient to provide the same input (and program state) in order to reproduce the output. For real-time systems (RTS), on the other hand, we need also to control, or observe, the timing and order of the inputs. If the system additionally is multitasking, we also need to take timing and the concurrency of the executing tasks into account. In this paper we present a method for deterministic testing of multitasking RTS, which allows explorative investigations of real-time system behavior. The method includes an analysis technique that given a set of tasks and a schedule derives all execution orderings that can occur during run-time. These orderings correspond to the possible inter-leavings of the executing tasks. The method also includes a testing strategy that using the derived execution orderings can achieve deterministic, and even reproducible, testing of RTS. Since, each execution ordering can be regarded as a sequential program, it becomes possible to use techniques for testing of sequential software in testing multitasking real-time system software. We also show how this analysis and testing strategy can be extended to encompass distributed computations, communication latencies and the effects of global clock synchronization. The number of execution orderings is an objective measure of the testability of a system since it indicates how many behaviors the system can exhibit during runtime. In the pursuit of finding errors we must thus cover all these execution orderings. The fewer the orderings the better the testability.

A communication protocol for hard and soft real-time systems

The distributed real-time community is mainly divided into two camps: hard real-time and soft real-time; hybrid systems have been considered but not to any great extent. In this paper we propose a hybrid communication protocol, which forms a foundation for interconnection of nodes in a distributed hybrid real-time system. The protocol will enable both hard and soft real-time frames on a broadcast communication bus, yet still guarantees the hard real-time behaviour. The protocol will utilise the communication bandwidth more efficiently, which is relevant to cost-sensitive embedded applications.

Handling interrupts in testing of distributed real-time systems

Testing techniques for sequential software are well established. Unfortunately, these testing techniques are not directly applicable to distributed real-time systems. In previous work we have proposed a method for reproducible and deterministic testing of distributed real-time systems. This method identifies all possible orderings of task starts, preemptions and completions. Each ordering is regarded as a sequential program, and can as such be tested with standard sequential testing techniques. In this paper we extend our method to also handle interrupts. This is essential since critical parts of most real world systems are managed by interrupts.

Hardware support for priority inheritance

Previous work has shown that a system-on-a-chip lock cache (SoCLC) reduces on-chip memory traffic, provides a fair and fast lock hand-off, simplifies software, increases the real-time predictability of the system and improves performance. In this research work, we extend the SoCLC mechanism with a priority inheritance support implemented in hardware. Priority inheritance provides a higher level of real-time guarantees for synchronizing application tasks. Experimental results indicate that our SoCLC hardware mechanism with priority inheritance achieves a 36% speedup in lock delay, 88% speedup in lock latency, and 15% speedup in the overall execution time when compared to its software counterpart. The cost in terms of additional hardware area for the SoCLC with priority inheritance is approximately 10000 NAND2 gates.

Dynamic Patching of Embedded Software

In this paper, we present a method for patching embedded multitasking real-time systems applications during runtime, for instrumentation purposes. The method uses binary modification techniques and automates the entire patch process. The method makes it possible to insert and remove instrumentation code without preparing the original source code. The method makes it possible to invoke code patches during run-time, without having to rely on dynamic linking of object files, or predeployment prepared dormant code. The actual modification of the executing target binary is performed in a safe and controlled manner by a dedicated low interference mutation task

Testing of Multi-Tasking Real-Time Systems with Critical Sections

In this paper we address the problem of testing real-time software in the functional domain. In order to achieve reproducible and deterministic test results of an entire multitasking real-time system it is essential not to only consider inputs and outputs, but also the order in which tasks communicate and synchronize with each other. We present a deterministic white-box system-level control-flow testing method for deterministic integration testing of real-time system software. We specifically address fixed priority scheduled real-time systems where synchronization is resolved using the Priority Ceiling Emulation Protocol or offsets in time. The method includes a testing strategy where the coverage criterion is defined by the number of paths in the system control flow. The method also includes a reachability algorithm for deriving all possible paths in terms of orderings of task starts, preemptions and completions of tasks executing in a real-time system. The deterministic testing strategy allows test methods for sequential programs to be applied, since each identified ordering can be regarded as a sequential program.