Mário Zenha Rela

RIFLE: A General Purpose Pin-level Fault Injector

This paper discusses the problems of pin-level fault injection for dependability validation and presents the architecture of a pin-level fault injector called RIFLE. This system can be adapted to a wide range of target systems and the faults are mainly injected in the processor pins. The injection of the faults is deterministic and can be reproduced if needed. Faults of different nature can be injected and the fault injector is able to detect whether the injected fault has produced an error or not without the requirement of feedback circuits. RIFLE can also detect specific circumstances in which the injected faults do not affect the target system. Sets of faults with specific impact on the target system can be generated. The paper also presents fault injection results showing the coverage and latency achieved with a set of simple behavior based error detection mechanisms. It is shown that up to 72,5% of the errors can be detected with fairly simple mechanisms. Furthermore, for over 90% of the faults the target system has behaved according to the fail-silent model, which suggests that a traditional computer equipped with simple error detection mechanisms is relatively close to a fail-silent computer.

Experimental evaluation of the fail-silent behaviour in programs with consistency checks

An important research topic deals with the investigation of whether a non-duplicated computer can be made fail-silent, since that behaviour is a-priori assumed in many algorithms. However, previous research has shown that in systems using a simple behaviour based error detection mechanism invisible to the programmer (e.g. memory protection), the percentage of fail-silent violations could be higher than 10%. Since the study of these errors has shown that they were mostly caused by pure data errors, we evaluate the effectiveness of software techniques capable of checking the semantics of the data, such as assertions, to detect these remaining errors. The results of injecting physical pin-level faults show that these tests can prevent about 40% of the fail-silent model violations that escape the simple hardware-based error detection techniques. In order to decouple the intrinsic limitations of the tests used from other factors that might affect its error detection capabilities, we evaluated a special class of software checks known for its high theoretical coverage: algorithm based fault tolerance (ABFT). The analysis of the remaining errors showed that most of them remained undetected due to short range control flow errors. When very simple software-based control flow checking was associated to the semantic tests, the target system, without any dedicated error detection hardware, behaved according to the fail-silent model for about 98% of all the faults injected.

Practical issues in the use of ABFT and a new failure model

We study the behavior of algorithm based fault tolerance (ABFT) techniques under faults injected according to a quite general fault model. Besides the problem of roundoff error in floating point arithmetic we identify two further weakpoints, namely lack of protection of data during input and output, and incorrect execution of the correctness checks. We propose the robust ABFT technique to handle those weakpoints. We then generalize it to programs that use assertions, where similar problems arise, leading to the technique of robust assertions, whose effectiveness is shown by fault injection experiments on a realistic control application. With this technique a system follows a new failure model, that we call fail-bounded, where with high probability all results produced are either correct or, if wrong, they are within a certain bound of the correct value, whose exact value depends on the output assertions used. We claim that this failure model is very useful to describe the behavior of many low redundancy systems.

A study of failure models in feedback control systems

Feedback control systems have a peculiar behavior that allows them to compensate for disturbances in the controlled application. This paper investigates whether this resilience also extends to disturbances originating from faults in the controller itself. The question of what kind of failure model is more effective in this type of system is addressed, with three different models being studied: arbitrary failure, fail-silent, and fail-bounded. The study is conducted essentially by experimental fault injection in the controller of one of the best known and most demanding of the benchmarks used in the control systems area: an inverted pendulum. The considered failure models are compared according to criteria based on the quality of the control action. Other insights gained from the experiments made are described, for instance on how to significantly increase dependability at a very low cost in the feedback controllers, and on the need for a different kind of real-time scheduling algorithm.

On the use of disaster prediction for failure-tolerance in feedback control systems

Feedback control algorithms are inherently designed to compensate for external disturbances that the controlled system may suffer. This resilience is also extensible to late or wrong control actions produced by a failed controller computer, providing a degree of fault tolerance without the use of any particular mechanism. However, some controller failures, due to their duration or value, may indeed collapse the system, and thus other recovery measures must be taken. This paper proposes the inclusion of an Oracle that calculates, in a timely manner, the controlled system behavior under a failed controller, and triggers recovery when the control algorithm is predictably no more able to compensate for a particular controller failure. The systems so built follow the Fail-Bounded model. The main contribution of this paper is to show how this model can be implemented in a practical way for the very important class of applications based on feedback control, thus turning that model into a technique that can be used effectively to build production systems. The method was validated experimentally through fault injection on the controller computer of an inverted pendulum, one of the most time-demanding control system benchmarks.

A strategy for evaluating feasible and unfeasible test cases for the evolutionary testing of object-oriented software

Evolutionary Testing is an emerging methodology for automatically producing high quality test data. The focus of our on-going work is precisely on generating test data for the structural unit-testing of object-oriented Java programs. The primary objective is that of efficiently guiding the search process towards the definition of a test set that achieves full structural coverage of the test object. However, the state problem of object-oriented programs requires specifying carefully fine-tuned methodologies that promote the traversal of problematic structures and difficult control-flow paths - which often involves the generation of complex and intricate test cases, that define elaborate state scenarios. This paper proposes a methodology for evaluating the quality of both feasible and unfeasible test cases - i.e., those that are effectively completed and terminate with a call to the method under test, and those that abort prematurely because a runtime exception is thrown during test case execution. With our approach, unfeasible test cases are considered at certain stages of the evolutionary search, promoting diversity and enhancing the possibility of achieving full coverage.

Reset-Driven Fault Tolerance

A common approach in embedded systems to achieve fault-tolerance is to reboot the computer whenever some non-permanent error is detected. All the system code and data are recreated from scratch, and a previously established checkpoint, hopefully not corrupted, is used to restart the application data. The confidence is thus restored on the activity of the computer. The idea explored in this paper is that of unconditionally resetting the computer in each control frame (the classic read sensors → calculate control action → update actuators cycle). A stable-storage based in RAM is used to preserve the systems state between consecutive cleanups and a standard watchdog timer guarantees that a reset is forced whenever an error crashes the system. We have evaluated this approach by using fault-injection in the controller of a standard temperature control system. The experimental observations show that the Reset-Driven Fault Tolerance is a very simple yet effective technique to improve reliability at an extremely low cost since it is a conceptually simple, software only solution with the advantage of being application independent.

Constraints on the Use of Boundary-Scan for Fault Injection

The Boundary-Scan technology was proposed fifteen years ago to overcome the limitations of testing digital devices due to the increasing complexity and greater miniaturization of integrated circuits and boards. The use of pin-level fault-injection faced similar difficulties and became obsolete for that reason. In this paper we discuss the use of the Boundary-Scan infrastructure for fault-injection purposes. Several fundamental constraints of such approach are identified. Generic digital systems and processors with Boundary-Scan based OCD are considered as target system candidates. We observe that by combining OCD mechanisms with modified boundary-scan cells most of the constraints reported are solved. Finally, some key properties of the technology such as the orthogonality to the purely functional architecture and the low abstraction level access as well as the standard interface and description language provided make it a good candidate to provide a standardized flexible fault injection framework.

The fault-tolerant architecture of the safe system

Abstract This paper presents a fault-tolerant architecture for industrial control applications aiming at achieving fault-tolerance with low redundancy rates by means of new error detection mechanisms, efficient reconfiguration schemes, and a cost effective use of dynamic redundancy techniques. The paper also stresses some novel aspects of this architecture such as a new method for fast stable-storage implementation and the use of a new approach to concurrent system-level error detection based on signature monitoring. This new error detection scheme has the potential to detect a larger kind of errors than traditional signature monitoring approaches. Furthermore, this new approach does not require special assemblers and loaders as is the case with the already existing techniques.

On the nature of deadlines

Abstract This article discusses the timeliness of real-time control services as seen by control engineering and real-time scientific communities, arguing that computer-controllers must be designed to meet nominal deadlines that, under special circumstances, can be missed as long as hard deadlines are still met. This is because the distinction between nominal and hard deadlines is central to the design of fault-tolerant real-time systems. This article justifies this view and, using concepts and methodologies derived from control systems engineering and real-time literature, develops a unified approach for establishing the nominal and the hard deadline of a time-critical control service. Additionally, it illustrates the introduced concepts with a case study concerning the control of a highly powered hydraulic press and suggests new areas of research.