Peter Folkesson

Comparison of physical and software-implemented fault injection techniques

This paper addresses the issue of characterizing the respective impact of fault injection techniques. Three physical techniques and one software-implemented technique that have been used to assess the fault tolerance features of the MARS fault-tolerant distributed real-time system are compared and analyzed. After a short summary of the fault tolerance features of the MARS architecture and especially of the error detection mechanisms that were used to compare the erroneous behaviors induced by the fault injection techniques considered, we describe the common distributed testbed and test scenario implemented to perform a coherent set of fault injection campaigns. The main features of the four fault injection techniques considered are then briefly described and the results obtained are finally presented and discussed. Emphasis is put on the analysis of the specific impact and merit of each injection technique.

GOOFI: generic object-oriented fault injection tool

We present a new fault injection tool called GOOFI (Generic Object-Oriented Fault Injection). GOOFI is designed to be adaptable to various target systems and different fault injection techniques. The tool is highly portable between different host platforms since it relies on the Java programming language and an SQL compatible database. The current version of the tool supports pre-runtime software implemented fault injection and scan-chain implemented fault injection.

A comparison of simulation based and scan chain implemented fault injection

This paper compares two fault injection techniques: scan chain implemented fault injection (SCIFI), i.e. fault injection in a physical system using built in test logic, and fault injection in a VHDL software simulation model of a system. The fault injections were used to evaluate the error detection mechanisms included in the Thor RISC microprocessor, developed by Saab Ericsson Space AB. The Thor microprocessor uses several advanced error detection mechanisms including control flow checking, stack range checking and variable constraint checking. A newly developed tool called FIMB UL (Fault Injection and Monitoring using BUilt in Logic), which uses the Test Access Port (TAP) of the Thor CPU to do fault injection, is presented. The simulations were carried out using the MEFISTO-C tool and a highly detailed VHDL model of the Thor processor. The results show that the larger fault set available in the simulations caused only minor differences in the error detection distribution compared to SCIFI and that the overall error coverage was lower using SCIFI (90-94% vs. 94-96% using simulation based fault injection).

Integration and Comparison of Three Physical Fault Injection Techniques

This paper describes and compares three physical fault injection techniques — heavy-ion radiation, pin-level injection, and electromagnetic interference — and their use in the validation of MARS, a fault-tolerant distributed real-time system. The main features of the injection techniques are first summarised and analysed, and then the MARS error detection mechanisms are described. The distributed testbed set-up and the common test scenario implemented to perform a coherent set of experiments by applying the three fault injection techniques are also described. The results are presented and discussed; special emphasis is put on the comparison of the specific impact of each technique.

Assembly-Level pre-injection analysis for improving fault injection efficiency

This paper describes a fully automated pre-injection analysis technique aimed at reducing the cost of fault injection campaigns. The technique optimizes the fault-space by utilizing assembly-level knowledge of the target system in order to place single bit-flips in registers and memory locations only immediately before these are read by the executed instructions. This way, faults (time-location pairs) that are overwritten or have identical impact on program execution are removed. Experimental results obtained by random sampling of the optimized fault-space and the complete (non-optimized) fault-space are compared for two different workloads running on a MPC565 microcontroller. The pre-injection analysis yields an increase of one order of magnitude in the effectiveness of faults, a reduction of the fault-space of two orders of magnitude in the case of CPU-registers and four to five orders of magnitude in the case of memory locations, while preserving a similar estimation of the error detection coverage.

Reducing critical failures for control algorithms using executable assertions and best effort recovery

Systems that use f+1 computer nodes to tolerate f node failures ordinarily require that the computer nodes have strong failure semantics, i.e. a node should either produce correct results or no results at all. We show that this requirement can be relaxed for control applications, as control algorithms inherently compensate for a class of value failures. Value failures occur when an error escapes the error detection mechanisms in the computer node and an erroneous value is sent to the actuators of the control system. Fault injection experiments show that 89% of the value failures caused by bit flips in a CPU had no or minor impact on the controlled object. However, the experiments also show that 11% of the value failures had severe consequences. These failures were caused by bit flips affecting the state variables of the control algorithm. Another set of fault injection experiments showed that the percentage of value failures with severe consequences was reduced to 3% when the state variables were protected with executable assertions and best-effort recovery mechanisms.

Experimental evaluation of time-redundant execution for a brake-by-wire application

This paper presents an experimental evaluation of a brake-by-wire application that tolerates transient faults by temporal error masking. A specially designed real-time kernel that masks errors by triple time-redundant execution and voting executes the application on a fail-stop computer node. The objective is to reduce the number of node failures by masking errors at the computer node level. The real-time kernel always executes the application twice to detect errors, and ensures that a fail-stop failure occurs if there is not enough CPU-time available for a third execution and voting. Fault injection experiments show that temporal error masking reduced the number of fail-stop failures by 42% compared to executing the brake-by-wire task without time redundancy.

A framework for node-level fault tolerance in distributed real-time systems

This paper describes a framework for achieving node-level fault tolerance (NLFT) in distributed real-time systems. The objective of NLFT is to mask errors at the node level in order to reduce the probability of node failures and thereby improve system dependability. We describe an approach called lightweight NLFT where transient faults are masked locally in the nodes by time-redundant execution of application tasks. The advantages of light-weight NLFT is demonstrated by a reliability analysis of an example brake-by-wire architecture. The results show that the use of light-weight NLFT may provide 55% higher reliability after one year and almost 60% higher MTTF, compared to using fail-silent nodes.

Experimental dependability evaluation of a fail-bounded jet engine control system for unmanned aerial vehicles

This paper presents an experimental evaluation of a prototype jet engine controller intended for unmanned aerial vehicles (UAVs). The controller is implemented with commercial off-the-shelf (COTS) hardware based on the Motorola MPC565 microcontroller. We investigate the impact of single event upsets (SEUs) by injecting single bit-flip faults into main memory and CPU registers via the Nexus on-chip debug interface of the MPC565. To avoid the injection of non-effective faults, automated pre-injection analysis of the assembly code was utilized. Due to the inherent robustness of the software, most injected faults were still non-effective (69.4%) or caused bounded failures having only minor effect on the jet engine (7.0%), while 20.1% of the errors were detected by hardware exceptions and 1.9% were detected by executable assertions in the software. The remaining 1.6% is classified as critical failures. A majority of the critical failures were caused by erroneous Booleans or type conversions involving Booleans.

On the design of robust integrators for fail-bounded control systems

This paper describes the design and evaluation of a robust integrator for software-implemented control systems. The integrator is constructed as a generic component in the Simulink design tool, and can thus be used for robust implementation of a wide range of control algorithms. The integrator is designed to support the failbounded failure model for transient bit-flips that may occur in the CPU, main memory and I/O circuits of a control system. In particular, it allows the control system to detect and recover from bit-flips that cause data errors. Robustness is achieved by sequentially executing duplicated integrator code on the same processor to support error detection, and through the use of a recovery buffer that allows a roll-back to the previous integrator state when an error is detected. The effectiveness of the robust integrator was evaluated through fault injection experiments with a PI controller, where single bit flips were injected inside the CPU of the control system. No violations of the fail-bounded model were observed in the experiments.