Yves Crouzet

Fault injection for dependability validation: a methodology and some applications

The authors address the problem of validating the dependability of fault-tolerant computing systems, in particular, the validation of the fault-tolerance mechanisms. The proposed approach is based on the use of fault injection at the physical level on a hardware/software prototype of the system considered. The place of this approach in a validation-directed design process and with respect to related work on fault injection is clearly identified. The major requirements and problems related to the development and application of a validation methodology based on fault injection are presented and discussed. Emphasis is put on the definition, analysis, and use of the experimental dependability measures that can be obtained. The proposed methodology has been implemented through the realization of a general pin-level fault injection tool (MESSALINE), and its usefulness is demonstrated by the application of MESSALINE to the experimental validation of two systems: a subsystem of a centralized computerized interlocking system for railway control applications and a distributed system corresponding to the current implementation of the dependable communication system of the ESPRIT Delta-4 Project. >

Fault injection and dependability evaluation of fault-tolerant systems

The authors describe a dependability evaluation method based on fault injection that establishes the link between the experimental evaluation of the fault tolerance process and the fault occurrence process. The main characteristics of a fault injection test sequence aimed at evaluating the coverage of the fault tolerance process are presented. Emphasis is given to the derivation of experimental measures. The various steps by which the fault occurrence and fault tolerance processes are combined to evaluate dependability measures are identified and their interactions are analyzed. The method is illustrated by an application to the dependability evaluation of the distributed fault-tolerant architecture of the Esprit Delta-4 Project. >

Comparison of physical and software-implemented fault injection techniques

This paper addresses the issue of characterizing the respective impact of fault injection techniques. Three physical techniques and one software-implemented technique that have been used to assess the fault tolerance features of the MARS fault-tolerant distributed real-time system are compared and analyzed. After a short summary of the fault tolerance features of the MARS architecture and especially of the error detection mechanisms that were used to compare the erroneous behaviors induced by the fault injection techniques considered, we describe the common distributed testbed and test scenario implemented to perform a coherent set of fault injection campaigns. The main features of the four fault injection techniques considered are then briefly described and the results obtained are finally presented and discussed. Emphasis is put on the analysis of the specific impact and merit of each injection technique.

Fault injection for dependability validation of fault-tolerant computing systems

The authors address the dependability validation of fault-tolerant computing systems and more specifically the validation of the fault-tolerance mechanisms. Their approach is based on the use of fault injection at the physical level on a hardware/software prototype of the system considered. The place of this approach in a validation-directed design process as well as its place with respect to related works on fault injection are identified. The major requirements and problems related to the development and application of a validation methodology based on fault injection are presented and discussed. The proposed methodology has been implemented through the realization of a general physical-fault injection tool (MESSALINE) whose usefulness is demonstrated by its application to the experimental validation of a subsystem of a computerized interlocking system for railway control applications.<<ETX>>

MEFISTO-L: a VHDL-based fault injection tool for the experimental assessment of fault tolerance

The early assessment of the adequacy of fault tolerance mechanisms (FTMs), and the subsequent removal of fault tolerance deficiency faults (ftd-faults), are essential tasks in the design process of dependable computer systems. The paper is centered on the description and application of the features of MEFISTO-L, the fault injection tool for VHDL models, being developed at LAAS for supporting the strategy that we have proposed for testing FTMs. The paper first describes the overall testing framework in which MEFISTO-L is incorporated. The main guidelines for the design of MEFISTO-L and its objectives, attributes, implementation and use are then described. Special attention is given to the main original and innovative features: i) the embedded VHDL code analyzer, ii) the observation and injection mechanisms, iii) their synchronization, and iv) their automatic placement in the target VHDL model.

An experimental study on software structural testing: deterministic versus random input generation

The fault revealing power of different test patterns derived from ten structural test criteria currently referred to in unit testing is investigated. Experiments performed on four programs that are pieces of a real-life software system from the nuclear field are reported. Three test input generation techniques are studied: (1) deterministic choice, (2) random selection based on an input probability distribution determined according to the adopted structural test criterion, and (3) random selection from a uniform distribution on the input domain. Mutation analysis is used to assess the test set efficiency with respect to error detection. The experimental results involve a total of 2914 mutants. They show that structural statistical testing, which exhibits the highest mutation scores, leaving alive only six from 2816 nonequivalent mutants within short testing times, is the most efficient. A regards unit testing of programs whose structure remains tractable, the experiments show the adequacy of a fault removal strategy combining statistical and deterministic test patterns.<<ETX>>

Experimental evaluation of the fault tolerance of an atomic multicast system

The authors present a study of the validation of a dependable local area network providing multipoint communication services based on an atomic multicast protocol. This protocol is implemented in specialized communication servers, that exhibit the fail-silent property, i.e. a kind of halt-on-failure behavior enforced by self-checking hardware. The tests that have been carried out utilize physical fault injection and have two objectives: (1) to estimate the coverage of the self-checking mechanisms of the communication servers, and (2) to test the properties that characterize the service provided by the atomic multicast protocol in the presence of faults. The testbed that has been developed to carry out the fault-injection experiments is described, and the major results are presented and analyzed. It is concluded that the fault-injection test sequence has evidenced the limited performance of the self-checking mechanisms implemented on the tested NAC (network attachment controller) and justified (especially for the main board) the need for the improved self-checking mechanisms implemented in an enhanced NAC architecture using duplicated circuitry. >

Fault injection for formal testing of fault tolerance

This study addresses the use of fault injection for explicitly removing design/implementation faults in complex fault-tolerance algorithms and mechanisms (FTAM), viz, fault-tolerance deficiency faults. A formalism is introduced to represent the FTAM by a set of assertions. This formalism enables an execution tree to be generated, where each path from the root to a leaf of the tree is a well-defined formula. The set of well-defined formulas constitutes a useful framework that fully characterizes the test sequence. The input patterns of the test sequence (fault and activation domains) then are determined to fewer specific structural criteria over the execution tree (activation of proper sets of paths). This provides a framework for generating a functional deterministic test for programs that implement complex FTAM. This methodology has been used to extend a debugging tool aimed at testing fault tolerance protocols developed by BULL France. It has been applied successfully to the injection of faults in the inter-replica protocol that supports the application-level fault-tolerance features of the architecture of the ESPRIT-funded Delta-4 project. The results of these experiments are analyzed in detail. In particular, even though the target protocol had been independently verified formally, the application of the proposed testing strategy revealed two fault-tolerance deficiency faults.

Fault injection for the formal testing of fault tolerance

The authors address the issue of the use of fault injection for explicitly removing design/implementation faults in fault tolerance algorithms and mechanisms. A formalism is introduced that represents the fault tolerance algorithms and mechanisms by means of a set of assertions. This formalism enables the execution tree to be presented, where each path from the root to a leaf of the tree is a well-defined formula. It provides a framework for the generation of a functional deterministic test for programs implementing complex fault tolerance algorithms and mechanisms. This methodology has been used to extend a debugging tool aimed at testing fault tolerance protocols developed by BULL France. It has been successfully applied to the injection of faults in the inter-replica protocol supporting the application-level fault tolerance features of the architecture of the ESPRIT-funded Delta-4 project. The results of these experiments are discussed and analyzed.<<ETX>>

Software Statistical Testing

Statistical testing is based on a probabilistic generation of test data: structural or functional criteria serve as guides for defining an input profile and a test size. The method is intended to compensate for the imperfect connection of criteria with software faults, and should not be confused with random testing, a blind approach that uses a uniform profile over the input domain. First, the motivation and the theoretical foundation of statistical testing are presented. Then the feasibility of designing statistical test patterns is exemplified on a safety-critical component from the nuclear industry, and the fault-revealing power of these patterns is assessed through experiments conducted at two different levels: (i) unit testing of four functions extracted from the industrial component, statistical test data being designed according to classical structural criteria; (ii) testing of the whole component, statistical test data being designed from behaviour models deduced from the component specification. The results show the high fault-revealing power of statistical testing, and its greater efficiency in comparison to deterministic and random testing.