Maha Kooli

A survey on simulation-based fault injection tools for complex systems

Dependability is a key decision factor in todays global business environment. A powerful method that permits to evaluate the dependability of a system is the fault injection. The principle of this approach is to insert faults into the system and to monitor its responses in order to observe its behavior in the presence of faults. Several fault injection techniques and tools have been developed and experimentally tested. They could be mainly grouped into three categories: hardware fault injection, simulation-based fault injection, and emulation-based fault injection. This paper presents a survey on the simulation-based fault injection techniques, with a focus on complex micro-processor based systems.

Software testing and software fault injection

Reliability is one of the most important characteristics of the system quality. It is defined as the probability of failure-free operation of system for a specified period of time in a specified environment. For micro-processor based systems, reliability includes both software and hardware reliability. Many methods and techniques have been proposed in the literature so far to evaluate and test both software faults (e.g., Mutation Testing, Control Flow Testing, Data Flow Testing) and hardware faults (e.g. Fault Injection). In this paper, we present a survey of proposed techniques and methods to evaluate software and hardware reliability, and we study the possibility to explore them to evaluate the role of the software stack to evaluate system reliability face to hardware faults.

Fault injection tools based on Virtual Machines

Transient and permanent faults in complex digital systems used for safety-critical applications may result in catastrophic effects. It becomes therefore extremely important to adopt techniques such as fault injection to observe the behavior of the system in the presence of faults. Several tools have been proposed in the literature that support fault injection. However, few of them allow observing complex computer-based systems. This paper presents current advances in this field, by focusing on Low Level Virtual Machine (LLVM) based fault injectors and FAUMachine. We give an overview of the LLVM environment, and two based fault injection tools: LLFI and KULFI. Moreover, we introduce FAUmachine as virtual machine that supports fault injection in different components of the system (memory, disk and network). We present limitations and difficulties of the tool, and we propose a new implementation that allows injecting faults into the register of the target processor. The paper concludes with a comparison between the fault injection tools based on virtual machine in a first level, and between the LLVM-based fault injection tools in a second level.

Cache-aware reliability evaluation through LLVM-based analysis and fault injection

Reliability evaluation is a high costly process that is mainly carried out through fault injection or by means of analytical techniques. While the analytical techniques are fast but inaccurate, the fault injection is more accurate but extremely time consuming. This paper presents an hybrid approach combining analytical and fault injection techniques in order to evaluate the reliability of a computing system, by considering errors that affect both the data and the instruction cache. Compared to existing techniques, instead of targeting the hardware model of the cache (e.g., VHDL description), we only consider the running application (i.e., the software layer). The proposed approach is based on the Low-Level Virtual Machine (LLVM) framework coupled with a cache emulator. As input, the tool requires the application source code, the cache size and policy, and the target microprocessor instruction set. The main advantage of the proposed approach is the achieved speed up quantified in magnitude orders compared to existing fault injection techniques. For the validation, we compare the simulation results to those obtained with an FPGA-based fault injector. The similarity of the results proves the accuracy of the approach.

Cache- and register-aware system reliability evaluation based on data lifetime analysis

Developing new methods to evaluate the software reliability in an early design stage of the system can save the design costs and efforts, and will positively impact product time-to-market. This paper introduces a new approach to evaluate, at early design phase, the reliability of a computing system running a software. The approach can be used when the hardware architecture is not completely defined yet. In order to be independent of the hardware architecture and at the same time accurate, we propose to use the Low-Level Virtual Machine (LLVM) framework. In addition, to reduce the reliability evaluation time, our approach consists in analyzing the variable lifetimes to compute the probability of masked faults. Finally, to achieve a better characterization we propose to consider also the presence of caches and register files. For this purpose, a cache emulator as well as a register file emulator are developed. Simulations run with our approach produce very similar results to those run with a hardware-based fault injector. This proves the accuracy of our approach to evaluate system reliability with a gain in the simulation time and without requiring a hardware platform.

System-level reliability evaluation through cache-aware software-based fault injection

Developing new methods to evaluate the software reliability in an early design stage of the system can save the design costs and efforts, and will positively impact the product time-to-market. In this paper, we propose a novel fault injection technique to evaluate the reliability of a computing system running a software at early design stage where the hardware architecture is not completely defined yet. The proposed approach efficiently operates on the original source code of the software in order to inject transient faults in the data or the instructions. To be accurate and to achieve a better characterization of the system, we simulate faults occurring in the system memory units such as the data cache and the RAM by developing a system emulator. To validate our approach, we compare the simulation results to those obtained with an FPGA-based fault injector. The similarity of the results proves the accuracy of our approach to evaluate system reliability with a gain in the execution time and without requiring a fully defined hardware system.

Cross-layer reliability evaluation, moving from the hardware architecture to the system level

Advanced computing systems realized in forthcoming technologies hold the promise of a significant increase of computational capabilities. However, the same path that is leading technologies toward these remarkable achievements is also making electronic devices increasingly unreliable. Developing new methods to evaluate the reliability of these systems in an early design stage has the potential to save costs, produce optimized designs and have a positive impact on the product time-to-market.CLERECO European FP7 research project addresses early reliability evaluation with a cross-layer approach across different computing disciplines, across computing system layers and across computing market segments. The fundamental objective of the project is to investigate in depth a methodology to assess system reliability early in the design cycle of the future systems of the emerging computing continuum. This paper presents a general overview of the CLERECO project focusing on the main tools and models that are being developed that could be of interest for the research community and engineering practice.

Cross-layer system reliability assessment framework for hardware faults

System reliability estimation during early design phases facilitates informed decisions for the integration of effective protection mechanisms against different classes of hardware faults. When not all system abstraction layers (technology, circuit, microarchitecture, software) are factored in such an estimation model, the delivered reliability reports must be excessively pessimistic and thus lead to unacceptably expensive, over-designed systems. We propose a scalable, cross-layer methodology and supporting suite of tools for accurate but fast estimations of computing systems reliability. The backbone of the methodology is a component-based Bayesian model, which effectively calculates system reliability based on the masking probabilities of individual hardware and software components considering their complex interactions. Our detailed experimental evaluation for different technologies, microarchitectures, and benchmarks demonstrates that the proposed model delivers very accurate reliability estimations (FIT rates) compared to statistically significant but slow fault injection campaigns at the microarchitecture level.

Computing reliability: On the differences between software testing and software fault injection techniques

System reliability has become a main concern during the computer-based system design process. It is one of the most important characteristics of the system quality. The continuous increase of the system complexity makes the reliability evaluation extremely costly. Therefore, there is need to develop new methods with less cost and effort. Furthermore, the system is vulnerable to both software and hardware faults. While the software faults are usually introduced by the programmer either at the design or the implementation stage of the software, the hardware faults are caused by physical phenomena affecting the hardware components, such as environmental perturbations, manufacturing defects, and aging-related phenomena. The software faults can only impact the software components. However, the hardware faults can propagate through the different system layers, and affect both the hardware and the software. This paper discusses the differences between the software testing and the software fault injections techniques used for reliability evaluation. We describe the mutation analysis as a method mainly used in software testing. Then, we detail the fault injection as a technique to evaluate the system reliability. Finally, we discuss how to use software mutation analysis in order to evaluate, at software level, the system reliability against hardware faults. The main advantage of this technique is its usability at early design stage of the system, when the instruction set architecture is not available. Experimental results run to evaluate faults occurring the memory show that the proposed approach significantly reduces the complexity of the system reliability evaluation in terms of time and cost.

Reliability of computing systems: From flip flops to variables

Reliability evaluation is a critical task in computing systems. From one side, the results must be accurate enough not to under-or over-estimate the overall system reliability (thus either resulting in a non-reliable system, or a system for which too expensive solutions have been adopted). On the other side, the time required for the analysis should be kept at the minimum. This paper presents some new advances in the reliability assessment of computing systems, by showing techniques targeting both hardware and software levels, and the combination of both.