Roberto Natella

On Fault Representativeness of Software Fault Injection

The injection of software faults in software components to assess the impact of these faults on other components or on the system as a whole, allowing the evaluation of fault tolerance, is relatively new compared to decades of research on hardware fault injection. This paper presents an extensive experimental study (more than 3.8 million individual experiments in three real systems) to evaluate the representativeness of faults injected by a state-of-the-art approach (G-SWFIT). Results show that a significant share (up to 72 percent) of injected faults cannot be considered representative of residual software faults as they are consistently detected by regression tests, and that the representativeness of injected faults is affected by the fault location within the system, resulting in different distributions of representative/nonrepresentative faults across files and functions. Therefore, we propose a new approach to refine the faultload by removing faults that are not representative of residual software faults. This filtering is essential to assure meaningful results and to reduce the cost (in terms of number of faults) of software fault injection campaigns in complex software. The proposed approach is based on classification algorithms, is fully automatic, and can be used for improving fault representativeness of existing software fault injection approaches.

A survey of software aging and rejuvenation studies

Software aging is a phenomenon plaguing many long-running complex software systems, which exhibit performance degradation or an increasing failure rate. Several strategies based on the proactive rejuvenation of the software state have been proposed to counteract software aging and prevent failures. This survey article provides an overview of studies on Software Aging and Rejuvenation (SAR) that have appeared in major journals and conference proceedings, with respect to the statistical approaches that have been used to forecast software aging phenomena and to plan rejuvenation, the kind of systems and aging effects that have been studied, and the techniques that have been proposed to rejuvenate complex software systems. The analysis is useful to identify key results from SAR research, and it is leveraged in this article to highlight trends and open issues.

Software Aging Analysis of the Linux Operating System

Software systems running continuously for a long time tend to show degrading performance and an increasing failure occurrence rate, due to error conditions that accrue over time and eventually lead the system to failure. This phenomenon is usually referred to as \textit{Software Aging}. Several long-running mission and safety critical applications have been reported to experience catastrophic aging-related failures. Software aging sources (i.e., aging-related bugs) may be hidden in several layers of a complex software system, ranging from the Operating System (OS) to the user application level. This paper presents a software aging analysis at the Operating System level, investigating software aging sources inside the Linux kernel. Linux is increasingly being employed in critical scenarios; this analysis intends to shed light on its behaviour from the aging perspective. The study is based on an experimental campaign designed to investigate the kernel internal behaviour over long running executions. By means of a kernel tracing tool specifically developed for this study, we collected relevant parameters of several kernel subsystems. Statistical analysis of collected data allowed us to confirm the presence of aging sources in Linux and to relate the observed aging dynamics to the monitored subsystems behaviour. The analysis output allowed us to infer potential sources of aging in the kernel subsystems.

Assessing Dependability with Software Fault Injection: A Survey

With the rise of software complexity, software-related accidents represent a significant threat for computer-based systems. Software Fault Injection is a method to anticipate worst-case scenarios caused by faulty software through the deliberate injection of software faults. This survey provides a comprehensive overview of the state of the art on Software Fault Injection to support researchers and practitioners in the selection of the approach that best fits their dependability assessment goals, and it discusses how these approaches have evolved to achieve fault representativeness, efficiency, and usability. The survey includes a description of relevant applications of Software Fault Injection in the context of fault-tolerant systems.

Fault triggers in open-source software: An experience report

With software systems becoming increasingly large and complex, many difficulties in coping with software bugs arise for developers. Despite good development practices, thorough testing, and proper maintenance policies, a non-negligible number of bugs remain in the released software. Understanding the type of residual bugs is fundamental for adopting proper countermeasures in current and future software releases. Depending on the fault triggering conditions that lead to a failure, developers can introduce fault-tolerance mechanisms and plan verification and validation strategies. In this paper, we analyze bugs in four large open-source software systems during their lifecycle, based on the concept of fault triggers. We first investigate how the type of system affects the bug type proportions, and their evolution over years. Then, an analysis of bug subtypes is performed, so as to better understand their nature, followed by a comparison with respect to attributes such as their average time to fix and severity.

Software Aging and Rejuvenation: Where We Are and Where We Are Going

After 16 years, a significant body of knowledge has been established in the area of Software Aging and Rejuvenation (SAR). In this paper, we survey papers about SAR that appeared in IEEE conferences and journals, identify where SAR research has been mostly focused, and highlight some aspects deserving more attention, with the aim to provoke a constructive discussion among SAR researches about where SAR has arrived and where it should be headed in the next future.

Predicting aging-related bugs using software complexity metrics

Long-running software systems tend to show degraded performance and an increased failure occurrence rate. This problem, known as Software Aging, which is typically related to the runtime accumulation of error conditions, is caused by the activation of the so-called Aging-Related Bugs (ARBs). This paper aims to predict the location of Aging-Related Bugs in complex software systems, so as to aid their identification during testing. First, we carried out a bug data analysis on three large software projects in order to collect data about ARBs. Then, a set of software complexity metrics were selected and extracted from the three projects. Finally, by using such metrics as predictor variables and machine learning algorithms, we built fault prediction models that can be used to predict which source code files are more prone to Aging-Related Bugs.

Experimental Analysis of Binary-Level Software Fault Injection in Complex Software

The injection of software faults (i.e., bugs) by mutating the binary executable code of a program enables the experimental dependability evaluation of systems for which the source code is not available. This approach requires that programming constructs used in the source code should be identified by looking only at the binary code, since the injection is performed at this level. Unfortunately, it is a difficult task to inject faults in the binary code that correctly emulate software defects in the source code. The accuracy of binary-level software fault injection techniques is therefore a major concern for their adoption in real-world scenarios. In this work, we propose a method for assessing the accuracy of binary-level fault injection, and provide an extensive experimental evaluation of a binary-level technique, G-SWFIT, in order to assess its limitations in a real-world complex software system. We injected more than 12 thousand binary-level faults in the OS and application code of the system, and we compared them with faults injected in the source code by using the same fault types of G-SWFIT. The method was effective at highlighting the pitfalls that can occur in the implementation of G-SWFIT. Our analysis shows that G-SWFIT can achieve an improved degree of accuracy if these pitfalls are avoided.

Fault Injection for Software Certification

As software becomes more pervasive and complex, its increasingly important to ensure that a system will be safe even in the presence of residual software faults (or bugs). Software fault injection consists of the deliberate introduction of software faults for assessing the impact of faulty software on a system and improving its fault tolerance. SFI has been included as a recommended practice in recent safety standards and has therefore gained interest among practitioners, but its still unclear how it can be effectively used for certification purposes. In this article, the authors discuss the adoption of SFI in the context of safety certification, present a tool for the injection of realistic software faults, and show the usage of that tool in evaluating and improving the robustness of an operating system used in the avionic domain.

Memory leak analysis of mission-critical middleware

Memory leaks are recognized to be one of the major causes of memory exhaustion problems in complex software systems. This paper proposes a practical approach to detect aging phenomena caused by memory leaks in distributed objects Off-The-Shelf middleware, which are commonly used to develop critical applications. The approach, which is validated on a real-world case study from the Air Traffic Control domain, defines algorithms and ad hoc support tools to perform data filtering and to find the best trade off between experimentation time and statistical accuracy of aging trend estimates. Experiments show that fixing memory leaks is not always the key to solve memory exhaustion problems.