Ricardo Barbosa

Injection of faults at component interfaces and inside the component code: are they equivalent?

The injection of interface faults through API parameter corruption is a technique commonly used in experimental dependability evaluation. Although the interface faults injected by this approach can be considered as a possible consequence of actual software faults in real applications, the question of whether the typical exceptional inputs and invalid parameters used in these techniques do represent the consequences of software bugs is largely an open issue. This question may not be an issue in the context of robustness testing aimed at the identification of weaknesses in software components. However, the use of interface faults by API parameter corruption as a general approach for dependability evaluation in component-based systems requires an in depth study of interface faults and a close observation of the way internal component faults propagate to the component interfaces. In this paper we present the results of experimental evaluation of realistic component-based applications developed in Java and C using the injection of interface faults by API parameter corruption and the injection of software faults inside the components by modification of the target code. The faults injected inside software components emulate typical programming errors and are based on an extensive field data study previously published. The results show the consequences of internal component faults in several operational scenarios and provide empirical evidences that interface faults and software component faults cause different impact in the system

Experimental Risk Assessment and Comparison Using Software Fault Injection

One important question in component-based software development is how to estimate the risk of using COTS components, as the components may have hidden faults and no source code available. This question is particularly relevant in scenarios where it is necessary to choose the most reliable COTS when several alternative components of equivalent functionality are available. This paper proposes a practical approach to assess the risk of using a given software component (COTS or non-COTS). Although we focus on comparing components, the methodology can be useful to assess the risk in individual modules. The proposed approach uses the injection of realistic software faults to assess the impact of possible component failures and uses software complexity metrics to estimate the probability of residual defects in software components. The proposed approach is demonstrated and evaluated in a comparison scenario using two real off-the-shelf components (the RTEMS and the RTLinux real-time operating system) in a realistic application of a satellite data handling application used by the European Space Agency.

Experimental Analysis of Binary-Level Software Fault Injection in Complex Software

The injection of software faults (i.e., bugs) by mutating the binary executable code of a program enables the experimental dependability evaluation of systems for which the source code is not available. This approach requires that programming constructs used in the source code should be identified by looking only at the binary code, since the injection is performed at this level. Unfortunately, it is a difficult task to inject faults in the binary code that correctly emulate software defects in the source code. The accuracy of binary-level software fault injection techniques is therefore a major concern for their adoption in real-world scenarios. In this work, we propose a method for assessing the accuracy of binary-level fault injection, and provide an extensive experimental evaluation of a binary-level technique, G-SWFIT, in order to assess its limitations in a real-world complex software system. We injected more than 12 thousand binary-level faults in the OS and application code of the system, and we compared them with faults injected in the source code by using the same fault types of G-SWFIT. The method was effective at highlighting the pitfalls that can occur in the implementation of G-SWFIT. Our analysis shows that G-SWFIT can achieve an improved degree of accuracy if these pitfalls are avoided.

Verification and Validation of (Real Time) COTS Products using Fault Injection Techniques

With the goal of reducing time to market and project costs, the current trend of real time business and mission critical systems is evolving from the development of custom made applications to the use of commercial off the shelf (COTS) products. Obviously, the same confidence and quality of the custom made software components is expected from the commercial applications. In most cases, such products (COTS) are not designed with stringent timing and/or safety requirements as priorities. Thus, to decrease the gap between the use of custom made components and COTS components, this paper presents a methodology for evaluating COTS products in the scope of dependable, real time systems, through the application of fault injection techniques at key points of the software engineering process. By combining the use of robustness testing (fault injection at interface level) with software fault injection (using educated fault injection operators), a COTS component can be assessed in the context of the system it will belong to, with special emphasis given to timing and safety constraints that are usually imposed by the target real time dependable environment. In the course of this work, three case studies have been performed to assess the methodology using realistic scenarios that used common COTS products. Results for one case study are presented

A view on the past and future of fault injection

Fault injection is a well-known technology that enables assessing dependability attributes of computer systems. Many works on fault injection have been developed in the past, and fault injection has been used in different application domains. This fast abstract briefly revises previous applications of fault injection, especially for embedded systems, and puts forward ideas on its future use, both in terms of application areas and business markets.

Qualification and Selection of Off-the-Shelf Components for Safety Critical Systems: A Systematic Approach

Mission critical systems are increasingly been developed by means of Off-The-Shelf (OTS) items since this allows reducing development costs. Crucial issues to be properly treated are (i) to assess the quality of each potential OTSitem to be used and (ii) to select the one that better fits the system requirements. Despite the importance of these issues, the current literature lacks a systematic approach to perform the previous two operations. The aim of this paper is to present a framework that can overcome this lack. Reasoning from the available product assurance standards for certifying mission critical systems, the proposed approach is based on the customized quality model that describes the quality attributes. Such quality model will guide a proper evaluation of OTS products, and the choice of which product to use is based on the outcomes of such an evaluation process. This framework represents a key solution to have a dominant role in the market of mission critical systems due to the demanding request by manufactures of such systems for an efficient qualification/certification process.

A preliminary fault injection framework for evaluating multicore systems

Multicore processors are becoming more and more attractive in embedded and safety-critical domains because they allow increasing the performance by ensuring reduced power consumption. However, moving to multicore systems raises novel dependability challenges: the number of cores, concurrency issues, shared resources and interconnections among cores make it hard to develop and validate software deployed on the top of multicore processors. This paper discusses a preliminary fault injection framework, which aims to investigate dependability properties of multicore-based systems. The proposed framework leverages the error reporting architecture provided by modern processors and has been instantiated in the context of the Intel Core i7 processor. Fault injection campaigns have been conducted under the Linux OS to show the benefits of the framework.

On the Use of Boundary Scan for Code Coverage of Critical Embedded Software

Code coverage tools are becoming increasingly popular as valuable aids in assessing and improving the quality of software structural tests. For some industries, such as aeronautics or space, they are mandatory in order to comply with standards and to help reduce the validation time of the applications. These tools usually rely on code instrumentation, thus introducing important time and memory overheads that may jeopardize its applicability to embedded and real-time systems. This paper explores the use of IEEE 1149.1 (boundary scan) infrastructure and on-chip debugging facilities from embedded processors for collecting the program execution trace during tests, without the introduction of any extra code, and then extracting detailed code coverage analysis and profiling information. We are currently developing an extension to the csXception tool to include such capabilities, in order to study the advantages, difficulties and impediments of using boundary scan for code coverage.

The PreCertification Kit for Operating Systems in Safety Domains

In present-day, software is taking over functionalities traditionally implemented in hardware, therefore the software architecture has been more complex and large. In such software architecture is common to be present an Operating System (OS). However, in safety domains (e.g., avionic, railway) it is mandatory to be compliant with a safety standard (e.g., DO178B), this means that evidence on the software life cycle of the software components, and therefore also of the OS, should be available. Those evidences that represent the certification package of the OS might not be available for commercial or Open Source OSs, hence their certification requires a complementary creation of evidence to serve as certification inputs. The certification process is costly, thus the system integrator must carefully select the candidate OS. Hence, it would be of great value to support the system integrator in selecting the more suitable OS to certify. In this position paper, we introduce our future research on the development of a Precertification kit (PK), that is, a framework that supports the evaluation of OS in what concerns certification requirements. Also, the PK is a valuable tool that can be integrated in the development toolchain for the implementation of safer and higher quality OS and, provides additional evidences to use for the certification package.

Safety-Critical Standards for Verification and Validation

Verification and Validation represent key activities to be properly conducted during the development of safety-critical systems. Due to their importance, international organizations have issued regulations to disciple how these activities have to be performed in order to achieve systems of high quality. In particular, each of them indicates a definition of what safety means, proper qualitative and quantitative properties for evaluating the quality of the system under development, and a set of methodologies to be used for assessing the fulfillment of the mentioned properties. These standards are today an essential tool for ensuring the required safety levels in many domains that require extremely high dependability. This paper summarizes the analysis on a set of well-known safety standards in different domains of critical systems with the intend of highlighting similarities and differences among them, pointing out common areas of interest and reporting on which features the newest (and upcoming) standards are focusing.