Julio Gallardo

Application of statistical testing to smart device code

Statistical testing can produce dependability information by estimating the probability of failure on demand for a system. In this paper we explore issues related to the construction of statistical test-cases for the firmware of a smart device. Our aim is to share our own experience with the technique and we expect to derive some general insights into what needs to be taken into account when designing such test-cases for smart devices. This provides useful information for situations where one would like to perform statistical testing but where no access to the device code is available. This paper describes the status quo of this project, further results are expected to emerge from this work

A case for new statistical software testing models

There is growing interest in statistical software testing (SST) as a software assurance technique. While the approach has major attractions, there is a need for new statistical models to infer failure probabilities from SST. The paper constructs a simple but realistic case in which the traditional binomial model does not work. The paper shows that if possible test failure dependencies are neglected, could the failure probability would be underestimated. The paper compares the results of our new probability model based on pairwise failures with results achieved when applying the traditional single-urn model, i.e., assuming no dependencies in the failure process

Assessing Diagnostic Techniques for Fault Tolerance in Software

One of the main concerns in software safety critical applications is to ensure sufficient reliability if one cannot prove the absence of faults. Fault tolerance (FT) provides a plausible method for improving reliability claims in the presence of systematic failures in software. It is plausible that some software FT techniques offer increased protection than others. However, the extent of claims that can be made for different FT software architectures remains unclear. We investigate an approach to FT that integrates data diversity (DD) assertions and traditional assertions (TA). We also present the principles of a method to assess the effectiveness of the approach. The aim of this approach is to make it possible to evolve more powerful FT and thereby improve reliability. This is a step towards the aim of understanding the effectiveness of FT safety-critical applications and thus making it easier to use FT in safety arguments

Assessment of Data Diversity Methods for Software Fault Tolerance Based on Mutation Analysis

One of the main concerns in safety-critical software is to ensure sufficient reliability because proof of the absence of systematic failures has proved to be an unrealistic goal. fault-tolerance (FT) is one method for improving reliability claims. It is reasonable to assume that some software FT techniques offer more protection than others, but the relative effectiveness of different software FT schemes remains unclear. We present the principles of a method to assess the effectiveness of FT using mutation analysis. The aim of this approach is to observe the power of FT directly and use this empirical process to evolve more powerful forms of FT. We also investigate an approach to FT that integrates data diversity (DD) assertions and TA. This work is part of a longer term goal to use FT in quantitative safety arguments for safety critical systems.

Use of Graphical Probabilistic Models to build SIL claims based on software safety standards such as IEC61508-3

Software reliability assessment is ‘different’ from traditional reliability techniques and requires a different process. The use of development standards is common in current good practice. Software safety standards recommend processes to design and assure the integrity of safety-related software. However the reasoning on the validity of these processes is complex and opaque. In this paper an attempt is made to use Graphical Probability Models (GPMs) to formalise the reasoning that underpins the construction of a Safety Integrity Level (SIL) claim based upon a safety standard such as IEC61508 Part 3. There are three major benefits: the reasoning becomes compact and easy to comprehend, facilitating its scrutiny, and making it easier for experts to develop a consensus using a common formal framework; the task of the regulator is supported because to some degree the subjective reasoning which underpins the expert consensus on compliance is captured in the structure of the GPM; the users will benefit from software tools that support implementation of IEC61508, such tools even have the potential to allow cost-benefit analysis of alternative safety assurance techniques.

The Effectiveness of Statistical Testing when Applied to Logic Systems

In this paper we demonstrate the effectiveness of statistical testing for error detection on the example of a Programmable Logic System (PLS). The introduction of statistical testing arose from the wish to quantify the PLSs reliability. An appropriate statistical testing algorithm was devised and implemented, which is described in detail in this paper. We compare the results of statistical testing with those of a variety of other testing methods employed on the PLS. In terms of differences detected per number of tests, statistical testing showed an outstanding effectiveness. Furthermore, it detected a problem, which was missed by all other testing techniques. This together with its potential for reliability quantification illustrates its importance for system validation as part of a risk-based safety-case.