Hélène Waeselynck

STATEMATE applied to statistical software testing

This paper is concerned with the use of statistical testing as a verification technique for complex software. Statistical testing involves exercising a program with random inputs, the test profile and the number of generated inputs being determined according to criteria based on program structure or software functionality. In case of complex programs, the probabilistic generation must be based on a black box analysis, the adopted criteria being defined from behavior models deduced from the specification. The proposed approach refers to a hierarchical specification produced in the STATEMATE environment. Its feasiblity is exemplified on a safety-critical module from the nuclear field, and the efficiency in revealing actual faults is investigated through experiments involving two versions of the module.

Testing levels for object-oriented software

One of the characteristics of object-oriented software is the complex dependency that may exist between classes due to inheritance, association and aggregation relationships. Hence, where to start testing and how to define an integration strategy are issues that require further investigation. This paper presents an approach to define a test order by exploiting a model produced during design stages (e.g., using OMT, UML), namely the class diagram. Our goal is to minimize the number of stubs to be constructed in order to decrease the cost of testing. This is done by testing a class after the classes it depends on. The novelty of the test order lies in the fact that it takes account of: (i) dynamic (polymorphism) dependencies; (ii) abstract classes that cannot be instantiated, making some testing levels infeasible. The test order is represented by a graph showing which testing levels must be done in sequence and which ones may be done independently. It also provides information about the classes involved in each level and how they are involved (e.g., instantiation or not). The approach is implemented in a tool called TOONS (testing level generator for object-oriented software). It is applied to an industrial case study from the avionics domain.

Towards a statistical approach to testing object-oriented programs

Statistical testing is based on a probabilistic generation of test data: structural or functional criteria serve as guides for defining an input profile and a test size. Previous work has confirmed the high fault revealing power of this approach for procedural programs; it is now investigated for object-oriented programs. A method for incremental statistical testing is defined at the cluster level, based on the class inheritance hierarchy. Starting from the root class of the program, descendant class(es) are gradually added and test data are designed for (i) structural testing of newly defined features and, (ii) regression testing of inherited features. The feasibility of the method is exemplified by a small case study (a Travel Agency) implemented in Eiffel.

Property-oriented testing: a strategy for exploring dangerous scenarios

Property-oriented testing uses the specification of a property to drive the testing process. The aim is to validate a program with respect to a target property, that is, to exercise the program and observe whether the property is violated or not. The paper defines a test strategy for safety properties in cyclic control systems. It consists of the stepwise construction of test scenarios. Each step explores possible continuations of the dangerous scenarios found at the previous step, using black-box sampling techniques. The feasibility of the strategy is illustrated on a stem boiler case study. The target property is the non explosion of the boiler in presence of faults in the physical devices. The experimental results are promising since four different explosive scenarios have been identified.

On Functional Statistical Testing Designed from Software Behavior Models

Statistical testing involves exercising a piece of software by supplying it with input values that are randomly selected according to a defined probability distribution over its input domain. This paper focuses on functional statistical testing, that is, when an input distribution and a number of random inputs are determined according to criteria relating to software functionality. The criteria based on models of behavior deduced from specification, i.e., finite-state machines and decision tables, are defined. The modeling approach involves a hierarchical decomposition of software functionality. It is applied to a module from the nuclear field. Functional statistical test sets are designed and applied to two versions of the module: the real version, and that developed by a student. Twelve residual faults are revealed, eleven of which affect the student’s version. The other fault is quite subtle, since it resides in the driver that we have developed for the real version in our experimental test harness. Two other input distributions are experimented with: the uniform distribution over the input domain and a structural distribution determined so as to rapidly exercise all the instructions of the student’s version. The results show that the functional statistical test sets have the highest fault revealing power and are the most cost-effective.

Show Me New Counterexamples: A Path-Based Approach

We consider lightweight usage of model-checking for the debugging of Simulink models. A problem is that model-checkers typically return only one counterexample, which may slow down the debugging process. We propose an approach and a tool to produce several counterexamples, exemplifying different property violation patterns for a given version of the design. The approach uses data collected during the replay of the counterexamples to synthesize queries for the model-checker, so that it finds counterexamples that activate new paths. The approach is applied to an academic example and an industrial model from the automotive domain.

Test Languages for In-the-Loop Avionics Tests

The current state of the art of in-the-loop testing of avionics embedded systems is very heterogeneous, with many different in-house test languages, and it is unable to respond to the evolving needs of the stakeholders in this field. To lay the foundations of more homogeneous test development solutions, this paper offers an overview and analysis of the features of six different test languages. Best practices and pitfalls to avoid are discussed. The analysis focuses on four major categories of features: test organization, abstraction of and access to the system under test interfaces, test language instructions, and time management. Four of the test languages in the sample set are currently employed in the field of avionics. The other two, used, respectively, in the automotive and telecommunications industries, have been chosen for comparison purposes. The paper reports the key findings of the analysis and presents the resulting ongoing research work.

STELAE — A model-driven test development environment for avionics systems

In this paper we present STELAE, a model-driven test development environment for avionics embedded systems, implemented on top of a real integration test platform. It is the result of an R&D project between two research laboratories and a test solution provider, aiming to introduce model-driven engineering methodologies and technologies for the development of tests. Our work was motivated by the multiplicity of proprietary test languages in this industrial context, which no longer respond to the stakeholder needs. We present the early prototype functionalities (test model definition, automatic code generation and execution) on a case study inspired from real-life. Our feedback on the used technologies concludes this paper.