Bruno Marre

Software testing based on formal specifications: a theory and a tool

This paper addresses the problem of constructing test data sets from formal specifications. Starting from a notion of an ideal exhaustive test data set which is derived from the notion of satisfaction of the formal specification, it is shown how to select by refinements a practicable test set, i.e. computable, not rejecting correct programs (unbiased), and accepting only correct programs (valid), assuming some hypotheses. The hypotheses play an important role: they formalize common test practices and they express the gap between the success of the test and correctness ; the size of the test set depends on the strength of the hypotheses. The paper shows an application of this theory in the case of algebraic specifications and presents the actual procedures used to mechanically produce such test sets, using Horn clause logic. These procedures are embedded in an interactive system which, given some general hypotheses schemes and an algebraic specification, produces a test set and the corresponding hypotheses.

PathCrawler: automatic generation of path tests by combining static and dynamic analysis

We present the PathCrawler prototype tool for the automatic generation of test-cases satisfying the rigorous all-paths criterion, with a user-defined limit on the number of loop iterations in the covered paths. The prototype treats C code and we illustrate the test-case generation process on a representative example of a C function containing data-structures of variable dimensions, loops with variable numbers of iterations and many infeasible paths. PathCrawler is based on a novel combination of code instrumentation and constraint solving which makes it both efficient and open to extension. It suffers neither from the approximations and complexity of static analysis, nor from the number of executions demanded by the use of heuristic algorithms in function minimisation and the possibility that they fail to find a solution. We believe that it demonstrates the feasibility of rigorous and systematic testing of sequential programs coded in imperative languages.

Test sequences generation from LUSTRE descriptions: GATEL

We describe a test sequence generation method from LUSTRE descriptions and its companion tool, GATEL. The LUSTRE language is declarative and describes synchronous data-flow computations. It is used for reactive control/command systems, mainly for electrical power production applications. Such critical applications require a high level of reliability. While this language benefits from powerful verification tools, there is still a demand for adequate testing techniques. The method and the tool described can be applied during unit and integration testing, according to a structural (glass box) or functional (black box) test selection strategy. The test generation tool uses some interpretation of the language constructs as boolean and integer interval constraints. Test sequence generation is automated using constraint logic programming techniques. The method and the tool are illustrated on an example extracted from an industrial case study.

Using algebraic specifications in software testing: a case study on the software of an automatic subway

Abstract This article reports an experiment using a formal specification of a piece of critical software as a basis to derive test data sets. The software was the onboard part of the driving system of an automatic subway. It was formally specified using algebraic techniques. Then, two modules of the specification were used as input to the LOFT test selection tool, which is based on narrowing and some other aspects of advanced logic programming. The experience established that the approach is practicable for industrial examples.

On-the-fly generation of k-path tests for C functions

We propose a novel method, called PathCrawler, for the automatic generation of structural tests satisfying the all-paths criterion or its k-path variant. The source code is instrumented so as to recover the symbolic execution path each time that the program under test is executed. This code is first executed using inputs arbitrarily selected from the input domain. The resulting symbolic path is transformed into a path predicate by projection of the conditions onto the input variables. The next test is obtained by using constraint logic programming to find new input values outside the domain of the path which is already covered. The instrumented code is then executed on this test and so on, until all feasible paths have been covered. Our method combines static and dynamic analysis in a way that avoids the disadvantages of both. It is currently being implemented for the C language.

Generation of All-Paths Unit Test with Function Calls

Structural testing is usually restricted to unit tests and based on some clear definition of source code coverage. In particular, the all-paths criterion, which requires at least one test-case per feasible path of the function under test, is recognised as offering a high level of software reliability. This paper deals with the difficulties of using structural unit testing to test functions which call other functions. To limit the resulting combinatorial explosion in the number of paths, we choose to abstract the called functions by their specification. We incorporate the functional information on the called functions within the structural information on the function under test, given as a control flow graph (CFG). This representation combining functional and structural descriptions may be viewed as an extension of the classic CFG and allows us to characterise test selection criteria ensuring the coverage of the source code of the function under test. Two new criteria will be proposed. The first criterion corresponds to the coverage of all the paths of this new representation, including all the paths arising from the functional description of the called functions. The second criterion covers all the feasible paths of the function under test only. We describe how we automate test-data generation with respect to such grey-box (combinations of black- box and white-box) test selection strategies, and we apply the resulting extension of our PathCrawler tool to examples coded in the C language.

Testing from algebraic specifications: test data set selection by unfolding axioms

This paper deals with test data set selection from algebraic specifications. Test data sets are generated from selection criteria which are usually defined to cover specification axioms. The unfolding selection criterion consists in covering the input domain of an operation using case analysis. The unfolding procedure can be iterated in order to split input domains of operations into finer subdomains. In this paper we propose to extend an unfolding procedure previously developed in [5, 19] that could only be performed on very low level, i.e. executable specifications. On the contrary, our new unfolding procedure can be applied to any positive conditional specification. We show that our unfolding procedure is sound (no test is added) and complete (no test is lost) with respect to the starting reference test data set.

Behind the scenes in SANTE: a combination of static and dynamic analyses

While the development of one software verification tool is often seen as a difficult task, the realization of a tool combining various verification techniques is even more complex. This paper presents an innovative tool for verification of C programs called Sante (Static ANalysis and TEsting). We show how several tools based on heterogeneous techniques such as abstract interpretation, dependency analysis, program slicing, constraint solving and test generation can be combined within one tool. We describe the integration of these tools and discuss particular aspects of each underlying tool that are beneficial for the whole combination.

Dynamic Testing from Bounded Data Type Specifications

Due to practical limitations in software and hardware, data type implementations are always bounded. Such limitations are a frequent source of faults which are difficult to find out. As soon as boundaries are clearly identified in a specification, functional testing should be able to address any boundary fault.

Handling State-Machines Specifications with GATeL

GATeL proposes a testing environment for Lustre/Scade programs. Its main component is a resolution procedure based on a CLP interpretation of its input language. This paper presents a two-tier extension of GATeL in order to take into account state-machines descriptions. This extension relies on a compilation of these constructs into multi-clocked expressions. Our first contribution is a definition of explicit constraints to manage the clock type hierarchy in GATeL. The second one is the definition of constraints reflecting properties of state-machines built by the Scade compilation schema.