Fabien Peureux

Automated Boundary Testing from Z and B

We present a method for black-box boundary testing from B and Z formal specifications. The basis of the method is to test every operation of the system at every boundary state using all input boundary values of that operation. The test generation process is highly automated. It starts by calculating boundary goals from Pre/Post predicates derived from the formal model. Then each boundary goal is instantiated to a reachable boundary state, by searching for a sequence of operations that reaches the boundary goal from the initial state. This process makes intensive use of a set-oriented constraint technology, both for boundary computation and to traverse the state space. The method was designed on the basis of industrial applications in the domain of critical software (Smart card and transportation). Application results show the effectiveness and the scalability of the method. In this paper, we give an overview of the method and focus on the calculation of the boundary goals and states.

Generation of test sequences from formal specifications: GSM 11-11 standard case study

This paper presents the results of a case study on generating test cases for a fragment of the smart card GSM 11‐11 standard. The generation method is based on an original approach using the B notation and techniques of constraint logic programming with sets. The GSM 11‐11 technical specifications were formalized with the B notation. From this B specification, a system of constraints was derived, equivalent to this formal model. Using a set constraint solver, boundary states were computed and test cases were obtained by traversing the constrained reachability graph of the specifications. The purpose of this project was to evaluate the contribution of this testing environment, called B‐TESTING‐TOOLS, in an industrial process on a real life‐size application, by comparing the generated test sequences with the already used and high‐quality manually‐designed tests. This comparison enabled us to validate our approach and showed its effectiveness in the validation process of critical applications: the case study gives a wide coverage (about 85%) of the generated tests compared to the pre‐existing tests and a saving of 30% in test design time. Copyright

A subset of precise UML for model-based testing

This paper presents an original model-based testing approach that takes a UML behavioural view of the system under test and automatically generates test cases and executable test scripts according to model coverage criteria. This approach is embedded in the LEIRIOS Test Designer tool and is currently deployed in domains such as Enterprise IT and electronic transaction applications. This model-based testing approach makes it possible to automatically produce the traceability matrix from requirements to test cases as part of the test generation process. This paper defines the subset of UML used for model-based testing and illustrates it using a small example.

Boundary coverage criteria for test generation from formal models

This paper proposes a new family of model-based coverage criteria, based on formalizing boundary-value testing heuristics. The new criteria form a hierarchy of data-oriented coverage criteria, and can be applied to any formal notation that uses variables and values. They can be used either to measure the coverage of an existing test set, or to generate tests from a formal model. We give algorithms that can be used to generate tests that satisfy the criteria. These algorithms and criteria have been incorporated into the BZ-TESTING-TOOLS (BZ-TT) tool-set for automated test case generation from B, Z and UML/OCL specifications, and have been used and validated on several industrial applications in the domain of critical software, particularly smart cards and transport systems.

CLPS–B – A constraint solver to animate a B specification

This paper proposes an approach to evaluating B formal specifications using constraint logic programming with sets (CLPS). This approach is used to animate and generate test sequences from B formal specifications. The solver, called CLPS–B, is described in terms of constraint domains, consistency verification, and constraint propagation. It is more powerful than most constraint systems because it allows the domain of variable to contain other variables, which increases the level of abstraction. The constrained state propagates the nondeterminism of the B specifications and reduces the number of states in a reachability graph. We illustrate this approach by comparing the constrained state graph exploration with the concrete one in a simple example – process scheduler. We also describe the automated test generation method that uses the CLPS–B solver to better control combinational explosion.

Requirements traceability in automated test generation: application to smart card software validation

Automated test case and test driver generation from a formal model is becoming a more widely used practice in the smart card area. This innovative approach for validation testing makes it possible to ensure the functional coverage of the test suite and to automate the production of executable test scripts. This paper presents an approach to automatically produce the Traceability Matrix from requirements to test cases, as part of the test generation process. This approach is embedded in the LEIRIOS Test Generator (LTG) tool, and has been used for several real-life applications in the smart card industry. This paper introduces the approach to annotating the formal model and using it to generate the Traceability Matrix. It also discusses some lessons learnt from our experience using Model-Based Testing for smart card software validation.

CLPS-B - A Constraint Solver for B

This paper proposes an approach to the evaluation of B formal specifications using Constraint Logic Programming with sets. This approach is used to animate and generate test sequences from B formal specifications. The solver, called CLPS-B, is described in terms of constraint domains, consistency verification and constraint propagation. It is more powerful than most constraint systems, because it allows the domain of variable to contain other variables, which increase the level of abstraction. The constrained state propagates the non-determinism of the B specifications and reduces the number of states in a reachability graph. We illustrate this approach by comparing the constrained states graph exploration with the concrete one in a simple example: Process scheduler.

Generation of functional test sequences from B formal specifications presentation and industrial case-study

The paper presents an original method to generate test sequences. From formal specifications of the system to be tested, an equivalent system of constraints is derived, and then the domain of each state variable of this system is partitioned into subdomains. Using this partition, limit states are computed with a specific solver that uses constraint logic programming with sets. This specific solver is then used to build test sequences by traversing the constrained reachability graph of the specifications. Finally, the formal specifications are used as an oracle by using them to determine the expected output for a given input. The results of an industrial case-study of the Smart Card GSM 11-11 standard are presented and discussed.

Controlling test case explosion in test generation from B formal models

BZ‐TESTING‐TOOLS (BZ‐TT) is a tool set for automated test case generation from B and Z specifications. BZ‐TT uses boundary and cause–effect testing on the basis of the formal model. It has been used and validated on several industrial applications in the domain of critical software, particularly smart card and transport systems. This paper presents the test coverage criteria supported by BZ‐TT. On the one hand, these correspond to various classical structural coverage criteria, but specialized to the case of B abstract machines. The paper gives algorithms for these in Prolog. On the other hand, BZ‐TT introduces new coverage criteria for complex data structures, based on boundary analysis: this paper defines weak and strong state‐boundary coverage, input‐boundary coverage and output‐boundary coverage. Finally, the paper describes how BZ‐TT presents a unified view of these criteria to the validation engineer, and allows him or her to control the test case explosion on a coarse basis (choosing from a range of coverage criteria) as well as a fine basis (selecting options for each state or input variable). Copyright

Preamble computation in automated test case generation using constraint logic programming

BZ‐Testing‐Tools (BZ‐TT) is a tool‐set for automated model‐based test case generation from B abstract machines and Z specifications. BZ‐TT uses boundary testing as well as cause–effect testing on the basis of the formal model. It has been used and validated on several industrial case studies in the domain of critical software: in particular for smart card applications and automotive embedded systems. The main idea of BZ‐TT is to compute a boundary goal for each effect of the operations of the model and then to compute a preamble sequence of operations to place the system under test in such a state that satisfies the goal.