Jan Tretmans

Conformance testing with labelled transition systems: implementation relations and test generation

This paper studies testing based on labelled transition systems, presenting two test generation algorithms with their corresponding implementation relations. The first algorithm assumes that implementations communicate with their environment via symmetric, synchronous interactions. It is based on the theory of testing equivalence and preorder, as is most of the testing theory for labelled transition systems, and it is found in the literature in some slightly different variations. The second algorithm is based on the assumption that implementations communicate with their environment via inputs and outputs. Such implementations are formalized by restricting the class of labelled transition systems to those systems that can always accept input actions. For these implementations a testing theory is developed, analogous to the theory of testing equivalence and preorder. It consists of implementation relations formalizing the notion of conformance of these implementations with respect to labelled transition system specifications, test cases and test suites, test execution, the notion of passing a test suite, and the test generation algorithm, which is proved to produce sound test suites for one of the implementation relations.

Test generation based on symbolic specifications

Classical state-oriented testing approaches are based on simple machine models such as Labelled Transition Systems (LTSs), in which data is represented by concrete values. To implement these theories, data types which have infinite universes have to be cut down to finite variants, which are subsequently enumerated to fit in the model. This leads to an explosion of the state space. Moreover, exploiting the syntactical and/or semantical information of the involved data types is non-trivial after enumeration. To overcome these problems, we lift the family of testing relations ioco

Test Generation with Inputs, Outputs, and Quiescence

_\mathcal{F}

On-the-fly conformance testing using Spin

to the level of Symbolic Transition Systems (STSs). We present an algorithm based on STSs, which generates and executes tests on-the-fly on a given system. It is sound and complete for the ioco

Testing Concurrent Systems: A Formal Approach

_\mathcal{F}

Testing of Software and Communicating Systems

testing relations.

Audition of web services for testing conformance to open specified protocols

This paper studies testing based on labelled transition systems, using the assumption that implementations communicate with their environment via inputs and outputs. Such implementations are formalized by restricting the class of transition systems to those systems that can always accept input actions, as in input/output automata. Implementation relations, formalizing the notion of conformance of these implementations with respect to labelled transition system specifications, are defined analogous to the theory of testing equivalence and preorder. A test generation algorithm is given, which is proved to produce a sound and exhaustive test suite from a specification, i.e., a test suite that fully characterizes the set of correct implementations.

On Conformance Testing for Timed Systems

Abstract.In this paper we report on the construction of a tool for conformance testing based on Spin. The Spin tool has been adapted such that it can derive the building blocks for constructing test cases, called test primitives, from systems described in Promela. The test primitives support the on-the-fly conformance testing process. Traditional derivation of tests from formal specifications suffers from the state-space explosion problem. Spin is one of the most advanced model checkers with respect to handling large state spaces. This advantage of Spin has been used for the derivation of test primitives from a Promela description. To reduce the state space, we introduce the on-the-fly testing framework. One of the components within this framework is the Primer. The Primer is responsible for deriving test primitives from a model of a system according to a well-defined and complete testing theory. Algorithms are presented which enable us to derive test primitives from a Promela description. These algorithms have been implemented in the adapted version of the Spin tool which acts as the Primer in the framework. Promising experiments have been carried out on an example case study. As a result of this study it is concluded that it is possible to derive test primitives automatically from Promela descriptions, construct test cases from these test primitives, and execute the test cases on-the-fly.

Test Selection, Trace Distance and Heuristics

This paper discusses the use of formal methods in testing of concurrent systems. It is argued that formal methods and testing can be mutually profitable and useful. A framework for testing based on formal specifications is presented. This framework is elaborated for labelled transition systems, providing formal definitions of conformance, test execution and test derivation. A test derivation algorithm is given and its tool implementation is briefly discussed.

Software Engineering with Formal Methods: The Development of a Storm Surge Barrier Control System Revisiting Seven Myths of Formal Methods

Conformiq Qtronic is a commercial tool for model driven testing. It derives tests automatically from behavioral system models. These are black-box tests [1] by nature, which means that they depend on the model and the interfaces of the system under test, but not on the internal structure (e.g. source code) of the implementation. In this essay, which accompanies my invited talk, I survey the nature of Conformiq Qtronic, the main implementation challenges that we have encountered and how we have approached them.