Bernhard K. Aichernig

Test case generation by OCL mutation and constraint solving

Fault-based testing is a technique where testers anticipate errors in a system under test in order to assess or generate test cases. The idea is to have enough test cases capable of detecting these anticipated errors. This paper presents a method of fault-based test case generation for pre- and postcondition specifications. Here, errors are anticipated on the specification level by mutating the pre- and postconditions. We present the underlying theory by giving test cases a formal semantics and translate this general testing theory to a constraint satisfaction problem. A prototype test case generator serves to demonstrate the automatization of the method. The current tool works with OCL specifications, but the theory and method are general and apply to many state-based specification languages.

Mutation Testing in the Refinement Calculus

This article discusses mutation testing strategies in the context of refinement. Here, a novel generalisation of mutation testing techniques is presented to be applied to contracts ranging from formal specifications to programs. It is demonstrated that refinement and its dual abstraction are the key notions leading to a precise and yet simple theory of mutation testing. The refinement calculus of Back and von Wright is used to express concepts like contracts, useful mutations, test cases and test coverage.

Killing strategies for model-based mutation testing

This article presents the techniques and results of a novel model‐based test case generation approach that automatically derives test cases from UML state machines. The main contribution of this article is the fully automated fault‐based test case generation technique together with two empirical case studies derived from industrial use cases. Also, an in‐depth evaluation of different fault‐based test case generation strategies on each of the case studies is given and a comparison with plain random testing is conducted. The test case generation methodology supports a wide range of UML constructs and is grounded on the formal semantics of Backs action systems and the well‐known input–output conformance relation. Mutation operators are employed on the level of the specification to insert faults and generate test cases that will reveal the faults inserted. The effectiveness of this approach is shown and it is discussed how to gain a more expressive test suite by combining cheap but undirected random test case generation with the more expensive but directed mutation‐based technique. Finally, an extensive and critical discussion of the lessons learnt is given as well as a future outlook on the general usefulness and practicability of mutation‐based test case generation. Copyright

From faults via test purposes to test cases: on the fault-based testing of concurrent systems

Fault-based testing is a technique where testers anticipate errors in a system under test in order to assess or generate test cases. The idea is to have enough test cases capable of detecting these anticipated errors. This paper presents a theory and technique for generating fault-based test cases for concurrent systems. The novel idea is to generate test purposes from faults that have been injected into a model of the system under test. Such test purposes form a specification of a more detailed test case that can detect the injected fault. The theory is based on the notion of refinement. The technique is automated using the TGV test case generator and an equivalence checker of the CADP tools. A case study of testing web servers demonstrates the practicability of the approach.

Protocol Conformance Testing a SIP Registrar: an Industrial Application of Formal Methods

Various research prototypes and a well-founded theory of model based testing (MBT) suggests the application of MBT to real-world problems. In this article we report on applying the well-known TGV tool for protocol conformance testing of a Session Initiation Protocol (SIP) server. Particularly, we discuss the performed abstractions along with corresponding rationales. Furthermore, we show how to use structural and fault-based techniques for test purpose design. We present first empirical results obtained from applying our test cases to a commercial implementation and to a popular open source implementation of a SIP Registrar. Notably, in both implementations our input output labeled transition system model proved successful in revealing severe violations of the protocol.

Automated Conformance Verification of Hybrid Systems

Due to the combination of discrete events and continuous behavior the validation of hybrid systems is a challenging task. Nevertheless, as for other systems the correctness of such hybrid systems is a major concern. In this paper we present a new approach for verifying the input-output conformance of two hybrid systems. This approach can be used to generate mutation-based test cases. We specify a hybrid system within the framework of Qualitative Action Systems. Here, besides conventional discrete actions, the continuous dynamics of hybrid systems is described with so called qualitative actions. This paper then shows how labeled transition systems can be used to describe the trace semantics of Qualitative Action Systems. The labeled transition systems are used to verify the conformance between two Qualitative Action Systems. Finally, we present first experimental results on a water tank system.

Mutation testing in UTP

This paper presents a theory of testing that integrates into Hoare and He’s Unifying Theory of Programming (UTP). We give test cases a denotational semantics by viewing them as specification predicates. This reformulation of test cases allows for relating test cases via refinement to specifications and programs. Having such a refinement order that integrates test cases, we develop a testing theory for fault-based testing.Fault-based testing uses test data designed to demonstrate the absence of a set of pre-specified faults. A well-known fault-based technique is mutation testing. In mutation testing, first, faults are injected into a program by altering (mutating) its source code. Then, test cases that can detect these errors are designed. The assumption is that other faults will be caught, too. In this paper, we apply the mutation technique to both, specifications and programs.Using our theory of testing, two new test case generation laws for detecting injected (anticipated) faults are presented: one is based on the semantic level of UTP design predicates, the other on the algebraic properties of a small programming language.

A Proof Obligation Generator for VDM-SL

In this paper an extension of the IFAD VDM-SL Toolbox with a proof obligation generator is described. Static type checking in VDM is undecidable in general and therefore the type checker must be incomplete. Hence, for the “difficult” parts introducing undecidability, it is up to the user to verify the consistency of a specification. Instead of providing error messages and warnings, the approach of generating proof obligations for the consistency of VDM-SL specifications is taken. The overall goal of this work is to automate the generation of proof obligations for VDM-SL. Proof obligation generation has already been carried out for a number of related notations, but VDM-SL contains a number of challenging constructs (e.g. patterns, non-disjoint union types, and operations) for which new research is presented in this paper.

Validating voice communication requirements using lightweight formal methods

To show that lightweight approaches can facilitate the technological transfer of formal development methods, the authors report on their experience using VDM++ to specify a safety-critical air traffic control voice communication system. Their approach raised both the quality of the informal system specification and the efficiency of the system test suites they used.

Automated Black-Box Testing with Abstract VDM Oracles

In this paper the possibilities to automate black-box testing through formal requirement specifications are explored. More precisely, the formal method VDM (Vienna Development Method) serves to demonstrate that abstract requirement models can be used as test oracles for concrete software. The automation of the resulting testing frame-work is based on modern CASE-tools that support a light-weight approach to formal methods. The specification language used is VDMSL, but the results are easily transferred into similar model oriented methods such as B, Z or RAISE.