Elisabeth Jöbstl

Killing strategies for model-based mutation testing

This article presents the techniques and results of a novel model‐based test case generation approach that automatically derives test cases from UML state machines. The main contribution of this article is the fully automated fault‐based test case generation technique together with two empirical case studies derived from industrial use cases. Also, an in‐depth evaluation of different fault‐based test case generation strategies on each of the case studies is given and a comparison with plain random testing is conducted. The test case generation methodology supports a wide range of UML constructs and is grounded on the formal semantics of Backs action systems and the well‐known input–output conformance relation. Mutation operators are employed on the level of the specification to insert faults and generate test cases that will reveal the faults inserted. The effectiveness of this approach is shown and it is discussed how to gain a more expressive test suite by combining cheap but undirected random test case generation with the more expensive but directed mutation‐based technique. Finally, an extensive and critical discussion of the lessons learnt is given as well as a future outlook on the general usefulness and practicability of mutation‐based test case generation. Copyright

MoMut::UML Model-Based Mutation Testing for UML

Model-based mutation testing (MBMT) is a promising testing methodology that relies on a model of the system under test (SUT) to create test cases. Hence, MBMT is a so-called black-box testing approach. It also is fault based, as it creates test cases that are guaranteed to reveal certain faults: after inserting a fault into the model of the SUT, it looks for a test case revealing this fault. This turns MBMT into one of the most powerful and versatile test case generation approaches available as its tests are able to demonstrate the absence of certain faults, can achieve both, control-flow and data-flow coverage of model elements, and also may include information about the behaviour in the failure case. The latter becomes handy whenever the test execution framework is bound in the number of observations it can make and - as a consequence - has to restrict them. However, this versatility comes at a price: MBMT is computationally expensive. The tool MoMuT::UML (https://www.momut.org) is the result of a multi-year research effort to bring MBMT from the academic drawing board to industrial use. In this paper we present the current stable version, share the lessons learnt when applying two generations of MoMuT::UML in an industrial setting, and give an outlook on the upcoming, third,generation.

Efficient Mutation Killers in Action

This paper presents the techniques and results of a novel model-based test case generation approach that automatically derives test cases from UML state machines. Mutation testing is applied on the modeling level to generate test cases. We present the test case generation approach, discuss the tool chain, and present the properties of the generated test cases. The main contribution of this paper is an empirical study of a car alarm system where different strategies for killing mutants are compared. We present detailed figures on the effectiveness of the test case generation technique. Although UML serves as an input language, all techniques are grounded on solid foundations: we give UML state transition diagrams a formal semantics by mapping them to Backs action systems.

UML in action: a two-layered interpretation for testing

This paper presents a novel model-based test case generation approach that automatically derives test cases from UML state machines. UML is given a two-layered formal semantics by (1) mapping UML class diagrams and state charts to Backs Action Systems, (2) by interpreting these action systems as labeled transition systems. The first semantics provides a formal framework to capture the object-oriented machinery: classes, objects, inheritance, transitions, time-outs, signals, nested and parallel regions. The second mapping represents the testers view on the interface in terms of input and output actions. Tretmans input-output conformance relation (ioco) forms the basis of our fault models. Mutation analysis on the models is used to generate test cases. A car alarm system serves as a running example

Model-based mutation testing of hybrid systems

This paper presents a novel model-based testing approach developed in the MOGENTES project. The aim is to test embedded systems controlling a continuous environment, i.e., hybrid systems. We present our two key abstractions against which we systematically test for conformance. (1) Classical action systems are used to model the discrete controller behavior. (2) Qualitative differential equations are used to model the evolutions of the environment. The latter is based on a technique from the domain of Artificial Intelligence called qualitative reasoning. Mutation testing on these models is used to generate effective test cases. A test case generator has been developed that searches for all test cases that would kill a mutant. The mutant models represent our fault models. The generated test cases are then executed on the implementation in order to systematically exclude the possibility that a mutant has been implemented.

Model-based mutation testing via symbolic refinement checking

In model-based mutation testing, a test model is mutated for test case generation. The resulting test cases are able to detect whether the faults in the mutated models have been implemented in the system under test. For this purpose, a conformance check between the original and the mutated model is required. The generated counterexamples serve as basis for the test cases. Unfortunately, conformance checking is a hard problem and requires sophisticated verification techniques. Previous attempts using an explicit conformance checker suffered state space explosion. In this paper, we present several optimisations of a symbolic conformance checker using constraint solving techniques. The tool efficiently checks the refinement between non-deterministic test models. Compared to previous implementations, we could reduce our runtimes by 97%. In a new industrial case study, our optimisations can reduce the runtime from over 6 hours to less than 3 minutes. We deal with model- and mutation-based test case generation.The main focus lies on optimisations of the underlying conformance check.We explain the construction of test cases based on the conformance check.We allow for non-determinism in the test models.We demonstrate the effectiveness of our optimisations on industrial case studies.

Model-Based Mutation Testing of an Industrial Measurement Device

MoMuT::UML is a model-based mutation testing tool for UML models. It maps UML state machines to a formal semantics and performs a conformance check between an original and a set of mutated models to automatically generate test cases. The resulting test suite is able to detect whether a system under test implements one of the faulty models instead of the correct, original model. In this work, we illustrate the whole model-based mutation testing process by means of an industrial case study. We test the control logic of a device that counts the particles in exhaust gases. First, we model the system under test in UML. Then, MoMuT::UML is used to automatically generate three test suites from the UML test model: one mutation-based test suite, one set of random test cases, and a third test suite combining random and mutation-based test case generation. The test cases are executed on the system under test and effectively reveal several errors. Finally, we compare the fault detection capabilities of the three test suites on a set of faulty systems, which were created by intentionally injecting faults into the implementation.

Incremental Refinement Checking for Test Case Generation

We combine model-based testing and mutation testing to automatically generate a test suite that achieves a high mutation adequacy score. The original model representing the system under test is mutated. To generate test cases that detect whether a modelled fault has been implemented, we perform a refinement check between the original and the mutated models. Action systems serve as formal models. They are well-suited to model reactive systems and allow non-determinism. We extend our previous work by two techniques to improve efficiency: (1) a strategy to efficiently handle a large number of mutants and (2) incremental solving. A case study illustrates the potential of our improvements. The runtime for checking appr. 200 mutants could be reduced from 20s to 3s. We implemented our algorithms in two versions: one uses a constraint solver, the other one an SMT solver. Both show similar performance.

Fault-based generation of test cases from UML-Models: approach and some experiences

In principle, automated test case generation - both from source code and models - is a fairly evolved technology, which is on the way to common use in industrial testing and quality assessment of safety-related, softwareintensive systems. However, common coverage measures such as branch or MC/DC1 for source code and states or transitions for state-based models provide only very limited information about the covered (implementation) faults. Fault-based test case generation tries to improve this situation by looking for detecting faults explicitly. This paper describes an approach combining fault- and model-based testing which has been realized in the European project MOGENTES2, using UML state machines for representing requirements, and discusses results of its application to a use case from the automotive domain.

When BDDs Fail: Conformance Testing with Symbolic Execution and SMT Solving

Model-based testing is a well known technique that allows one to validate the correctness of software with respect to its model. If a lot of data is involved, symbolic techniques usually outperform explicit data enumeration. In this paper, we focus on a new symbolic test case generation technique. Our approach is based on symbolic execution and on satisfiability (modulo theory; SMT) solving. Our work was motivated by the complete failure of a well-known existing symbolic test case generator to produce any test cases for an industrial Session Initiation Protocol (SIP) implementation. Hence, we have replaced the BDD-based analysis of the existing tool with a combination of symbolic execution and SMT solving. Our new tool generates the test cases for SIP in seconds. However, further experiments showed that our approach is not a substitutive but a complementary approach: we present the technique and the results obtained for two protocol specifications, the first supporting our new technique, the second being witness for the classic BDD-technique.