Harald Brandl

Killing strategies for model-based mutation testing

This article presents the techniques and results of a novel model‐based test case generation approach that automatically derives test cases from UML state machines. The main contribution of this article is the fully automated fault‐based test case generation technique together with two empirical case studies derived from industrial use cases. Also, an in‐depth evaluation of different fault‐based test case generation strategies on each of the case studies is given and a comparison with plain random testing is conducted. The test case generation methodology supports a wide range of UML constructs and is grounded on the formal semantics of Backs action systems and the well‐known input–output conformance relation. Mutation operators are employed on the level of the specification to insert faults and generate test cases that will reveal the faults inserted. The effectiveness of this approach is shown and it is discussed how to gain a more expressive test suite by combining cheap but undirected random test case generation with the more expensive but directed mutation‐based technique. Finally, an extensive and critical discussion of the lessons learnt is given as well as a future outlook on the general usefulness and practicability of mutation‐based test case generation. Copyright

Automated Conformance Verification of Hybrid Systems

Due to the combination of discrete events and continuous behavior the validation of hybrid systems is a challenging task. Nevertheless, as for other systems the correctness of such hybrid systems is a major concern. In this paper we present a new approach for verifying the input-output conformance of two hybrid systems. This approach can be used to generate mutation-based test cases. We specify a hybrid system within the framework of Qualitative Action Systems. Here, besides conventional discrete actions, the continuous dynamics of hybrid systems is described with so called qualitative actions. This paper then shows how labeled transition systems can be used to describe the trace semantics of Qualitative Action Systems. The labeled transition systems are used to verify the conformance between two Qualitative Action Systems. Finally, we present first experimental results on a water tank system.

MoMut::UML Model-Based Mutation Testing for UML

Model-based mutation testing (MBMT) is a promising testing methodology that relies on a model of the system under test (SUT) to create test cases. Hence, MBMT is a so-called black-box testing approach. It also is fault based, as it creates test cases that are guaranteed to reveal certain faults: after inserting a fault into the model of the SUT, it looks for a test case revealing this fault. This turns MBMT into one of the most powerful and versatile test case generation approaches available as its tests are able to demonstrate the absence of certain faults, can achieve both, control-flow and data-flow coverage of model elements, and also may include information about the behaviour in the failure case. The latter becomes handy whenever the test execution framework is bound in the number of observations it can make and - as a consequence - has to restrict them. However, this versatility comes at a price: MBMT is computationally expensive. The tool MoMuT::UML (https://www.momut.org) is the result of a multi-year research effort to bring MBMT from the academic drawing board to industrial use. In this paper we present the current stable version, share the lessons learnt when applying two generations of MoMuT::UML in an industrial setting, and give an outlook on the upcoming, third,generation.

Efficient Mutation Killers in Action

This paper presents the techniques and results of a novel model-based test case generation approach that automatically derives test cases from UML state machines. Mutation testing is applied on the modeling level to generate test cases. We present the test case generation approach, discuss the tool chain, and present the properties of the generated test cases. The main contribution of this paper is an empirical study of a car alarm system where different strategies for killing mutants are compared. We present detailed figures on the effectiveness of the test case generation technique. Although UML serves as an input language, all techniques are grounded on solid foundations: we give UML state transition diagrams a formal semantics by mapping them to Backs action systems.

UML in action: a two-layered interpretation for testing

This paper presents a novel model-based test case generation approach that automatically derives test cases from UML state machines. UML is given a two-layered formal semantics by (1) mapping UML class diagrams and state charts to Backs Action Systems, (2) by interpreting these action systems as labeled transition systems. The first semantics provides a formal framework to capture the object-oriented machinery: classes, objects, inheritance, transitions, time-outs, signals, nested and parallel regions. The second mapping represents the testers view on the interface in terms of input and output actions. Tretmans input-output conformance relation (ioco) forms the basis of our fault models. Mutation analysis on the models is used to generate test cases. A car alarm system serves as a running example

Model-based mutation testing of hybrid systems

This paper presents a novel model-based testing approach developed in the MOGENTES project. The aim is to test embedded systems controlling a continuous environment, i.e., hybrid systems. We present our two key abstractions against which we systematically test for conformance. (1) Classical action systems are used to model the discrete controller behavior. (2) Qualitative differential equations are used to model the evolutions of the environment. The latter is based on a technique from the domain of Artificial Intelligence called qualitative reasoning. Mutation testing on these models is used to generate effective test cases. A test case generator has been developed that searches for all test cases that would kill a mutant. The mutant models represent our fault models. The generated test cases are then executed on the implementation in order to systematically exclude the possibility that a mutant has been implemented.

Conformance Testing of Hybrid Systems with Qualitative Reasoning Models

Embedded systems are of growing importance in industry. For example, in a todays vehicle a huge number of embedded and communicating systems can be found. Exhaustive testing of such systems is a requirement, because changes after delivery and use are expensive and sometimes even impossible. In this paper we propose the use of qualitative models, which are an abstraction of quantitative physical models, for test case generation and test execution. In particular, we show how Simulink models from which control programs are automatically extracted can be tested with respect to qualitative models. Since Simulink models are heavily used in industry, the approach is of practical interest.

Qualitative Action Systems

An extension to action systems is presented facilitating the modeling of continuous behavior in the discrete domain. The original action system formalism has been developed by Back et al. in order to describe parallel and distributed computations of discrete systems, i.e. systems with discrete state space and discrete control. In order to cope with hybrid systems, i.e. systems with continuous evolution and discrete control, two extensions have been proposed: hybrid action systems and continuous action systems . Both use differential equations (relations) to describe continuous evolution. Our version of action systems takes an alternative approach by adding a level of abstraction: continuous behavior is modeled by Qualitative Differential Equations that are the preferred choice when it comes to specifying abstract and possibly non-deterministic requirements of continuous behavior. Because their solutions are transition systems, all evolutions in our qualitative action systems are discrete. Based on hybrid action systems, we develop a new theory of qualitative action systems and discuss how we have applied such models in the context of automated test-case generation for hybrid systems.

Test Case Generation from QR Models

In this paper we present the application of AI to automated test case generation. In particular we introduce a technique for deriving abstract test cases from qualitative models. In the case of reactive systems, the behavior of the device depends on its internal state and the perception of the environment via its sensors. Such systems are well suited for modeling with using Qualitative Reasoning methods. Qualitative Reasoning enables one to specify system behavior that also acquires changing environmental conditions and hence provides a good foundation to derive more realistic test cases. In the first part of this paper we give a short introduction to Qualitative Reasoning and Garp3, the tool we use for model creation and simulation. We also present a method for modeling differential equations within Garp3. In the second part we deal with abstract test case generation from QR models and present first results obtained.

Qr-model based testing

As reactive and embedded systems continuously interact with their environment, it is important to test as many as possible interactions. The use of qualitative models of the environment and hardware has the potential to provide test cases that might not be considered with traditional testing methods. We present an approach that derives abstract test cases from such models using qualitative reasoning, which is a well known artificial intelligence technique to represent and reason about physical behavior. For this purpose we introduce the underlying concepts of qualitative reasoning, show the test case generation process, and provide the results of a case study.