Rupert Schlick

Killing strategies for model-based mutation testing

This article presents the techniques and results of a novel model‐based test case generation approach that automatically derives test cases from UML state machines. The main contribution of this article is the fully automated fault‐based test case generation technique together with two empirical case studies derived from industrial use cases. Also, an in‐depth evaluation of different fault‐based test case generation strategies on each of the case studies is given and a comparison with plain random testing is conducted. The test case generation methodology supports a wide range of UML constructs and is grounded on the formal semantics of Backs action systems and the well‐known input–output conformance relation. Mutation operators are employed on the level of the specification to insert faults and generate test cases that will reveal the faults inserted. The effectiveness of this approach is shown and it is discussed how to gain a more expressive test suite by combining cheap but undirected random test case generation with the more expensive but directed mutation‐based technique. Finally, an extensive and critical discussion of the lessons learnt is given as well as a future outlook on the general usefulness and practicability of mutation‐based test case generation. Copyright

MoMut::UML Model-Based Mutation Testing for UML

Model-based mutation testing (MBMT) is a promising testing methodology that relies on a model of the system under test (SUT) to create test cases. Hence, MBMT is a so-called black-box testing approach. It also is fault based, as it creates test cases that are guaranteed to reveal certain faults: after inserting a fault into the model of the SUT, it looks for a test case revealing this fault. This turns MBMT into one of the most powerful and versatile test case generation approaches available as its tests are able to demonstrate the absence of certain faults, can achieve both, control-flow and data-flow coverage of model elements, and also may include information about the behaviour in the failure case. The latter becomes handy whenever the test execution framework is bound in the number of observations it can make and - as a consequence - has to restrict them. However, this versatility comes at a price: MBMT is computationally expensive. The tool MoMuT::UML (https://www.momut.org) is the result of a multi-year research effort to bring MBMT from the academic drawing board to industrial use. In this paper we present the current stable version, share the lessons learnt when applying two generations of MoMuT::UML in an industrial setting, and give an outlook on the upcoming, third,generation.

Model-Based Mutation Testing of an Industrial Measurement Device

MoMuT::UML is a model-based mutation testing tool for UML models. It maps UML state machines to a formal semantics and performs a conformance check between an original and a set of mutated models to automatically generate test cases. The resulting test suite is able to detect whether a system under test implements one of the faulty models instead of the correct, original model. In this work, we illustrate the whole model-based mutation testing process by means of an industrial case study. We test the control logic of a device that counts the particles in exhaust gases. First, we model the system under test in UML. Then, MoMuT::UML is used to automatically generate three test suites from the UML test model: one mutation-based test suite, one set of random test cases, and a third test suite combining random and mutation-based test case generation. The test cases are executed on the system under test and effectively reveal several errors. Finally, we compare the fault detection capabilities of the three test suites on a set of faulty systems, which were created by intentionally injecting faults into the implementation.

Fault-based generation of test cases from UML-Models: approach and some experiences

In principle, automated test case generation - both from source code and models - is a fairly evolved technology, which is on the way to common use in industrial testing and quality assessment of safety-related, softwareintensive systems. However, common coverage measures such as branch or MC/DC1 for source code and states or transitions for state-based models provide only very limited information about the covered (implementation) faults. Fault-based test case generation tries to improve this situation by looking for detecting faults explicitly. This paper describes an approach combining fault- and model-based testing which has been realized in the European project MOGENTES2, using UML state machines for representing requirements, and discusses results of its application to a use case from the automotive domain.

Integration of Requirements Engineering and Test-Case Generation via OSLC

We present a requirement-centered analysis and testing framework that integrates methods and tools for capturing and formalizing textual customer requirements, analyzing requirements consistency, generating test cases from formalized requirements and executing them on the implementation model. The framework preserves a fine grained traceability of informal and formal requirements, test cases and implementation models throughout every step of the workflow. We instantiate the framework with concrete tools that we integrate via a file repository and Open Services for Lifecycle Collaboration (OSLC). The standardized integration ensures that the framework remains generic -- any specific tool used in our instantiation can be replaced by another one with compatible functionality. We apply our framework on an industrial airbag control chip case study that we use to illustrate step-by-step our requirements-driven analysis and test methodology.

Model-Based Development of Distributed Embedded Real-Time Systems with the DECOS Tool-Chain

The increasing complexity of distributed embedded systems, as found today in airplanes or cars, becomes more and more a critical cost-factor for their development. Model-based approaches have recently demonstrated their potential for both improving and accelerating (software) development processes. Therefore, in the project DECOS1, which aims at improving system architectures and development of distributed safety-critical embedded systems, an integrated, model-driven tool-chain is established, accompanying the system development process from design to deployment. This paper gives an overview of this tool-chain and outlines important design decisions and features. Copyright

Checking SCADE models for correct usage of physical units

Mismatches of units and of scales of values in physical calculations are disastrous, but rather common, in the development of embedded control systems. They can be as plain as mixing feet and metres, or as hidden as a wrong exponent in a complex calculation formula. These errors can be found by a checking algorithm, following some simple rules, if information on the units of the used variables is provided. This paper describes a developer friendly approach of providing this checking functionality in SCADE, a model-based graphical development tool for safety-critical embedded applications.

Checking Response-Time Properties of Web-Service Applications Under Stochastic User Profiles

Performance evaluation of critical software is important but also computationally expensive. It usually involves sophisticated load-testing tools and demands a large amount of computing resources. Analysing different user populations requires even more effort, becoming infeasible in most realistic cases. Therefore, we propose a model-based approach. We apply model-based test-case generation to generate log-data and learn the associated distributions of response times. These distributions are added to the behavioural models on which we perform statistical model checking (SMC) in order to assess the probabilities of the required response times. Then, we apply classical hypothesis testing to evaluate if an implementation of the behavioural model conforms to these timing requirements. This is the first model-based approach for performance evaluation combining automated test-case generation, cost learning and SMC for real applications. We realised this method with a property-based testing tool, extended with SMC functionality, and evaluate it on an industrial web-service application.

Model-based, mutation-driven test case generation via heuristic-guided branching search

This work introduces a heuristic-guided branching search algorithm for model-based, mutation-driven test case generation. The algorithm is designed towards the efficient and computationally tractable exploration of discrete, non-deterministic models with huge state spaces. Asynchronous parallel processing is a key feature of the algorithm. The algorithm is inspired by the successful path planning algorithm Rapidly exploring Random Trees (RRT). We adapt RRT in several aspects towards test case generation. Most notably, we introduce parametrized heuristics for start and successor state selection, as well as a mechanism to construct test cases from the data produced during search. We implemented our algorithm in the existing test case generation framework MoMuT. We present an extensive evaluation of our heuristics and parameters based on a diverse set of demanding models obtained in an industrial context. In total we continuously utilized 128 CPU cores on three servers for two weeks to gather the experimental data presented. Using statistical methods we determine which heuristics are performing well on all models. With our new algorithm, we are now able to process models consisting of over 2300 concurrent objects. To our knowledge there is no other mutation driven test case generation tool that is able to process models of this magnitude.