Florian Lorber

Time for Mutants — Model-Based Mutation Testing with Timed Automata

Model-based testing is a popular technology for automatic and systematic test case generation (TCG), where a system-under-test (SUT) is tested for conformance with a model that specifies its intended behavior. Model-based mutation testing is a specific variant of model-based testing that is fault-oriented. In mutation testing, the test case generation is guided by a mutant, an intentionally altered version of the original model that specifies a common modeling error.

Integration of Requirements Engineering and Test-Case Generation via OSLC

We present a requirement-centered analysis and testing framework that integrates methods and tools for capturing and formalizing textual customer requirements, analyzing requirements consistency, generating test cases from formalized requirements and executing them on the implementation model. The framework preserves a fine grained traceability of informal and formal requirements, test cases and implementation models throughout every step of the workflow. We instantiate the framework with concrete tools that we integrate via a file repository and Open Services for Lifecycle Collaboration (OSLC). The standardized integration ensures that the framework remains generic -- any specific tool used in our instantiation can be replaced by another one with compatible functionality. We apply our framework on an industrial airbag control chip case study that we use to illustrate step-by-step our requirements-driven analysis and test methodology.

Bounded Determinization of Timed Automata with Silent Transitions

Deterministic timed automata are strictly less expressive than their non-deterministic counterparts, which are again less expressive than those with silent transitions. As a consequence, timed automata are in general non-determinizable. This is unfortunate since deterministic automata play a major role in model-based testing, observability and implementability. However, by bounding the length of the traces in the automaton, effective determinization becomes possible. We propose a novel procedure for bounded determinization of timed automata. The procedure unfolds the automata to bounded trees, removes all silent transitions and determinizes via disjunction of guards. The proposed algorithms are optimized to the bounded setting and thus are more efficient and can handle a larger class of timed automata than the general algorithms. The approach is implemented in a prototype tool and evaluated on several examples. To our best knowledge, this is the first implementation of this type of procedure for timed automata.

Require, Test and Trace IT

We propose a framework for requirement-driven test generation that combines contract-based interface theories with model-based testing. We design a specification language, requirement interfaces, for formalizing different views (aspects) of synchronous data-flow systems from informal requirements. Multiple views of a system, modeled as requirement interfaces, are naturally combined by conjunction.

Towards generation of adaptive test cases from partial models of determinized timed automata

The well-defined timed semantics of timed automata as specification models provide huge advantages for the verification and validation of real-time systems. Thus, timed automata have already been applied in many different areas, including model-based testing. Unfortunately, there is one drawback in using timed automata for test-case generation: if they contain non-determinism or silent transitions, the problem of language inclusion between timed automata becomes undecidable. In recent work, we developed and implemented a technique to determinize timed automata up to a certain depth k. The resulting timed automata are unfolded to directed acyclic graphs (DAGs) up to depth k. The unfolding caused an exponential state-space explosion. Consequently, our model-based test-case generation algorithm for deterministic timed automata, which uses language inclusion, did not scale anymore. Within this paper we investigate how to split the determinized DAGs into partial models, to overcome the problems caused by the increased state space and find effective ways to use the deterministic DAGs for model-based test case generation.

Integrating Model-Based Testing and Analysis Tools via Test Case Exchange

Europes industry in embedded system design is currently aiming for a better integration of tools that support their development, validation and verification processes. The idea is to combine model-driven development with model-based testing and model-based analysis. The interoperability of tools shall be achieved with the help of meta-models that facilitate the mapping between different modelling notations. However, the syntactic and semantic integration of tools is a complex and costly task. A common problem is that different tools support different subsets of a language. Furthermore, semantic differences are a major obstacle to sound integration efforts. In this paper we advocate an alternative, more pragmatic approach. We propose the exchange of test cases generated from the models instead of exchanging the models themselves. The advantage is that test cases have a much simpler syntax and semantics, and hence, the mapping between different tools is easier to implement and to maintain. With a formal testing approach with adequate testing criteria a set of test cases can be viewed as partial models that can be formally analysed. We demonstrate an integration of our test case generator Ulysses with the CADP toolbox by means of test case exchange. We generate test cases in Ulysses and verify properties in CADP. We also generate test cases in CADP and perform a mutation analysis in Ulysses.

Model-Based Mutation Testing of Synchronous and Asynchronous Real-Time Systems

A lot of systems nowadays, especially in safety-critical areas, have to comply to very strict real-time requirements. Deviation from the specified time behavior can lead to severe faults. Thus the components need to be thoroughly tested to ensure a high quality standard. Model-based testing is a popular technology for automatic and systematic test-case generation, where a system-under-test is tested for conformance with a model that specifies its intended behavior. Model-based mutation testing is a specific fault-oriented variant of model- based testing. In this variant of testing, the test case generation is guided by a mutant, an intentionally altered version of the original model that specifies a common modeling error, to generate a specific test case capable of identifying that error. In the presented thesis we investigate ways to apply model-based mutation testing to real-time applications. The plan is to investigate two specification languages: timed automata and timed contracts. We already developed an algorithm and an implementation for testing deterministic timed automata and a theory and implementation for untimed contracts. In the last year of the thesis we will concentrate on determinizing timed automata restricted to the bounded case, supporting the test-case generation from networks of timed automata and adding real-time support to our contract-based testing tool. The final step of the thesis will be the evaluation on industrial case studies.

Formal test-driven development with verified test cases

In this paper we propose the combination of several techniques into an agile formal development process: model-based testing, formal models, refinement of models, model checking, and test-driven development. The motivation is a smooth integration of formal techniques into an existing development cycle. Formal models are used to generate abstract test cases. These abstract tests are verified against requirement properties by means of model checking. The motivation for verifying the tests and not the model is two-fold: (1) in a typical safety-certification process the test cases are essential, not the models, (2) many common modelling tools do not provide a model checker. We refine the models, check refinement, and generate additional test cases capturing the newly added details. The final refinement step from a model to code is done with classical test-driven development. Hence, a developer implements one generated and formally verified test case after another, until all tests pass. The process is scalable to actual needs. Emphasis can be shifted between formal refinement of models and test-driven development. A car alarm system serves as a demonstrating case-study. We use Backs Action Systems as modelling language and mutation analysis for test case generation. We define refinement as input-output conformance (ioco). Model checking is done with the CADP toolbox.

Debugging with Timed Automata Mutations

Model-based Debugging is an application of Model-based Diagnosis techniques to debugging computer systems. Its basic principle is to compare a model, i.e., a description of the correct behaviour of a system, to the observed behaviour of the system. In this paper we show how this technique can be applied in the context of model-based mutation testing (MBMT) with timed automata. In MBMT we automatically generate a set of test sequences out of a test model. In contrast to general model-based testing, the test cases of MBMT cover a pre-defined set of faults that have been injected into the model (model mutation). Our automatic debugging process is purely black-box. If a test run fails, our tool reports a diagnosis as a set of model mutations. These mutations provide possible explanations why the test case has failed. For reproducing the failure, we also generate a set of minimal test cases leading to the implementation fault. The technique is implemented for Uppaals timed automata models and is based on a language inclusion check via bounded model checking. It adds debugging capability to our existing test-case generators. A car-alarm system serves as illustrating case study.

Require, test, and trace IT

We propose a framework for requirement-driven test generation that combines contract-based interface theories with model-based testing. We design a specification language, requirement interfaces, for formalizing different views (aspects) of synchronous data-flow systems from informal requirements. Various views of a system, modeled as requirement interfaces, are naturally combined by conjunction. We develop an incremental test generation procedure with several advantages. The test generation is driven by a single requirement interface at a time. It follows that each test assesses a specific aspect or feature of the system, specified by its associated requirement interface. Since we do not explicitly compute the conjunction of all requirement interfaces of the system, we avoid state space explosion while generating tests. However, we incrementally complete a test for a specific feature with the constraints defined by other requirement interfaces. This allows catching violations of any other requirement during test execution, and not only of the one used to generate the test. This framework defines a natural association between informal requirements, their formal specifications, and the generated tests, thus facilitating traceability. Finally, we introduce a fault-based test-case generation technique, called model-based mutation testing, to requirement interfaces. It generates a test suite that covers a set of fault models, guaranteeing the detection of any corresponding faults in deterministic systems under test. We implemented a prototype test generation tool and demonstrate its applicability in two industrial use cases.