Martin Tappler

Model-Based Testing IoT Communication via Active Automata Learning

This paper presents a learning-based approach to detecting failures in reactive systems. The technique is based on inferring models of multiple implementations of a common specification which are pair-wise cross-checked for equivalence. Any counterexample to equivalence is flagged as suspicious and has to be analysed manually. Hence, it is possible to find possible failures in a semi-automatic way without prior modelling. We show that the approach is effective by means of a case study. For this case study, we carried out experiments in which we learned models of five implementations of MQTT brokers/servers, a protocol used in the Internet of Things. Examining these models, we found several violations of the MQTT specification. All but one of the considered implementations showed faulty behaviour. In the analysis, we discuss effectiveness and also issues we faced.

Symbolic Input-Output Conformance Checking for Model-Based Mutation Testing

This paper presents an approach to use symbolic input output conformance checking for mutation-based test case generation. In this approach, a possibly non-deterministic action system model is used as basis for generating a number of mutants. Subsequently after the generation of mutants, the original model and the mutants are simultaneously symbolically executed and tested for conformance. Distinguishing test cases are generated, if non-conformance is detected during this process. Several optimisations of the conformance check are presented and their effectiveness is underpinned by listing experimental results.

Learning from Faults: Mutation Testing in Active Automata Learning

System verification is often hindered by the absence of formal models. Peled et al. proposed black-box checking as a solution to this problem. This technique applies active automata learning to infer models of systems with unknown internal structure.

Model Learning and Model-Based Testing

We present a survey of the recent research efforts in integrating model learning with model-based testing. We distinguished two strands of work in this domain, namely test-based learning (also called test-based modeling) and learning-based testing. We classify the results in terms of their underlying models, their test purpose and techniques, and their target domains.

Probabilistic Black-Box Reachability Checking

Model checking has a long-standing tradition in software verification. Given a system design it checks whether desired properties are satisfied. Unlike testing, it cannot be applied in a black-box setting. To overcome this limitation Peled et al. introduced black-box checking, a combination of testing, model inference and model checking. The technique requires systems to be fully deterministic. For stochastic systems, statistical techniques are available. However, they cannot be applied to systems with non-deterministic choices. We present a black-box checking technique for stochastic systems that allows both, non-deterministic and probabilistic behaviour. It involves model inference, testing and probabilistic model-checking. Here, we consider reachability checking, i.e., we infer near-optimal input-selection strategies for bounded reachability.

Efficient Active Automata Learning via Mutation Testing

System verification is often hindered by the absence of formal models. Peled et al. proposed black-box checking as a solution to this problem. This technique applies active automata learning to infer models of systems with unknown internal structure. This kind of learning relies on conformance testing to determine whether a learned model actually represents the considered system. Since conformance testing may require the execution of a large number of tests, it is considered the main bottleneck in automata learning. In this paper, we describe a randomised conformance testing approach which we extend with fault-based test selection. To show its effectiveness we apply the approach in learning experiments and compare its performance to a well-established testing technique, the partial W-method. This evaluation demonstrates that our approach significantly reduces the cost of learning. In multiple experiments, we reduce the cost by at least one order of magnitude.

Conformance Checking of Real-Time Models

We compare conformance checking based on symbolic execution to conformance checking via bounded model checking. The application context is fault-based test case generation, focusing on real-time faults. The existing bounded model checking approach is performed on timed automata. It supports time-relevant mutation operators and a preprocessing functionality for removing silent transitions and non-determinism. The new symbolic execution approach is performed on timed action systems, which are a novel variant of Backs action systems augmented by clock variables and real-time semantics. It supports the same set of mutation operators, silent transitions, non-determinism and data variables. We show how to encode timed automata as timed action systems and perform experiments on three variants of a car alarm system, to investigate the influence of silent transitions, non-determinism and data variables. Both approaches rely on the SMT solver Z3.

Does this fault lead to failure? Combining refinement and input–output conformance checking in fault-oriented test-case generation

Abstract In this paper we describe an advanced test-case generation technique that is implemented in our model-based test-case generator MoMuT::UML. The tool injects faults into a UML model and analyses if the faults propagate to the interface. If a fault does propagate to an observable failure, an explaining sequence of events is generated and converted into a test-case scenario. The faults are detected using a highly optimised refinement checker, their propagation is analysed with an input–output conformance (ioco) checker. We show that this combination is faster than pure input–output conformance checking. It has been used in a recent industrial application of testing automotive measurement devices. The refinement and ioco checker are implemented in Prolog using the SMT solver Z3.