Luis Llana

A General Testability Theory: Classes, Properties, Complexity, and Testing Reductions

In this paper we develop a general framework to reason about testing. The difficulty of testing is assessed in terms of the amount of tests that must be applied to determine whether the system is correct or not. Based on this criterion, five testability classes are presented and related. We also explore conditions that enable and disable finite testability, and their relation to testing hypotheses is studied. We measure how far incomplete test suites are from being complete, which allows us to compare and select better incomplete test suites. The complexity of finding that measure, as well as the complexity of finding minimum complete test suites, is identified. Furthermore, we address the reduction of testing problems to each other, that is, we study how the problem of finding test suites to test systems of some kind can be reduced to the problem of finding test suites for another kind of systems. This enables to export testing methods. In order to illustrate how general notions are applied to specific cases, many typical examples from the formal testing techniques domain are presented.

Input-Output Conformance Simulation (iocos) for Model Based Testing

A new model based testing theory built on simulation semantics is presented. At the core of this theory there is an input-output conformance simulation relation (iocos). As a branching semantics iocos can naturally distinguish the context of local choices. We show iocos to be a finer relation than the classic ioco conformance relation. It turns out that iocos is a transitive relation and therefore it can be used both as a conformance relation and a refinement preorder. An alternative characterisation of iocos is provided in terms of testing semantics. Finally we present an algorithm that produces a test suite for any specification. The resulting test suite is sound and exhaustive for the given specification with respect to iocos.

Formally transforming user-model testing problems into implementer-model testing problems and viceversa☆

Abstract There are several methods to assess the capability of a test suite to detect faults in a potentially wrong system. We explore two methods based on considering some probabilistic information. In the first one, we assume that we are provided with a probabilistic user model . This is a model denoting the probability that the entity interacting with the system takes each available choice. In the second one, we suppose that we have a probabilistic implementer model , that is, a model denoting the probability that the implementer makes each possible fault while constructing the system. We show that both testing scenarios are strongly related. In particular, we prove that any user can be translated into an implementer model in such a way that the optimality of tests is preserved, that is, a test suite is optimal for the user if and only if it is optimal for the resulting implementer. Another translation, working in the opposite direction, fulfills the reciprocal property. Thus, we conclude that any test selection criterium designed for one of these testing problems can be used for the other one, once the model has been properly translated. Besides, the applicability of user models to other kinds of testing approaches is considered.

Extending mCRL2 with ready simulation and iocos input-output conformance simulation

The mCRL2 toolset is a leading tool for the use of formal methods. It integrates modelling, analysis and verification methods and techniques strongly based on up to date research results and algorithms. In this paper we describe an extension of mCRL2 that integrates two branching semantics initially not present at the original bundle, the classic ready simulation and the newer input-output conformance simulation (iocos). We use systems from the Very Large Transition Systems (VLTS) benchmark, with states ranging from 103 to 106, to check the implementations and to compare the results with the simpler simulation semantics already included in mCRL2. The results show the feasibility and applicability of the ready and iocos semantics introduced in mCRL2. The good results, in general, highlights the interest of the family of branching semantics for their use in formal methods.

FLOP, a free laboratory of programming

The Test Driven Design (TDD) methodology [4, 23, 8] is currently a very common approach for programming and software engineering learning. On-line judges are widely used in everyday teaching, and their use in the scope of programming contests is currently especially well known. There are good tools and collections of programming problems available for exams as well as for contests. We have developed a simple, light, and practical open laboratory. The term open is used here in two senses: It is free for students to use and free to download and distribute under the GPL license. This laboratory hosts programming problems, it allows the instructor to easily add new ones, and it also automatically assesses the solutions sent by the students. In addition to the system, we have developed a collection of programming problems for CS1/2, designed from a pedagogical point of view and covering several levels of difficulty.

Formally comparing user and implementer model-based testing methods

There are several methods to assess the capability of a test suite to detect faults in a potentially wrong system. We explore two methods based on considering some probabilistic information. In the first one, we assume that we are provided with a probabilistic user model. This is a model denoting the probability that the entity interacting with the system takes each available choice. In the second one, we suppose that we have a probabilistic implementer model, that is, a model denoting the probability that the implementer makes each possible fault while constructing the system. We show that both testing scenarios are strongly related. In particular, we prove that any user can be translated into an implementer model in such a way that the optimality of tests is preserved, that is, a test suite is optimal for the user if and only if it is optimal for the resulting implementer. Another translation, working in the opposite direction, fulfills the reciprocal property. Thus, we conclude that any test selection criterium designed for one of these testing problems can be used for the other one, once the model has been properly translated.

Effectiveness for Input Output Conformance Simulation iocos̱

In this paper we continue the study of the input-output conformance simulation (ioco s). In particular, we focus on implementation aspects to show that ioco s is indeed an interesting semantic relation for formal methods. We address two complementary issues: a) In the context of model based testing (MBT) we present an online, also called on-the-fly, testing algorithm that checks whether an implementation conforms a given specification. Online testing combines test generation and execution and avoids the generation of the complete test suite for the specification. We prove both soundness and completeness of the online algorithm with respect to the ioco s relation. b) In the context of formal verification and model checking minimisation a key issue is to efficiently compute the considered semantic relations; we show how the coinductive flavour of our conformance relation ioco s makes it appropriate to be cast into an instance of the Generalised Coarsest Partition Problem (GCPP) and thus it can be efficiently computed.

IOCO as a Simulation

Since

Fuzzy-Timed automata

\mathbin {\mathsf {ioco}}