Mercedes G. Merayo

Formal testing from timed finite state machines

In this paper we present a formal methodology to test both the functional and temporal behaviors in systems where temporal aspects are critical. We extend the classical finite state machines model with features to represent timed systems. Our formalism allows three different ways to express the timing requirements of systems. Specifically, we consider that time requirements can be expressed either by means of fix time values, by using random variables, or by considering time intervals. Different implementation relations, depending on both the interpretation of time and on the non-determinism appearing in systems, are presented and related. We also study how test cases are defined and applied to implementations. Test derivation algorithms, producing sound and complete test suites, are also presented. That is, by deriving these test suites we relate the different notions of passing tests and the different implementation relations. In other words, for a given correctness criterion, a system represents an appropriate implementation of a given model if and only if the system successfully passes all the test belonging to the derived test suite.

Timed implementation relations for the distributed test architecture

In order to test systems that have physically distributed interfaces, called ports, we might use a distributed approach in which there is a separate tester at each port. If the testers do not synchronise during testing then we cannot always determine the relative order of events observed at different ports and this leads to new notions of correctness that have been described using corresponding implementation relations. We study the situation in which each tester has a local clock and timestamps its observations. If we know nothing about how the local clocks relate then this does not affect the implementation relation while if the local clocks agree exactly then we can reconstruct the sequence of observations made. In practice, however, we are likely to be between these extremes: the local clocks will not agree exactly but we have some information regarding how they can differ. We start by assuming that a local tester interacts synchronously with the corresponding port of the system under test and then extend this to the case where communications can be asynchronous, considering both the first-in-first-out (FIFO) case and the non-FIFO case. The new implementation relations are stronger than implementation relations for distributed testing that do not use timestamps but still reflect the distributed nature of observations. This paper explores these alternatives and derives corresponding implementation relations.

Extending EFSMs to Specify and Test Timed Systems with Action Durations and Time-Outs

In this paper, we introduce a timed extension of the extended finite state machines model. On one hand, we consider that (output) actions take time to be performed. This time may depend on several factors, such as the value of variables. On the other hand, our formalism allows us to specify time-outs. In addition to presenting our language, we develop a testing theory. First, we define 10 timed conformance relations and relate them. Second, we introduce a notion of timed test and define how to apply tests to implementations. Finally, we give an algorithm to derive sound and complete test suites with respect to the implementation relations presented in the paper. This paper represents an extended and improved version of [1].

HOTL: Hypotheses and observations testing logic☆

El proyecto realizado consiste en la implementacion de una herramienta capaz de simular el comportamiento descrito en una logica desarrollada (HOTL: Hypotheses and observations testing logic), en un excelente trabajo de investigacion, por parte de nuestro profesor director de proyecto D. Ismael Rodriguez Laguna y sus dos companeros D. Manuel Nunez y Dna. Mercedes G. Merayo. La implementacion ha sido integrada en una interfaz grafica que nos permitira ejecutar una seria de reglas definidas en la logica dado un modelo, y dadas una especificacion concreta, unas observaciones y unas hipotesis y la aplicacion de las reglas sobre estos elementos, nos diga si la implementacion es conforme o no respecto de la especificacion. Todo esto mostrandose de manera grafica para que el usuario sea consciente de lo ocurrido en cada momento, como por ejemplo; la carga de la especificacion, de las observaciones e hipotesis, los modelos generados a partir de las observaciones, la generacion de modelos que implica la ejecucion de las reglas una a una, etc. La herramienta ha sido implementada sobre una plataforma Java y su diseno se ha realizado llevando a cabo tecnicas y metodos empleados en ingenieria del software, tales como un arquitectura clara de la aplicacion (que facilitara la ampliacion futura de la herramienta), uso de patrones de diseno, division del trabajo en iteraciones, etc., ademas del uso de conocimientos de metodologia y tecnologia de la programacion para el desarrollo de ciertos algoritmos necesarios para la implementacion de la logica.[ABSTRACT]The made project consists of the implementation of a tool able to simulate the behavior described in a developed logic (HOTL: Hypotheses and observations testing logic), in an excellent work of investigation, on the part of our professor director of project D. Ismael Rodriguez Laguna and their two companions D. Manuel Nunez and Dna. Mercedes G. Merayo.[ABSTRACT] The implementation has been integrated in a graphical interface that will allow us to execute serious of rules defined in the given logic a model, and given a concrete specification, observations and a hypothesis and the application of the rules on these elements, says to us if the implementation is in agreement or nonrespect to the specification. All this being of graphical way so that the user is conscious of the happened thing at every moment, like for example; the load of the specification, the observations and hypothesis, the models generated from the observations, the generation of models that the execution of rules one to one implies, etc. The tool has been implemented on a Java platform and its technical design has been made carrying out and used methods in engineering of the software, such as a clear architecture of the application (that it will facilitate the future extension of the tool), use of design patterns, division of the work in iterations, etc., in addition to the use of knowledge of methodology and technology of the programming for the development of certain necessary algorithms for the implementation of the logic.

Implementation Relations for the Distributed Test Architecture

Some systems interact with their environment at a number of physically distributed interfaces called ports. When testing such a system under test (SUT) it is normal to place a local tester at each port and the local testers form a local test case. If the local testers cannot interact with one another and there is no global clock then we are testing in the distributed test architecture. In this paper we explore the effect of the distributed test architecture when testing an SUT against an input output transition system, adapting the ioco implementation relation to this situation. In addition, we define what it means for a local test case to be deterministic, showing that we cannot always implement a deterministic global test case as a deterministic local test case. Finally, we show how a global test case can be mapped to a local test case.

Mutation testing from probabilistic and stochastic finite state machines

Specification mutation involves mutating a specification, and for each mutation a test is derived that distinguishes the behaviours of the mutated and original specifications. This approach has been applied with finite state machine based models. This paper extends mutation testing to finite state machine models that contain non-functional properties. The paper describes several ways of mutating a finite state machine with probabilities (PFSM) or stochastic time (PSFSM) attached to its transitions and shows how we can generate test sequences that distinguish between such a model and its mutants. Testing then involves applying each test sequence multiple times, observing the resultant behaviours and using results from statistical sampling theory in order to compare the observed frequency and execution time of each output sequence with that expected.

Testing from a Stochastic Timed System with a Fault Model

In this paper we present a method for testing a system against a non-deterministic stochastic finite state machine. As usual, we assume that the functional behaviour of the system under test (SUT) is deterministic but we allow the timing to be non-deterministic. We extend the state counting method of deriving tests, adapting it to the presence of temporal requirements represented by means of random variables. The notion of conformance is introduced using an implementation relation considering temporal aspects and the limitations imposed by a black-box framework. We propose a new group of implementation relations and an algorithm for generating a test suite that determines the conformance of a deterministic SUT with respect to a non-deterministic specification. We show how previous work on testing from stochastic systems can be encoded into the framework presented in this paper as an instantiation of our parameterized implementation relation. In this setting, we use a notion of conformance up to a given confidence level.

Formal passive testing of timed systems: theory and tools

This paper presents a methodology to perform passive testing of timed systems. In passive testing, the tester does not interact with the implementation under test. On the contrary, execution traces are observed without interfering with the behaviour of the system. Invariants are used to represent the most relevant expected properties of the implementation under test. Intuitively, an invariant expresses the fact that each time the implementation under test performs a given sequence of actions, it must exhibit a behaviour in a lapse of time reflected in the invariant. There are two types of invariants: consequent and observational. The paper gives two algorithms to decide the correctness of proposed invariants with respect to a given specification and algorithms to check the correctness of a log, recorded from the implementation under test, with respect to an invariant. The soundness of this methodology is shown by relating it to an implementation relation. In addition to the theoretical framework, a tool called PASTE has been developed. This tool helps in the automation of the passive testing approach because it implements all the algorithms presented in this paper. PASTE takes advantage of mutation testing techniques in order to evaluate the goodness of an invariant according to its capability to detect errors in logs generated from mutants. An empirical study where PASTE was used to analyse a non‐trivial system is also reported. Copyright

Passive Testing of Timed Systems

This paper presents a methodology to perform passive testing based on invariants for systems that present temporal restrictions. Invariants represent the most relevant expected properties of the implementation under test. Intuitively, an invariant expresses the fact that each time the implementation under test performs a given sequence of actions, then it must exhibit a behavior in a lapse of time reflected in the invariant. In particular, the algorithm presented in this paper are fully implemented.

Mutation Testing from Probabilistic Finite State Machines

Mutation testing traditionally involves mutating a program in order to produce a set of mutants and using these mutants in order to either estimate the effectiveness of a test suite or to drive test generation. Recently, however, this approach has been applied to specifications such as those written as finite state machines. This paper extends mutation testing to finite state machine models in which transitions have associated probabilities. The paper describes several ways of mutating a probabilistic finite state machine (PFSM) and shows how test sequences that distinguish between a PFSM and its mutants can be generated. Testing then involves applying each test sequence multiple times, observing the resultant output sequences and using results from statistical sampling theory in order to compare the observed frequency of each output sequence with that expected.