Stephane Maag

FSM-based conformance testing methods: A survey annotated with experimental evaluation

The development of test cases is an important issue for testing software, communication protocols and other reactive systems. A number of methods are known for the development of a test suite based on a formal specification given in the form of a finite state machine. In this paper, we overview and experiment with these methods to assess their complexity, applicability, completeness, fault detection capability, length and derivation time of their test suites. The experiments are conducted on randomly generated specifications and on two realistic protocols called the Simple Connection Protocol and the ITU-T V.76 Recommendation.

A formal data-centric approach for passive testing of communication protocols

There is currently a high level of consciousness of the importance and impact of formally testing communicating networks. By applying formal description techniques and formal testing approaches, we are able to validate the conformance of implementations to the requirements of communication protocols. In this context, passive testing techniques are used whenever the system under test cannot be interrupted or access to its interfaces is unavailable. Under such conditions, communication traces are extracted from points of observation and compared to the expected behavior formally specified as properties. Since most works on the subject come from a formal model context, they are optimized for testing the control part of the communication with a secondary focus on the data parts. In the current work, we provide a data-centric approach for black-box testing of network protocols. A formalism is provided to express complex properties in a bottom-up fashion starting from expected data relations in messages. A novel algorithm is provided for evaluation of properties in protocol traces. Experimental results on Session Initiation Protocol (SIP) traces for IP Multimedia Subsystem (IMS) services are provided.

A validation model for the DSR protocol

We present a validation model for the dynamic source routing (DSR) protocol. This model includes a formal specification of the protocol and a set of scenarios. The scenarios test the conformance of a given implementation to some targeted system functionalities. The DSR protocol has been specified following the IETF draft [D.B. Johnson, et al., (2003)]. The formal specification has been performed using the SDL language and the scenarios have been generated from the specification using a method and a tool developed at INT [A. Cavalli, et al., (1999)]. The test generation method is based on a set of test purposes that express specific system properties and is completely automated. We also present the experimentation results of the application of our tool to the DSR protocol.

FSM-Based Test Derivation Strategies for Systems with Time-Outs

The paper presents a method for deriving a complete test suite for a timed deterministic FSM with time-outs when only the upper bound on the number of states and the largest finite time-out at a state of an implementation under test are known. We also show that a test suite derived for a corresponding classical FSM is much longer than this obtained directly from a FSM with time-outs. The application to a case study, the Loan Approval Service, illustrates how our approach can be applied for deriving tests for compositions of timed FSMs.

From UML models to automatic generated tests for the dotLRN e-learning platform

This paper presents a method for testing an e-learning, web based system. System specifications are provided using the UML modelling language and specifically the Sequence, Activity and Class diagrams of UML. These specifications are exported in XMI format which is parsed in order to produce the test cases. The system under consideration in this paper is dotLRN, an open source enterprise-class suite of web applications and portal framework for supporting course management, online communities and collaboration.

An Automated Passive Testing Approach for the IMS PoC Service

Although the adoption of the IP Multimedia Subsystem (IMS) keeps growing, IMS applications are often integrated to the system without being formally tested. In this work, we are interested in the IMS Push over Cellular (PoC) service, an OMA standard. We propose a conformance passive testing approach to check that its implementation respects the main standard requirements. This approach is based on a set of formal invariants representing the most relevant expected properties to be tested. Two testing phases are applied: the verification of the invariants against the service specification and their testing on the PoC collected execution traces.

Passive performance testing of network protocols

Complementary to performance evaluation, performance testing of communicating protocols is a qualitative and quantitative test of a system, aiming at checking whether performance requirements of protocols have been satisfied under certain conditions. It raises an interesting issue of accurately formalizing specified performance requirements by taking consideration of data values of the protocol messages. In this paper, we present a novel logic-based testing approach to check protocol performance requirements through real execution traces and formally specified properties. In order to evaluate and assess our methodology, we develop a prototype and present experiments through a set of Session Initiation Protocol (SIP) properties. Finally, a performance benchmark method is proposed and relevant verdicts and discussions are provided.

Interoperability testing of a MANET routing protocol using a node self-similarity approach

Interoperability testing for ad hoc routing protocols is crucial to the reliability of wireless mobile ad hoc networks. Nevertheless, most of the works in this area are devoted to simulations analyzing the performance of such protocols without taking into account their formal requirements and specifications. Our works deal with formal methods to test the interoperability of a MANET routing protocol, DSR. From previous works, many inconclusive verdicts were obtained on some test cases. We present therefore in this paper, a node self-similarity approach in order to improve the interoperability testing by reducing the number of these in conclusive verdicts.

Using passive testing based on symbolic execution and slicing techniques

This paper presents a new approach to perform passive testing based on the analysis of the control and data part of the system under test. Passive testing techniques are based on the observation and verification of properties on the behaviour of a system without interfering with its normal operation. Many passive testing techniques consider only the control part of the system and neglect data, or are confronted with an overwhelming amount of data values to process. In our approach, we consider control and data parts by integrating the concepts of symbolic execution and we improve trace analysis by introducing trace slicing techniques. Properties are described using Input-Output Symbolic Transition Systems (IOSTSs) and we illustrate in the paper how they can be tested on real execution traces optimizing the trace analysis. These properties can be designed to test the functional conformance of a protocol as well as security properties.In addition to the theoretical approach, we have developed a software tool that implements the algorithms presented in this paper. Finally, as a proof of concept of our approach and tool we have applied the techniques to a real-life case study: the SIP protocol. In particular, the proposed techniques are applied to a set of real execution traces extracted from an IMS/SIP architecture.

A Passive Testing Approach for Protocols in Internet of Things

Recently, XMPP protocol is gaining momentum in Internet of things (IoT). It has been widely used in chatting, message exchanging, unique addressing, etc. which raises an interesting issue: How to formally test the conformance and performance of XMPP in IoT environment. While conformance testing of communicating protocols is a functional test which verifies whether the behaviours of the protocol satisfy defined requirements, performance testing is a qualitative and quantitative test which aims at checking whether the performance requirements of the protocol are satisfied under certain conditions. In this paper, we present a logic-based approach to test the conformance and performance of XMPP protocol through real execution traces and formally specified properties. In order to evaluate and assess our methodology, we have developed a prototype and present experiments with a set of XMPP properties. Finally, the relevant verdicts and conclusions are provided.