Adenilso da Silva Simão

Proteum: a family of tools to support specification and program testing based on mutation

The qualification of the VV&T- Verification, Validation and Testing - activity is extremely relevant to the software development process. The establishment of a low-cost, effective testing and validation strategy and the development of supporting tools have been pursued by many researchers. This presentation discusses the main architectural and operational aspects of a family of tools that support specification and program testing based on mutation. The testing of C programs is supported by Proteum/IM 2.0, at the unit and at the integration level as well. Proteum is an acronym for PROgram Testing Using Mutants. At the specification level the application of mutation testing for validating Reactive Systems (RS) specifications based on Finite State Machines (FSM), Statecharts and Petri Nets is support by Proteum/RS.

Comparing finite state machine test

To plan testing activities, testers face the challenge of determining a strategy, including a test coverage criterion that offers an acceptable compromise between the available resources and test goals. Known theoretical properties of coverage criteria do not always help and, thus, empirical data are needed. The results of an experimental evaluation of several coverage criteria for finite state machines (FSMs) are presented, namely, state and transition coverage; initialisation fault and transition fault coverage. The first two criteria focus on FSM structure, whereas the other two on potential faults in FSM implementations. The authors elaborate a comparison approach that includes random generation of FSM, construction of an adequate test suite and test minimisation for each criterion to ensure that tests are obtained in a uniform way. The last step uses an improved greedy algorithm.

Generating Reduced Tests for FSMs with Extra States

We address the problem of generating tests from a deterministic Finite State Machine to provide full fault coverage even if the faults may introduce extra states in the implementations. It is well-known that such tests should include the sequences in the so-called traversal set, which contains all sequences of length defined by the number of extra states. Therefore, the only apparent opportunity to produce shorter tests is to find within a test suite a suitable arrangement of the sequences in the inescapable traversal set. We observe that the direct concatenation of the traversal set to a given state cover, suggested by all existing generation methods with full fault coverage, results in extensive test branching, when a test has to be repeatedly executed to apply all the sequences of the traversal set. In this paper, we state conditions which allow distributing these sequences over several tests. We then utilize these conditions to elaborate a method, called SPY-method, which shortens tests by avoiding test branching as much as possible. We present the results of the experimental comparison of the proposed method with an existing method which indicate that the resulting save can be up to 40%.

Generating Checking Sequences for Partial Reduced Finite State Machines

The problem of generating checking sequences for FSMs with distinguishing sequence has been attracting interest of researchers for several decades. In this paper, a solution is proposed for partial reduced FSMs with distinguishing sets, and either with or without reset feature. Sufficient conditions for a sequence to be a checking sequence for such FSMs are formulated. Based on these conditions, a method to generate checking sequence is elaborated. The results of an experimental comparison indicate that the proposed method produces shorter checking sequences than existing methods in most cases. The impact of using the reset feature on the length of checking sequence is also experimentally evaluated.

Checking Completeness of Tests for Finite State Machines

In testing from a Finite State Machine (FSM), the generation of test suites which guarantee full fault detection, known as complete test suites, has been a long-standing research topic. In this paper, we present conditions that are sufficient for a test suite to be complete. We demonstrate that the existing conditions are special cases of the proposed ones. An algorithm that checks whether a given test suite is complete is given. The experimental results show that the algorithm can be used for relatively large FSMs and test suites.

A Technique to Reduce the Test Case Suites for Regression Testing Based on a Self-Organizing Neural Network Architecture

This paper presents a technique to select subsets of the test cases, reducing the time consumed during the evaluation of a new software version and maintaining the ability to detect defects introduced. Our technique is based on a model to classify test case suites by using an ART-2A self-organizing neural network architecture. Each test case is summarized in a feature vector, which contains all the relevant information about the software behavior. The neural network classifies feature vectors into clusters, which are labeled according to software behavior. The source code of a new software version is analyzed to determine the most adequate clusters from which the test case subset will be selected. Experiments compared feature vectors obtained from all-uses code coverage information to a random selection approach. Results confirm the new technique has improved the precision and recall metrics adopted

Generating asynchronous test cases from test purposes

Abstract Context Input/output transition system (IOTS) models are commonly used when next input can arrive even before outputs are produced. The interaction between the tester and an implementation under test (IUT) is usually assumed to be synchronous. However, as the IUT can produce outputs at any moment, the tester should be prepared to accept all outputs from the IUT, or else be able to block (refuse) outputs of the implementation. Testing distributed, remote applications under the assumptions that communication is synchronous and actions can be blocked is unrealistic, since synchronous communication for such applications can only be achieved if special protocols are used. In this context, asynchronous tests can be more appropriate, reflecting the underlying test architecture which includes queues. Objective In this paper, we investigate the problem of constructing test cases for given test purposes and specification input/output transition systems, when the communication between the tester and the implementation under test is assumed to be asynchronous, performed via multiple queues. Method When issuing verdicts, asynchronous tests should take into account a distortion caused by the queues in the observed interactions. First, we investigate how the test purpose can be transformed to account for this distortion when there are a single input queue and a single output queue. Then, we consider a more general problem, when there may be multiple queues. Results We propose an algorithm which constructs a sound test case, by transforming the test purpose prior to composing it with the specification without queues. Conclusion The proposed algorithm mitigates the state explosion problem which usually occurs when queues are directly involved in the composition. Experimental results confirm the resulting state space reduction.

Fault Coverage-Driven Incremental Test Generation

In this paper, we consider a classical problem of complete test generation for deterministic finite-state machines (FSMs) in a more general setting. The first generalization is that the number of states in implementation FSMs can even be smaller than that of the specification FSM. Previous work deals only with the case when the implementation FSMs are allowed to have the same number of states as the specification FSM. This generalization provides more options to the test designer: when traditional methods trigger a test explosion for large specification machines, tests with a lower, but yet guaranteed, fault coverage can still be generated. The second generalization is that tests can be generated starting with a user-defined test suite, by incrementally extending it until the desired fault coverage is achieved. Solving the generalized test derivation problem, we formulate sufficient conditions for test suite completeness weaker than the existing ones and use them to elaborate an algorithm that can be used both for extending user-defined test suites to achieve the desired fault coverage and for test generation. We present the experimental results that indicate that the proposed algorithm allows obtaining a trade-off between the length and fault coverage of test suites.

Structural Testing for Semaphore-Based Multithread Programs

This paper presents structural testing criteria for validation of semaphore-based multithread programs exploring control, data, communication and synchronization information. A post-mortem method based on timestamps is defined to determine the implicit communication among threads using shared variables. The applicability of the coverage testing criteria is illustrated by a case study.

Model-based testing of software and systems: recent advances and challenges

Model-based testing is focused on testing techniques which rely on the use of models. The diversity of systems and software to be tested implies the need for research on a variety of models and methods for test automation. We briefly review this research area and introduce several papers selected from the 22nd International Conference on Testing Software and Systems (ICTSS).