Emanuela Gadelha Cartaxo

On the use of a similarity function for test case selection in the context of model-based testing

Test case selection in model‐based testing is discussed focusing on the use of a similarity function. Automatically generated test suites usually have redundant test cases. The reason is that test generation algorithms are usually based on structural coverage criteria that are applied exhaustively. These criteria may not be helpful to detect redundant test cases as well as the suites are usually impractical due to the huge number of test cases that can be generated. Both problems are addressed by applying a similarity function. The idea is to keep in the suite the less similar test cases according to a goal that is defined in terms of the intended size of the test suite. The strategy presented is compared with random selection by considering transition‐based and fault‐based coverage. The results show that, in most of the cases, similarity‐based selection can be more effective than random selection when applied to automatically generated test suites. Copyright

Test case generation by means of UML sequence diagrams and labeled transition systems

We present a systematic procedure of functional test case generation for feature testing of mobile phone applications. A feature is an increment of functionality, usually with a coherent purpose that is added on top of a basic system. Feature are usually developed and tested separately from the basic system as independent modules. The procedure is based on model-based testing techniques with test cases generated from UML sequence diagrams translated into labeled transition systems (LTSs). A case study is presented to illustrate the application of the procedure. The work is part of a research initiative for automation of test case generation, selection and evaluation of Motorola mobile phone applications.

LTS-BT: a tool to generate and select functional test cases for embedded systems

Automation of model-based testing for embedded systems is discussed. The focus is on feature and feature interruption test case generation and selection from behavioral specifications. For this, the LTS-BT tool is presented. The tool has been designed to suit embedded systems by focusing on selected notations for behavior specification and tailored techniques for test case generation and selection. This is motivated by the particularities of these systems that challenge cost-effective testing.

Revealing influence of model structure and test case profile on the prioritization of test cases in the context of model-based testing

BackgroundTest case prioritization techniques aim at defining an order of test cases that favor the achievement of a goal during test execution, such as revealing failures as earlier as possible. A number of techniques have already been proposed and investigated in the literature and experimental results have discussed whether a technique is more successful than others. However, in the context of model-based testing, only a few attempts have been made towards either proposing or experimenting test case prioritization techniques. Moreover, a number of factors that may influence on the results obtained still need to be investigated before more general conclusions can be reached.MethodsIn order to evaluate factors that potentially affect the performance of test case prioritization techniques, we perform three empirical studies, an exploratory one and two experiments. The first study focus on expose the techniques to a common and fair environment, since the investigated techniques have never been studied together, and observe their behavior. The following two experiments aim at observing the effects of two factors: the structure of the model and the profile of the test cases that fail. We designed the experiments using the one-factor-at-a-time strategy.ResultsThe first study suggests that the investigated techniques performs differently, however other factors, aside from the test suites and number of failures, affect the techniques, motivating further investigation. As results from the two experiments, on one hand, the model structure do not affect significantly the investigated techniques. On the other hand, we are able to state that the profile of the test case that fails may have a definite influence on the performance of the techniques investigated.ConclusionsThrough these studies, we conclude that, a fair evaluation involving test case prioritization techniques must take into account, in addition to the techniques and the test suites, different characteristics of the test cases that fail as variable.

Analysis of distance functions for similarity-based test suite reduction in the context of model-based testing

Test suite reduction strategies aim to produce a smaller and representative suite that presents the same coverage as the original one but is more cost-effective. In the model-based testing (MBT) context, reduction is crucial since automatic generation algorithms may blindly produce several similar test cases. In order to define the degree of similarity between test cases, researchers have investigated a number of distance functions. However, there is still little or no knowledge on whether and how they influence on the performance of reduction strategies, particularly when considering MBT practices. This paper investigates the effectiveness of distance functions in the scope of a MBT reduction strategy based on the similarity degree of test cases. We discuss six distance functions and apply them to three empirical studies. The first two studies are controlled experiments focusing on two real-world applications (and real faults) and ten synthetic specifications automatically generated from the configuration of each application (and faults randomly generated). In the third study, we also apply the reduction strategy to two subsequent versions of an industrial application by considering real faults detected. Results show that the choice of a distance function has little influence on the size of the reduced test suite. However, as reduced suites are different depending on the distance function applied, the choice can significantly affect the fault coverage. Moreover, it can also affect the stability of the reduction strategy regarding coverage of different sets of faults on different executions.

On the Influence of Model Structure and Test Case Profile on the Prioritization of Test Cases in the Context of Model-Based Testing

Test case prioritization techniques aim at defining an ordering of test cases that favor the achievement of a goal during test execution, such as revealing faults as earlier as possible. A number of techniques have already been proposed and investigated in the literature and experimental results have discussed whether a technique is more successful than others. However, in the context of model-based testing, only a few attempts have been made towards either proposing or experimenting test case prioritization techniques. Moreover, a number of factors that may influence on the results obtained still need to be investigated before more general conclusions can be reached. In this paper, we present empirical studies that focus on observing the effects of two factors: the structure of the model and the profile of the test case that fails. Results show that the profile of the test case that fails may have a definite influence on the performance of the techniques investigated.

Weighting Influence of User Behavior in Software Validation

Validation is an essential part of software development, and testing is a practical and widely used approach. The emerging methodology is model-based testing, in which test cases are derived from a model of software behaviour. In this paper we claim that also user behaviour should be taken into account for test planning purposes. We introduce a pragmatic approach called WSA, which derives the test cases from a state-based model, while also accounting for weights that consider relevance wrt user behaviour.

On the Use of Fault Abstractions for Assessing System Test Case Prioritization Techniques

Empirical studies often evaluate Test Case Prioritization (TCP) techniques by measuring their ability to uncover faults as early as possible. In this context, techniques reorder test cases using different sources of information. These prioritizations are evaluated considering faults detected by failure occurrences during testing execution. While this accounts for validity, results can be biased. For instance, conclusions regarding a TCP techniques behavior may be affected by the number of failures, faults and, in turn, the failure-fault relationship. We propose the use of fault abstractions, which relate faults to the related test cases that fail, to exercise techniques on preliminary empirical studies of system level TCP techniques. Rather than modeling specific classes of faults, our strategy is based on identifying cliques of test cases that are related according to a distance function as a fault abstraction, assuming that the test cases can fail due to a number of faults in common. Our strategy calculate these fault abstractions based on the set of maximal cliques obtained from test case distance graphs along with a set of essential test cases. Therefore, one can sample fault abstractions respecting different failure-fault relationships and apply on empirical studies. In our studies, even though our strategy is not predictive, a significant amount of fault abstractions calculated by the proposed technique corresponded to actual reported faults. Therefore Fault abstractions may represent the relationship between faults and failures to assess TCP techniques when none or a small amount of artifacts is available to perform a preliminary empirical evaluation.

Test case prioritization techniques for model-based testing: a replicated study

Recently, several test case prioritization (TCP) techniques have been proposed to order test cases for achieving a goal during test execution, particularly, revealing faults sooner. In the model-based testing (MBT) context, such techniques are usually based on heuristics related to structural elements of the model and derived test cases. In this sense, techniques’ performance may vary due to a number of factors. While empirical studies comparing the performance of TCP techniques have already been presented in literature, there is still little knowledge, particularly in the MBT context, about which factors may influence the outcomes suggested by a TCP technique. In a previous family of empirical studies focusing on labeled transition systems, we identified that the model layout, i.e., amount of branches, joins, and loops in the model, alone may have little influence on the effectiveness of TCP techniques investigated, whereas characteristics of test cases that actually fail definitely influences this aspect. However, we considered only synthetic artifacts in the study, which reduced the ability of representing properly the reality. In this paper, we present a replication of one of these studies, now with a larger and more representative selection of techniques and considering test suites from industrial systems as experimental objects. Our objective is to find out whether the results remain while increasing the validity in comparison to the original study. Results reinforce that there is no best performer among the investigated techniques and characteristics of test cases that fail represent an important factor, although adaptive random-based techniques are less affected by it.