Birgitta Lindström

An evaluation of combination strategies for test case selection

This paper presents results from a comparative evaluation of five combination strategies. Combination strategies are test case selection methods that combine “interesting” values of the input parameters of a test subject to form test cases. This research comparatively evaluated five combination strategies; the All Combination strategy (AC), the Each Choice strategy (EC), the Base Choice strategy (BC), Orthogonal Arrays (OA) and the algorithm from the Automatic Efficient Test Generator (AETG). AC satisfies n-wise coverage, EC and BC satisfy 1-wise coverage, and OA and AETG satisfy pair-wise coverage. The All Combinations strategy was used as a “gold standard” strategy; it subsumes the others but is usually too expensive for practical use. The others were used in an experiment that used five programs seeded with 128 faults. The combination strategies were evaluated with respect to the number of test cases, the number of faults found, failure size, and number of decisions covered. The strategy that requires the least number of tests, Each Choice, found the smallest number of faults. Although the Base Choice strategy requires fewer test cases than Orthogonal Arrays and AETG, it found as many faults. Analysis also shows some properties of the combination strategies that appear significant. The two most important results are that the Each Choice strategy is unpredictable in terms of which faults will be revealed, possibly indicating that faults are found by chance, and that the Base Choice and the pair-wise combination strategies to some extent target different types of faults.

On Strong Mutation and Subsuming Mutants

Mutation analysis is a powerful technique for software testing but it is also known to be computationally expensive. The main reason for the high computational cost is that many of the mutants are redundant and thus, do not contribute to the quality of the test suite. One of the most promising approaches to avoid producing redundant mutants is to identify subsumption relations among mutants, preferably before these are generated. Such relations have for example, been identified at an operator level for mutants created by the ROR operator. This reduced set of non-redundant mutants has been used in several recent studies and is also the default option in at least one mutation testing tool that supports strong mutation. This raises questions on whether the identified subsumption relations between the mutants hold in a context of strong mutation or variants of weak mutation that require some limited error propagation (firm mutation). We have conducted an experimental study to investigate the subsumption relations in the context of strong or firm mutation. We observed that it is possible to create a test suite that is 100% adequate for the reduced set of mutants while not being 100% adequate for the complete set. This shows that the subsumption relations do not hold for strong or firm mutation. We provide several examples on this behavior and discuss the root causes. Our findings are important since strong and firm mutation both are frequently used to evaluate test suites and testing criteria. The choice of whether to use a reduced set of mutants or an entire set should however, not be made without consideration of the context in which they are used (i.e., strong, firm or weak mutation) since the subsumption relations between ROR mutants do not hold for strong or firm mutation. Just as redundant mutants can give an overestimation of the mutation score for a test suite, using the reduced set of mutants can give an underestimation if used together with strong or firm mutation. Results reported from such studies should therefore, be accompanied by information on whether the reduced or complete set of mutants was used and if the researchers used strong, firm or weak mutation.

Testability and Software Robustness: A Systematic Literature Review

The concept of software testability has been researched in several different dimensions, however the relation of this important concept with other quality attributes is a grey area where existing evidence is scattered. The objective of this study is to present a state-of-the-art with respect to issues of importance concerning software testability and an important quality attribute: software robustness. The objective is achieved by conducting a systematic literature review (SLR) on the topic. Our results show that a variety of testability issues are in focus with observability and controllability issues being most researched. Fault tolerance, exception handling and handling external influence are prominent robustness issues in focus.

Robustness Testing of Embedded Software Systems: An Industrial Interview Study

Embedded software is at the core of current and future telecommunication, automotive, multimedia, and industrial automation systems. The success of practically any industrial application depends on the embedded software systems dependability, and one method to verify the dependability of a system is testing its robustness. The motivation behind this paper is to provide a knowledge base of the state of the practice in robustness testing of embedded software systems and to compare this to the state of the art. We have gathered the information on the state of the practice in robustness testing from seven different industrial domains (telecommunication, automotive, multimedia, critical infrastructure, aerospace, consumer products, and banking) by conducting 13 semi-structured interviews. We investigate the different aspects of robustness testing, such as the general view of robustness, relation to requirements engineering and design, test execution, failures, and tools. We highlight knowledge from the state of the practice of robustness testing of embedded software systems. We found different robustness testing practices that have not been previously described. This paper shows that the state of the practice, when it comes to robustness testing, differs between organizations and is quite different from the state of the art described in the scientific literature. For example, methods commonly described in the literature (e.g., the fuzzy approach) are not used in the organizations we studied. Instead, the interviewees described several ad hoc approaches that take specific scenarios into account (e.g., power failure or overload). Other differences we found concern the classification of robustness failures, the hypothesized root causes of robustness failures, and the types of tools used for robustness testing. This paper is a first step in capturing the state of the practice of robustness testing of embedded software systems. The results can be used by both researchers and practitioners. Researchers can use our findings to understand the gap between the state of the art and the state of the practice and develop their studies to fill this gap. Practitioners can also learn from this knowledge base regarding how they can improve their practice and acquire other practices.

Mutating aspect-oriented models to test cross-cutting concerns

Aspect-oriented (AO) modeling is used to separate normal behaviors of software from specific behaviors that affect many parts of the software. These are called “cross-cutting concerns,” and include things such as interrupt events, exception handling, and security protocols. AO modeling allow developers to model the behaviors of cross-cutting concerns independently of the normal behavior. Aspect-oriented models (AOM) are then transformed into code by “weaving” the aspects (modeling the cross-cutting concerns) into all locations in the code where they are needed. Testing at this level is unnecessarily complicated because the concerns are often repeated in many locations and because the concerns are muddled with the normal code. This paper presents a method to design robustness tests at the abstract, or model, level. The models are mutated with novel operators that specifically target the features of AOM, and tests are designed to kill those mutants. The tests are then run on the implementation level to evaluate the behavior of the woven cross-cutting concerns.

Transformation Rules for Platform Independent Testing: An Empirical Study

Most Model-Driven Development projects focus on model-level functional testing. However, our recent study found an average of 67% additional logic-based test requirements from the code compared to the design model. The fact that full coverage at the design model level does not guarantee full coverage at the code level indicates that there are semantic behaviors in the model that model-based tests might miss, e.g., conditional behaviors that are not explicitly expressed as predicates and therefore not tested by logic-based coverage criteria. Avionics standards require that the structure of safety critical software is covered according to logic-based coverage criteria, including MCDC for the highest safety level. However, the standards also require that each test must be derived from the requirements. This combination makes designing tests hard, time consuming and expensive to design. This paper defines a new model that uses transformation rules to help testers define tests at the platform independent model level. The transformation rules have been applied to six large avionic applications. The results show that the new model reduced the difference between model and code with respect to the number of additional test requirements from an average of 67% to 0% in most cases and less than 1% for all applications.

Using an existing suite of test objects: experience from a testing experiment

This workshop paper presents lessons learned from a recent experiment to compare several test strategies. The test strategies were compared in terms of the number of tests needed to satisfy them and in terms of faults found. The experimental design and conduct are discussed, and frank assessments of the decisions that were made are provided. The paper closes with a summary of the lessons that were learned.

Testability and software performance: a systematic mapping study

In most of the research on software testability, functional correctness of the software has been the focus while the evidence regarding testability and non-functional properties such as performance is sporadic. The objective of this study is to present the current state-of-the-art related to issues of importance, types and domains of software under test, types of research, contribution types and design evaluation methods concerning testability and software performance. We find that observability, controllability and testing effort are the main testability issues while timeliness and response time (i.e., time constraints) are the main performance issues in focus. The primary studies in the area use diverse types of software under test within different domains, with realtime systems as being a dominant domain. The researchers have proposed many different methods in the area, however these methods lack implementation in practice.

Model transformation impact on test artifacts: an empirical study

Development environments that support Model-Driven Development often focus on model-level functional testing, enabling verification of design models against their specifications. However, developers of safety-critical software systems are also required to show that tests cover the structure of the implementation. Unfortunately, the implementation structure can diverge from the model depending on choices such as the model compiler or target language. Therefore, structural coverage at the model level may not guarantee coverage of the implementation. We present results from an industrial experiment that demonstrates the model-compiler effect on test artifacts in xtUML models when these models are transformed into C++. Test artifacts, i.e., predicates and clauses, are used to satisfy the structural code coverage criterion, in this case MCDC, which is required by the US Federal Aviation Administration. The results of the experiment show not only that the implementation contains more test artifacts than the model, but also that the test artifacts can be deterministically enumerated during translation. The analysis identifies two major sources for these additional test artifacts.

Generating Trace-Sets for Model-based Testing

Model-checkers are powerful tools that can find individual traces through models to satisfy desired properties. These traces provide solutions to a number of problems. Instead of individual traces, software testing needs sets of traces that satisfy coverage criteria. Finding a trace set in a large model is difficult because model checkers generate single traces and use a lot of memory. Space and time requirements of modelchecking algorithms grow exponentially with respect to the number of variables and parallel automata of the model being analyzed. We present a method that generates a set of traces by iteratively invoking a model checker. The method mitigates the memory consumption problem by dynamically building partitions along the traces. This method was applied to a testability case study, and it generated the complete trace set, while ordinary model-checking could only generate 26%.