Man Fai Lau

A comparison of MC/DC, MUMCUT and several other coverage criteria for logical decisions

Many testing criteria, including condition coverage and decision coverage, are inadequate for software characterised by complex logical decisions, such as those in safety-critical software. In the past decade, more sophisticated testing criteria have been advocated. In particular, compliance of the MC/DC criterion has been mandated in the commercial aviation industry for the approval of airborne software. Recently, the MUMCUT criterion has been proposed as it guarantees the detection of certain faults in logical decisions in disjunctive normal form in which no variable is redundant. This paper compares MC/DC, MUMCUT and several other related coverage criteria for logical decisions by both formal and empirical analysis, focusing on the fault-detecting ability of test sets satisfying these testing criteria. Our results show that MC/DC test sets are effective, but they may still miss some faults that can almost always be detected by test sets satisfying the MUMCUT criterion.

An extended fault class hierarchy for specification-based testing

Kuhn, followed by Tsuchiya and Kikuno, have developed a hierarchy of relationships among several common types of faults (such as variable and expression faults) for specification-based testing by studying the corresponding fault detection conditions. Their analytical results can help explain the relative effectiveness of various fault-based testing techniques previously proposed in the literature. This article extends and complements their studies by analyzing the relationships between variable and literal faults, and among literal, operator, term, and expression faults. Our analysis is more comprehensive and produces a richer set of findings that interpret previous empirical results, can be applied to the design and evaluation of test methods, and inform the way that test cases should be prioritized for earlier detection of faults. Although this work originated from the detection of faults related to specifications, our results are equally applicable to program-based predicate testing that involves logic expressions.

Automatic generation of test cases from Boolean specifications using the MUMCUT strategy

A recent theoretical study has proved that the MUMCUT testing strategy (1) guarantees to detect seven types of fault in Boolean specifications in irredundant disjunctive normal form, and (2) requires only a subset of the test sets that satisfy the previously proposed MAX-A and MAX-B strategies, which can detect the same types of fault. This paper complements previous work by investigating various methods for the automatic generation of test cases to satisfy the MUMCUT strategy. We evaluate these methods by using several sets of Boolean expressions, including those derived from real airborne software systems. Our results indicate that the greedy CUN and UCN methods are clearly better than others in consistently producing significantly smaller test sets, whose sizes exhibit linear correlation with the length of the Boolean expressions in irredundant disjunctive normal form. This study provides empirical evidences that the MUMCUT strategy is indeed cost-effective for detecting the faults considered in this paper.

Fault-based test suite prioritization for specification-based testing

Context: Existing test suite prioritization techniques usually rely on code coverage information or historical execution data that serve as indicators for estimating the fault-detecting ability of test cases. Such indicators are primarily empirical in nature and not theoretically driven; hence, they do not necessarily provide sound estimates. Also, these techniques are not applicable when the source code is not available or when the software is tested for the first time. Objective: We propose and develop the novel notion of fault-based prioritization of test cases which directly utilizes the theoretical knowledge of their fault-detecting ability and the relationships among the test cases and the faults in the prescribed fault model, based on which the test cases are generated. Method: We demonstrate our approach of fault-based prioritization by applying it to the testing of the implementation of logical expressions against their specifications. We then validate our proposal by an empirical study that evaluates the effectiveness of prioritization techniques using two different metrics. Results: A theoretically guided fault-based prioritization technique generally outperforms other techniques under study, as assessed by two different metrics. Our empirical results also show that the technique helps to reveal all target faults by executing only about 72% of the prioritized test suite, thereby reducing the effort required in testing. Conclusions: The fault-based prioritization approach is not only applicable to the instance empirically validated in this paper, but should also be adaptable to other fault-based testing strategies. We also envisage new research directions to be opened up by our work.

Comparing several coverage criteria for detecting faults in logical decisions

Many testing coverage criteria, including decision coverage and condition coverage, are well-known to be inadequate for software characterised by complex logical decisions, such as those in safety-critical software. In the past decade, more sophisticated testing criteria have been advocated. In particular, compliance of MC/DC has been mandated in the aviation industry for the approval of airborne software. On the other hand, the MUMCUT criterion has been proved to guarantee the detection of certain faults in logical decisions in irredundant disjunctive normal form. We analyse and empirically evaluate the ability of test sets satisfying these testing criteria in detecting faults in logical decisions. Our results show that MC/DC test sets are effective, but they may still miss some faults that can almost always be detected by test sets satisfying the MUMCUT criterion.

On the relationships of faults for Boolean specification based testing

Various methods of generating test cases based on Boolean specifications have previously been proposed. These methods are fault-based in the sense that test cases are aimed at detecting particular types of faults. Empirical results suggest that these methods are good at detecting particular types of faults. However, there is no information on the ability of these test cases in detecting other types of faults. The paper summarizes the relationships of faults in a Boolean expression in the form of a hierarchy. A test case that detects the faults at the lower level of the hierarchy will always detect the faults at the upper level of the hierarchy. The hierarchy helps us to better understand the relationships of faults in a Boolean expression, and hence to select fault-detecting test cases in a more systematic and efficient manner.

Detecting Double Faults on Term and Literal in Boolean Expressions

Fault-based testing aims at selecting test cases to guarantee the detection of certain prescribed faults in programs. The detection conditions of single faults have been studied and used in areas like developing test case selection strategies, establishing relationships between faults and investigating the fault coupling effect. It is common, however, for programmers to commit more than one fault. Our previous studies on the detection conditions of faults in Boolean expressions show that (1) some test case selection strategies developed for the detection of single faults can also detect all double faults related to terms, but (2) these strategies cannot guarantee to detect all double faults related to literals. This paper supplements our previous studies and completes our series of analysis of the detection condition of all double fault classes in Boolean expressions. Here we consider the fault detection conditions of combinations of two single faults, in which one is related to term and the other is related to literal. We find that all such faulty expressions, except two, can be detected by some test case selection strategies for single fault detection. Moreover, the two exception faulty expressions can be detected by existing strategies when used together with a supplementary strategy which we earlier developed to detect double literal faults.

On Detection Conditions of Double FaultsRelated to Terms in Boolean Expressions

Detection conditions of specific classes of faults have recently been studied by many researchers. Under the assumption that at most one of these faults occurs in the software under test, these fault detection conditions were mainly used in two ways. First, they were used to develop test case selection strategies for detecting corresponding classes of faults. Second, they were used to study fault class hierarchies, where a test case that detects a particular class of faults can also detect some other classes of faults. In this paper, we study detection conditions of double faults. Besides developing new test case selection strategies and studying new fault class hierarchies, our analysis provides further insights to the effect of fault coupling. Moreover, these fault detection conditions can be used to compare effectiveness of existing test case selection strategies (which were originally developed for the detection of single occurrence of certain classes of faults) in detecting double faults that may be present in the software

Using the incremental approach to generate test sets: a case study

With the increasing complexity of software systems, the set of testing requirements can become very large. If the set of testing requirements can be naturally decomposed into smaller subsets, one may construct a test set separately to satisfy each subset of testing requirements, and then combine the test sets to form the complete test set. Such an approach is referred to as the union approach. On the other hand, the incremental approach attempts to incrementally expand a test set to satisfy the subsets of testing requirements, one at a time. This paper investigates empirically the effect of the incremental approach as compared to the union approach. Our case study indicates that the incremental approach can result in a significantly smaller test set, particularly when supplemented with the greedy heuristics.

On Comparing Testing Criteria for Logical Decisions

Various test case selection criteria have been proposed for quality testing of software. It is a common phenomenon that test sets satisfying different criteria have different sizes and fault-detecting ability. Moreover, test sets that satisfy a stronger criterion and detect more faults usually consist of more test cases. A question that often puzzles software testing professionals and researchers is: when a testing criterion C 1 helps to detect more faults than another criterion C 2, is it because C 1 specifically requires test cases that are more fault-sensitive than those for C 2, or is it essentially because C 1 requires more test cases than C 2? In this paper, we discuss several methods and approaches for investigating this question, and empirically compare several common coverage criteria for testing logical decisions, taking into consideration the different sizes of the test sets required by these criteria. Our results demonstrate evidently that the stronger criteria under study are more fault-sensitive than the weaker ones, not solely because the former require more test cases. More importantly, we have illustrated a general approach, which takes into account the size factor of the generated test sets, for demonstrating the superiority of a testing criterion over another.