Changhai Nie

A survey of combinatorial testing

Combinatorial Testing (CT) can detect failures triggered by interactions of parameters in the Software Under Test (SUT) with a covering array test suite generated by some sampling mechanisms. It has been an active field of research in the last twenty years. This article aims to review previous work on CT, highlights the evolution of CT, and identifies important issues, methods, and applications of CT, with the goal of supporting and directing future practice and research in this area. First, we present the basic concepts and notations of CT. Second, we classify the research on CT into the following categories: modeling for CT, test suite generation, constraints, failure diagnosis, prioritization, metric, evaluation, testing procedure and the application of CT. For each of the categories, we survey the motivation, key issues, solutions, and the current state of research. Then, we review the contribution from different research groups, and present the growing trend of CT research. Finally, we recommend directions for future CT research, including: (1) modeling for CT, (2) improving the existing test suite generation algorithm, (3) improving analysis of testing result, (4) exploring the application of CT to different levels of testing and additional types of systems, (5) conducting more empirical studies to fully understand limitations and strengths of CT, and (6) combining CT with other testing techniques.

Test Case Prioritization for Black Box Testing

Test case prioritization is an effective and practical technique that helps to increase the rate of regression fault detection when software evolves. Numerous techniques have been reported in the literature on prioritizing test cases for regression testing. However, existing prioritization techniques implicitly assume that source or binary code is available when regression testing is performed, and therefore cannot be implemented when there is no program source or binary code to be analyzed. In this paper, we presented a new technique for black box regression testing, and we performed an experiment to measure our technique. Our results show that the new technique is helpful to improve the effectiveness of fault detection when performing regression test in black box environment.

Greedy Heuristic Algorithms to Generate Variable Strength Combinatorial Test Suite

Combinatorial testing is a practical software testing approach that has been widely used in practice. Most research and applications of such approach focus on N-way combinatorial testing that provides a minimum coverage of all N-way interactions among factor. However, the strengths of different interactions may not be a fixed integer N, but a variable. Therefore, variable strength combinatorial testing approach is necessary in applications. Existing variable strength combinatorial testing, which allows some interactions have a higher strength than others, has a limitation that such higher-strength interactions must be disjoint. To avoid such a limitation, an improved variable strength combinatorial testing approach, which makes a more sufficient consideration on actual interaction relationship, is proposed in this article. Furthermore, two greedy heuristic algorithms are also proposed to generate combinatorial test suite. Compared to some existing algorithms and tools, the proposed algorithms have advantages on both the execution effectiveness and the optimality of generated test suite. Experimental results can prove such advantages.

The Minimal Failure-Causing Schema of Combinatorial Testing

Combinatorial Testing (CT) involves the design of a small test suite to cover the parameter value combinations so as to detect failures triggered by the interactions among these parameters. To make full use of CT and to extend its advantages, this article first gives a model of CT and then presents a theory of the Minimal Failure-causing Schema (MFS), including the concept of the MFS, proof of its existence, some of its properties, and a method of finding the MFS. Then we propose a methodology for CT based on this MFS theory and the existing research. Our MFS-based methodology emphasizes that CT should work on accurate testing requirements, and has the following advantages: 1) Detect failure to the greatest degree with the least cost. 2) Effectiveness is improved by emphasizing mining of the information in software and making full use of the information gained from test design and execution. 3) Determine the root causes of failures and reveal related faults near the exposed ones. 4) Provide a foundation and model for regression testing and software quality evaluation of CT. A case study is presented to illustrate the MFS-based CT methodology, and an empirical study on a real software developed by us shows that the MFS really exists and the methodology based on MFS can considerably improve CT.

A software debugging method based on pairwise testing

Pairwise testing is one of very practical and effective testing methods for various types of software systems. This paper proposes a novel debugging method based on pairwise testing. By analyzing the test cases and retesting with some complementary test cases, the method can narrow the factors that cause the errors in a very small range. So it can provide a very efficient and valuable guidance for the software testing and debugging.

A novel approach for test suite reduction based on requirement relation contraction

The goal of test suite reduction is to satisfy all testing requirements with the minimum number of test cases. Existing techniques can be applied well on the constructed test suite. However, it is possible and necessary to optimize testing requirements before test case generation. In this paper test suite reduction is solved by testing requirement optimization. A requirement relation graph is proposed to minimize the requirement set by graph contraction. An experiment on specification-based testing is designed and implemented. The empirical studies show that the testing requirements can be optimized by the graph contraction methods effectively.

Automatic test generation for n-way combinatorial testing

n-way combinatorial testing is a specification-based testing criterion, which requires that for a system consisting of a few parameters, every combination of valid values of arbitrary n (n ≥ 2) parameters be covered by at least one test. In this paper, we propose two different tests generation algorithms based on the combinatorial design for the n-way combination testing. We show that the produced tests can cover all the combinations of parameters to the greatest degree with the small quantity. We implemented the automatic test generators based on the algorithms and obtained some valuable empirical results.

A browser compatibility testing method based on combinatorial testing

For ensuring the displaying effects of Web applications, it is important to perform compatibility testing for browsers under the different configurations and it is a hard task to test all. So this paper focused on the improvements for browser compatibility testing. Firstly, we introduced the related work. Then we analysed the functional speciality of the current popular browsers, and provided the preconditions to simplify problems. Next we gave an instance and brought forward the single factor covering method and the pair-wise covering design to gain testing suits. Finally, we introduced the assistant tools we have developed and the future work.

An Effective Iterative Metamorphic Testing Algorithm Based on Program Path Analysis

Metamorphic testing (MT) is very practical and effective for programs with oracle problems, and many researches have been done in this field. In this article, we present a new iterative MT algorithm, APCEMSI, to avoid the blindness of existing MT methods. The test suites generated by APCEMSI satisfy a criterion which is defined upon program path analysis technique. Experiment result shows that APCEMSI could find errors effectively with much fewer test cases.

Comparing Fault-based Testing Strategies of General Boolean Specifications

Testing Boolean specifications in general form (GF) by the IDNF-oriented approaches always results in superabundant cost and missing detection of some faults. This paper proposes GF-oriented approaches to improve them. The experimental results show that the GF-oriented strategies could enhance the fault detection capability and reduce the sizes of test sets.