Ruilian Zhao

Empirical Study on the Efficiency of Search Based Test Generation for EFSM Models

Experimental work in software testing has generally focused on evaluating the effectiveness and efficiency on various source code programs. However, an important issue of testing efficiency on the model level has not been sufficiently addressed, and hitherto, no empirical studies exist. This paper presents an automated test data generation system for feasible transition paths (FTP) on Extended Finite State Machines (EFSM) models and investigates the statistical properties of testing efficiency using statistical tests for correlation and formalisation according to the test data generated by applying the system on four widely used EFSM models. An important and encouraging finding is a close positive correlation between test generation cost and the number of numerical equal operators in conditions (NNEOC) on a FTP. In addition, as the NNEOC increases, there is a raising correlation between the test generation cost and the length of path with events variables (LPEV) or the number of numerical event variables on a path (NNEV), and NNEV increases linearly with the LPEV. Furthermore, empirical study shows that there is very strong exponential relationship between test generation cost and NNEV or LPEV only when NNEOC is considerable. The results provide a significant guide to predict the testing efficiency for EFSM models.

A Fine-Grained Parallel Multi-objective Test Case Prioritization on GPU

Multi-Objective Evolutionary Algorithms MOEAs have been widely used to address regression test optimization problems, including test case selection and test suite minimization. GPU-based parallel MOEAs are proposed to increase execution efficiency to fulfill the industrial demands. When using binary representation in MOEAs, the fitness evaluation can be transformed a parallel matrix multiplication that is implemented on GPU easily and more efficiently. Such GPU-based parallel MOEAs may achieve higher level of speed-up for test case prioritization because the computation load of fitness evaluation in test case prioritization is more than that in test case selection or test suite minimization. However, the non-applicability of binary representation in the test case prioritization results in the challenge of parallel fitness evaluation on GPU. In this paper, we present a GPU-based parallel fitness evaluation and three novel parallel crossover computation schemes based on ordinal and sequential representations, which form a fine-grained parallel framework for multi-objective test case prioritization. The empirical studies based on eight benchmarks and one open source program show a maximum of 120x speed-up achieved.

Faster mutation-based fault localization with a novel mutation execution strategy

Checking program entities for finding faults is extremely tedious for developers. Fault localization techniques are designed to give a rank list of the probability that program entities incur faults to assist developers to locate faults. Mutation-based fault localization is a recently proposed fault localization approach via mutation analysis. With improved effectiveness, the mutation-based fault localization also brings huge execution cost. To reduce the execution cost of mutation-based fault localization technique, this paper proposes a dynamic mutation execution strategy which includes execution optimizations on both mutants and test cases. As fewer mutants and test cases are executed with the presented strategy, the fault localization process will be faster. The empirical studies show that mutation-based fault localization with the dynamic strategy we proposed, called Faster-MBFL, can reduce mutant-test execution times by 32.4% to 87% with keeping the fault localization accuracy unchanged; further, the additional run time required by utilizing our strategy can be ignored.

Search Based Test Suite Minimization for Fault Detection and Localization: A Co-driven Method

Fault detection and fault localization are two independent but important processes in software testing, in which test cases and associated information are usually used. As there are different goals of the two processes, i.e., detect faults early and locate faults accurately, different information of test cases are required and thus different subsets of test cases are selected to achieve the goals. In general, fault localization is adjacent to fault detection. However, independence of these two processes will restrict the automatic process of software testing. This paper proposes an automatic approach to combining fault detection and fault localization, where a multi-objective optimization for test suite minimization is presented to balance and achieve the both goals. Empirical studies show the proposed method can give consideration to both fault detection and fault localization with a high test suite reduction ratio.

Epistatic Genetic Algorithm for Test Case Prioritization

Search based technologies have been widely used in regression test suite optimization, including test case prioritization, test case selection and test suite minimization, to improve the efficiency and reduce the cost of testing. Unlike test case selection and test suite minimization, the evaluation of test case prioritization is based on the test case execution sequence, in which genetic algorithm is one of the most popular algorithms employed. When permutation encoding is used to represent the execution sequence, the execution of previous test cases can affect the presence of the following test cases, namely epistatic effect. In this paper, the application of epistatic domains theory in genetic algorithms for test case prioritization is analyzed, where Epistatic Test Case Segment is defined. Two associated crossover operators are proposed based on epistasis. The empirical studies show that the proposed two-point crossover operator, E-Ord, outperform the crossover PMX, and can produce higher fitness with a faster convergence.

Boundary Value Testing Using Integrated Circuit Fault Detection Rule

Boundary value testing is a widely used functional testing approach. This paper presents a new boundary value selection approach by applying fault detection rules for integrated circuits. Empirical studies based on Redundant Strapped-Down Inertial Measurement Unit of the 34 program versions and 426 mutants compare the new approach to the current boundary value testing methods. The results show that the approach proposed in this paper is remarkably effective in conquering test blindness, reducing test cost and improving fault coverage.

Epistasis Based ACO for Regression Test Case Prioritization

Metaheuristics that are inspired by natural systems have been widely applied into search-based software engineering. It has been shown that combining knowledge of the application domain with a biological theory for metaheuristics can narrow down the search space and speed up the convergence for metaheuristics based algorithms. This paper introduces Epistatic Test case Segment (ETS) for multiobjective search-based regression Test Case Prioritization (MoTCP), based on epistasis theory that reflects the correlation between genes in evolution process. An ETS-based pheromone update strategy for ant colony optimization (ACO) algorithm is proposed. The experiments with three benchmarks and a real industrial program V8 illustrate that proposed pheromone update strategy guided by the epistasis theory can significant improve the performance of ACO in terms of effectiveness and efficiency for search-based MoTCP.

An optimal mutation execution strategy for cost reduction of mutation-based fault localization

Abstract Identifying faulty program entities has been recognized as one of the most expensive, tedious and time-consuming processes in software debugging activity. Fault localization techniques are designed to assist developers in locating faults by giving a ranking of the probability that program entities incur failures. Mutation-based fault localization (MBFL) is a recently proposed fault localization approach via mutation analysis, which uses the location of mutants to identify the faulty statements. With improved effectiveness, the MBFL also brings huge execution cost. To reduce the execution cost of MBFL technique, this paper proposes a dynamic mutation execution strategy (DMES) to prioritize the execution on both mutants and test cases. As fewer mutants and test cases are executed with DMES, the whole process will become faster and the cost will be decreased. At the same time, it is proved that the fault localization accuracy of MBFL with DMES is the same as that of the original MBFL. Furthermore, this paper discusses the lowest mutation execution cost of MBFL in theory and gives a quick solution to compute the theoretical minimal mutation execution cost of MBFL in the case of keeping fault localization accuracy non-decreasing. The empirical studies show that MBFL with the dynamic strategy proposed in this paper can reduce mutant-test execution cost by 32.4–87% and is close to the theoretical optimal cost. Furthermore, the additional run time required by utilizing our strategy is minimum and can be ignored.

Fault Classification Oriented Spectrum Based Fault Localization

The commonly-used software fault localization approaches mainly utilize test coverage information and test cases execution results to calculate the suspiciousness of each program entity to identify the location of faults, namely spectrum based software fault localization (SBFL). It had been argued that such techniques are not helpful in real debugging process, since the low accuracy of localization and few information provided to programmers. In this paper we consider the combination of statement based fault classification with the SBFL, aiming at increasing accuracy of fault localization and provide additional possible fault information to programmers. An improved technique, fault classification oriented SBFL (FC-SBFL), is proposed in this paper, in which the suspiciousness value is adjusted dynamically based on the probability of statement being faulty. Experimental results on real application programs show that FC-SBFL is more effective than SBFL to locate faults, and studies with Tarantula and OP2 show that more than 75% faults have been identified in a better effectiveness.

Test Data Generation Efficiency Prediction Model for EFSM Based on MGGP

Most software testing researches on Extended Finite State Machine (EFSM) have focused on automatic test sequence and data generation. The analysis of test generation efficiency is still inadequate. In order to investigate the relationship between EFSM test data generation efficiency and its influence factors, according to the feasible transition paths of EFSMs, we build a multi-gene genetic programming (MGGP) predictive model to forecast EFSM test data generation efficiency. Besides, considering standard genetic programming (GP) and neural network are commonly employed in predictive models, we conduct experiments to compare MGGP model with GP model and back propagation (BP) neural network model on their predictive ability. The results show that, MGGP model is able to effectively predict EFSM test data generation efficiency, and compared with GP model and BP model, MGGP model’s predictive ability is stronger. Moreover, the correlation among the influence factors will not affect its predictive performance.