Nicos Malevris

Automatic Mutation Test Case Generation via Dynamic Symbolic Execution

The automatic test case generation is the principal issue of the software testing activity. Dynamic symbolic execution appears to be a promising approach to this matter as it has been shown to be quite powerful in producing the sought tests. Despite its power, it has only been effectively applied to the entry level criteria of the structural criteria hierarchy such as branch testing. In this paper an extension of this technique is proposed in order to effectively generate test data based on mutation testing. The proposed approach conjoins program transformation and dynamic symbolic execution techniques in order to successfully automate the test generation process. The propositions made in this paper have been incorporated into an automated framework for producing mutation based test cases. Its evaluation on a set of benchmark programs suggests that it is able to produce tests capable of killing most of the non equivalent introduced mutants. The same study also provides some evidence that by employing efficient heuristics it can be possible to perform mutation with reasonable resources.

An Empirical Evaluation of the First and Second Order Mutation Testing Strategies

Various mutation approximation techniques have been proposed in the literature in order to reduce the expenses of mutation. This paper presents results from an empirical study conducted for first and second order mutation testing strategies. Its scope is to evaluate the relative application cost and effectiveness of the different mutation strategies. The application cost was based: on the number of mutants, the equivalent ones and on the number of test cases needed to expose them by each strategy. Each strategys effectiveness was evaluated by its ability to expose a set of seeded faults. The results indicate that on the one hand the first order mutation testing strategies can be in general more effective than the second order ones. On the other hand, the second order strategies can drastically decrease the number of the introduced equivalent mutants, generally forming a valid cost effective alternative to mutation testing.

Mutation based test case generation via a path selection strategy

Context: Generally, mutation analysis has been identified as a powerful testing method. Researchers have shown that its use as a testing criterion exercises quite thoroughly the system under test while it achieves to reveal more faults than standard structural testing criteria. Despite its potential, mutation fails to be adopted in a widespread practical use and its popularity falls significantly short when compared with other structural methods. This can be attributed to the lack of thorough studies dealing with the practical problems introduced by mutation and the assessment of the effort needed when applying it. Such an incident, masks the real cost involved preventing the development of easy and effective to use strategies to circumvent this problem. Objective: In this paper, a path selection strategy for selecting test cases able to effectively kill mutants when performing weak mutation testing is presented and analysed. Method: The testing effort is highly correlated with the number of attempts the tester makes in order to generate adequate test cases. Therefore, a significant influence on the efficiency associated with a test case generation strategy greatly depends on the number of candidate paths selected in order to achieve a predefined coverage goal. The effort can thus be related to the number of infeasible paths encountered during the test case generation process. Results: An experiment, investigating well over 55 million of program paths is conducted based on a strategy that alleviates the effects of infeasible paths. Strategy details, along with a prototype implementation are reported and analysed through the experimental results obtained by its employment to a set of program units. Conclusion: The results obtained suggest that the strategy used can play an important role in making the mutation testing method more appealing and practical.

Evaluating Mutation Testing Alternatives: A Collateral Experiment

Mutation testing while being a successful fault revealing technique for unit testing, it is a rather expensive one for practical use. To bridge these two aspects there is a need to establish approximation techniques able to reduce its expenses while maintaining its effectiveness. In this paper several second order mutation testing strategies are introduced, assessed and compared along with weak mutation against strong. The experimental results suggest that they both constitute viable alternatives for mutation as they establish considerable effort reductions without greatly affecting the test effectiveness. The experimental assessment of weak mutation suggests that it reduces significantly the number of the produced equivalent mutants on the one hand and that the test criterion it provides is not as weak as is thought to be on the other. Finally, an approximation of the number of first order mutants needed to be killed in order to also kill the original mutant set is presented. The findings indicate that only a small portion of a set of mutants needs to be targeted in order to be killed while the rest can be killed collaterally.

Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing

Automating software testing activities can increase the quality and drastically decrease the cost of software development. Toward this direction, various automated test data generation tools have been developed. The majority of existing tools aim at structural testing, while a quite limited number aim at a higher level of testing thoroughness such as mutation. In this paper, an attempt toward automating the generation of mutation-based test cases by utilizing existing automated tools is proposed. This is achieved by reducing the killing mutants’ problem into a covering branches one. To this extent, this paper is motivated by the use of state of the art techniques and tools suitable for covering program branches when performing mutation. Tools and techniques such as symbolic execution, concolic execution, and evolutionary testing can be easily adopted toward automating the test input generation activity for the weak mutation testing criterion by simply utilizing a special form of the mutant schemata technique. The propositions made in this paper integrate three automated tools in order to illustrate and examine the method’s feasibility and effectiveness. The obtained results, based on a set of Java program units, indicate the applicability and effectiveness of the suggested technique. The results advocate that the proposed approach is able to guide existing automating tools in producing test cases according to the weak mutation testing criterion. Additionally, experimental results with the proposed mutation testing regime show that weak mutation is able to speedup the mutant execution time by at least 4.79 times when compared with strong mutation.

Towards automating the generation of mutation tests

Automating software testing activities can increase the quality and drastically decrease the cost of software development. Towards this direction various automated test data generation tools have been developed. The majority of them aim at branch testing, while a quite limited number aim at a higher level of testing thoroughness such as mutation. In this paper an automated framework that makes a joint use of diverse techniques and tools is introduced in the context of automating mutation based test generation. The motivation behind this work is the use of existing techniques and tools such as symbolic execution and evolutionary testing towards automating the test input generation activity according to the weak mutation testing criterion. The proposed framework integrates existing automated tools for branch testing in order to effectively generate mutation test data. To fulfill this suggestion three automated tools are used for illustration purposes and preliminary results are obtained by applying the proposed framework to a set of java program units indicating the applicability and effectiveness of the proposed approach.

The collateral coverage of data flow criteria when branch testing

Abstract When exercising program code with test data in an attempt to satisfy a given testing criterion, there will be a concurrent accrual of coverage in respect of other testing criteria. Knowledge of the extent of such ‘collateral coverage’ can be used to advantage both in providing better estimates of the overheads entailed by the overall testing exercise, and in helping to determine an optimal sequence for the application of a set of testing methods. In this paper, the results deriving from a set of experiments are reported. The aim of the experiments was to investigate the extent of the collateral coverage that is achieved in respect of the data-flow testing criteria when branch testing is undertaken.

Searching and generating test inputs for mutation testing

Mutation testing is usually regarded as an important method towards fault revealing. Despite this advantage, it has proved to be impractical for industrial use because of its expenses. To this extend, automated techniques are needed in order to apply and reduce the method’s demands. Whilst there is much evidence that automated test data generation techniques can effectively automate the testing process, there has been little work on applying them in the context of mutation testing. In this paper, search-based testing is used in order to effectively generate test inputs capable of revealing mutants. To this end, a dynamic execution scheme capable of introducing and guiding the search towards the sought mutants is proposed. Experimentation with the proposed approach reveals its superiority from the previously proposed methods. Additionally, the framework’s feasibility and practicality of producing mutation based test cases are also demonstrated.

A Symbolic Execution Tool Based on the Elimination of Infeasible Paths

Software testing forms a substantial activity of the software development cycle. Although important, it lacks from being automated mainly because of the various undecidable problems that it encounters. To this extend efficient heuristics have been proposed in order to bypass this problem. One such approach, called symbolic execution, is usually used for automating the test data generation activity. In this paper, an automated symbolic execution tool is proposed. The tool employs an efficient path heuristic, integrated with random testing for producing test cases. The tool handles the path explosion and constraint solving problems efficiently. This is achieved by targeting on specific likely to be feasible paths and by using a linear programming approach for the determination of their feasibility. Preliminary results are very encouraging as they show that a high coverage can be achieved within a limited amount of time-effort.

A path generation method for testing LCSAJs that restrains infeasible paths

Abstract The testing of all LCSAJs (Linear Code Sequences and Jumps) in a given code unit has been regarded as a very useful and important method for program testing that goes beyond testing program branches. It involves the generation of a set of paths and execution of these paths with sets of data. Thus it can be regarded as a path testing method too; see Woodward et al. 1 for example. Criteria for selecting such paths have, to date, received scant attention in the literature. In this paper, a suitable path selection strategy which aims at reducing the number of infeasible paths generated is proposed. The methods embodying this strategy are discussed and their application to a set of program units is reported and analysed. The ability of the methods in providing a high coverage for the LCSAJs in a program is also exposed.