David Schuler

Efficient mutation testing by checking invariant violations

Mutation testing measures the adequacy of a test suite by seeding artificial defects (mutations) into a program. If a mutation is not detected by the test suite, this usually means that the test suite is not adequate. However, it may also be that the mutant keeps the programs semantics unchanged-and thus cannot be detected by any test. Such equivalent mutants have to be eliminated manually, which is tedious. We assess the impact of mutations by checking dynamic invariants. In an evaluation of our JAVALANCHE framework on seven industrial-size programs, we found that mutations that violate invariants are significantly more likely to be detectable by a test suite. As a consequence, mutations with impact on invariants should be focused upon when improving test suites. With less than 3% of equivalent mutants, our approach provides an efficient, precise, and fully automatic measure of the adequacy of a test suite.

Javalanche: efficient mutation testing for Java

To assess the quality of a test suite, one can use mutation testing - seeding artificial defects (mutations) into the program and checking whether the test suite finds them. Javalanche is an open source framework for mutation testing Java programs with a special focus on automation, efficiency, and effectiveness. In particular, Javalanche assesses the impact of individual mutations to effectively weed out equivalent mutants; it has been demonstrated to work on programs with up to 100,000 lines of code.

The Impact of Equivalent Mutants

If a mutation is not killed by a test suite, this usuallymeans that the test suite is not adequate. However, itmay also be that the mutant keeps the program’s seman-tics unchanged—and thus cannot be detected by any test.We found such equivalent mutants to be surprisingly com-mon: In an experiment on the JAXEN XPATH query engine,8/20 = 40% of all mutations turned out to be equivalent.Worse, checking the equivalency took us 15 minutes for asingle mutation. Equivalent mutants thus make it impossi-ble to automatically assess test suites by means of mutationtesting. To identify equivalent mutants, we are currently investi-gating the impact of a mutation on the execution: the morea mutation alters the execution, the higher the chance of itbeing non-equivalent. First experiments assessing the im-pact on code coverage are promising.

(Un-)Covering Equivalent Mutants

Mutation testing measures the adequacy of a test suite by seeding artificial defects (mutations) into a program. If a test suite fails to detect a mutation, it may also fail to detect real defects-and hence should be improved. However, there also are mutations which keep the program semantics unchanged and thus cannot be detected by any test suite. Such equivalent mutants must be weeded out manually, which is a tedious task. In this paper, we examine whether changes in coverage can be used to detect non-equivalent mutants: If a mutant changes the coverage of a run, it is more likely to be non-equivalent. Ina sample of 140 manually classified mutations of seven Java programs with 5,000to 100,000 lines of code, we found that: (a) the problem is serious and widespread-about 45% of all undetected mutants turned out to be equivalent;(b) manual classification takes time-about 15 minutes per mutation; (c)coverage is a simple, efficient, and effective means to identify equivalent mutants-with a classification precision of 75% and a recall of 56%; and (d)coverage as an equivalence detector is superior to the state of the art, in particular violations of dynamic invariants. Our detectors have been released as part of the open source Javalanche framework; the data set is publicly available for replication and extension of experiments.

Mining usage expertise from version archives

In software development, there is an increasing need to find and connect developers with relevant expertise. Existing expertise recommendation systems are mostly based on variations of the Line 10 Rule: developers who changed a file most often have the most implementation expertise. In this paper, we introduce the concept of usage expertise, which manifests itself whenever developers are using functionality, e.g., by calling API methods. We present preliminary results for the ECLIPSE project that demonstrate that our technique allows to recommend experts for files with no or little history, identify developers with similar expertise, and measure the usage of API methods.

Covering and Uncovering Equivalent Mutants

Mutation testing measures the adequacy of a test suite by seeding artificial defects (mutations) into a program. If a test suite fails to detect a mutation, it may also fail to detect real defects—and hence should be improved. However, there are also mutations that keep the program semantics unchanged and thus cannot be detected by any test suite. Such equivalent mutants must be weeded out manually, which is a tedious task. In this paper, we examine whether changes in coverage can be used to detect non‐equivalent mutants: If a mutant changes the coverage of a run, it is more likely to be non‐equivalent. In a sample of 140 manually classified mutations of seven Java programs with 5000 to 100 000 lines of code, we found that (i) the problem is serious and widespread—about 45% of all undetected mutants turned out to be equivalent; (ii) manual classification takes time—about 15 min per mutation; (iii) coverage is a simple, efficient and effective means to identify equivalent mutants—with a classification precision of 75% and a recall of 56%; and (iv) coverage as an equivalence detector is superior to the state of the art, in particular violations of dynamic invariants. Our detectors have been released as part of the open‐source JAVALANCHE framework; the data set is publicly available for replication and extension of experiments. Copyright

Assessing Oracle Quality with Checked Coverage

A known problem of traditional coverage metrics is that they do not assess oracle quality -- that is, whether the computation result is actually checked against expectations. In this paper, we introduce the concept of checked coverage -- the dynamic slice of covered statements that actually influence an oracle. Our experiments on seven open-source projects show that checked coverage is a sure indicator for oracle quality -- and even more sensitive than mutation testing, its much more demanding alternative.

Expert recommendation with usage expertise

Global and distributed software development increases the need to find and connect developers with relevant expertise. Existing recommendation systems typically model expertise based on file changes (implementation expertise). While these approaches have shown success, they require a substantial recorded history of development for a project. Previously, we have proposed the concept of usage expertise, i.e., expertise manifested through the act of calling (using) a method. In this paper, we assess the viability of this concept by evaluating expert recommendations for the ASPECTJ and ECLIPSE projects. We find that both usage and implementation expertise have comparable levels of accuracy, which suggests that usage expertise may be used as a substitute measure. We also find a notable overlap of method calls across both projects, which suggests that usage expertise can be leveraged to recommend experts from different projects and thus for projects with little or no history.

Checked coverage: an indicator for oracle quality

A known problem of traditional coverage metrics is that they do not assess oracle quality—that is, whether the computation result is actually checked against expectations. In this paper, we introduce the concept of checked coverage—the dynamic slice of covered statements that actually influence an oracle. Our experiments on seven open‐source projects show that checked coverage is a sure indicator for oracle quality and even more sensitive than mutation testing. Copyright

Breeding High-Impact Mutations

Mutation testing was developed to measure the adequacy of a test suite by seeding artificial bugs (mutations) into a program, and checking whether the test suite detects them. An undetected mutation either indicates a insufficiency in the test suite and provides means for improvement, or it is an equivalent mutation that cannot be detected because it does not change the programs semantics. Impact metrics-that quantify the difference between a run of the original and the mutated version of a program-are one way to detectnon-equivalent mutants. In this paper we present a genetic algorithm that aims to produce a set of mutations that have a high impact, are not detected by the test suite, and at the same time are well spread all over the code. We believe that such a set is useful for improving a test suite, as a high impact of a mutation implies it caused a grave damage, which is not detected by the test suite, and that the mutation is likely to be non-equivalent. First results are promising: The number of undetected mutants in a set of evolved mutants increases from 20 to over 70 percent, the average impact of these undetected mutants grows at the same time by a factor of 5.