Improving Effectiveness of Spectrum-based Software Fault Localization using Mutation Testing

Abstract

The performance of software fault localization (SFL) is significant to software reliability and the software debugging process. Test suite effectiveness is crucial in improving the performance of spectrum-based fault localization (SBFL). Although code coverage metric (percentage of code merely executed by test cases) is a useful factor to assess the test cases quality but it has a drawback that it does not tell how much code actually checked and therefore this metric alone is not sufficient to measure the quality of test suites. For this reason, in this paper, we make use of mutation testing for the selection of effective test suites to improve the performance of spectrum-based fault localization (SBFL). We use two classic SBFL techniques Ochiai and Jaccard in our experiments to show the impact of test suite effectiveness on the performance of SBFL. We have used an open source mutation testing tool PIT to evaluate and select test cases and the unit testing framework JUnit has been used for writing and executing the test suites. Our experimental results show that the proposed approach can significantly improve the suspiciousness ranking of faulty statements. The results demonstrate that our approach effectively reduces the number of statements examined by an average of approx 50%.