Combining Code and Requirements Coverage with Execution Cost for Test Suite Reduction

Abstract

Test suites tend to become large and complex after software evolution iterations, thus increasing effort and cost to execute regression testing. In this context, test suite reduction approaches could be applied to identify subsets of original test suites that preserve the capability of satisfying testing requirements and revealing faults. In this paper, we propose Multi-Objective test suites REduction (named MORE+): a three-dimension approach for test suite reduction. The first dimension is the structural one and concerns the information on how test cases in a suite exercise the under-test application. The second dimension is functional and concerns how test cases exercise business application requirements. The third dimension is the cost and concerns the time to execute test cases. We define MORE+ as a multi-objective approach that reduces test suites so maximizing their capability in revealing faults according to the three considered dimensions. We have compared MORE+ with seven baseline approaches on 20 Java applications. Results showed, in particular, the effectiveness of MORE+ in reducing test suites with respect to these baselines, i.e., significantly more faults are revealed with test suites reduced by applying MORE+.