Thilo Mende

Revisiting the evaluation of defect prediction models

Defect Prediction Models aim at identifying error-prone parts of a software system as early as possible. Many such models have been proposed, their evaluation, however, is still an open question, as recent publications show. An important aspect often ignored during evaluation is the effort reduction gained by using such models. Models are usually evaluated per module by performance measures used in information retrieval, such as recall, precision, or the area under the ROC curve (AUC). These measures assume that the costs associated with additional quality assurance activities are the same for each module, which is not reasonable in practice. For example, costs for unit testing and code reviews are roughly proportional to the size of a module. In this paper, we investigate this discrepancy using optimal and trivial models. We describe a trivial model that takes only the module size measured in lines of code into account, and compare it to five classification methods. The trivial model performs surprisingly well when evaluated using AUC. However, when an effort-sensitive performance measure is used, it becomes apparent that the trivial model is in fact the worst.

Effort-Aware Defect Prediction Models

Defect Prediction Models aim at identifying error-prone modules of a software system to guide quality assurance activities such as tests or code reviews. Such models have been actively researched for more than a decade, with more than 100 published research papers. However, most of the models proposed so far have assumed that the cost of applying quality assurance activities is the same for each module. In a recent paper, we have shown that this fact can be exploited by a trivial classifier ordering files just by their size: such a classifier performs surprisingly good, at least when effort is ignored during the evaluation. When effort is considered, many classifiers perform not significantly better than a random selection of modules. In this paper, we compare two different strategies to include treatment effort into the prediction process, and evaluate the predictive power of such models. Both models perform significantly better when the evaluation measure takes the effort into account.

Supporting the Grow-and-Prune Model in Software Product Lines Evolution Using Clone Detection

Software product lines (SPL) can be used to create and maintain different variants of software-intensive systems by explicitly managing variability. Often, SPLs are organized as an SPL core, common to all products, upon which product-specific components are built. Following the so called grow-and-prune model, SPLs may be evolved by copy&paste at large scale. New products are created from existing ones and existing products are enhanced with functionalities specific to other products by copying and pasting code between product-specific code. To regain control of this unmanaged growth, such code may be pruned, that is, identified and refactored into core components upon success. This paper describes tool support for the grow-and- prune model in the evolution of software product lines by identifying similar functions which can be moved to the core. These functions are identified in two steps. First, token-based clone detection is used to detect pairs of functions sharing code. Second, Levenshtein distance measures the textual similarity among these functions. Sufficient similarity at function level is then lifted to the architectural level. The approach is evaluated by three case studies, one using an open source email client to simulate the initial creation of an SPL, and two monitoring existing industrial product lines from the embedded domain.

Replication of defect prediction studies: problems, pitfalls and recommendations

Background: The main goal of the PROMISE repository is to enable reproducible, and thus verifiable or refutable research. Over time, plenty of data sets became available, especially for defect prediction problems. Aims: In this study, we investigate possible problems and pitfalls that occur during replication. This information can be used for future replication studies, and serve as a guideline for researchers reporting novel results. Method: We replicate two recent defect prediction studies comparing different data sets and learning algorithms, and report missing information and problems. Results: Even with access to the original data sets, replicating previous studies may not lead to the exact same results. The choice of evaluation procedures, performance measures and presentation has a large influence on the reproducibility. Additionally, we show that trivial and random models can be used to identify overly optimistic evaluation measures. Conclusions: The best way to conduct easily reproducible studies is to share all associated artifacts, e.g. scripts and programs used. When this is not an option, our results can be used to simplify the replication task for other researchers.

Evaluating Defect Prediction Models for a Large Evolving Software System

A plethora of defect prediction models has been proposed and empirically evaluated, often using standard classification performance measures. In this paper, we explore defect prediction models for a large, multi-release software system from the telecommunications domain. A history of roughly 3 years is analyzed to extract process and static code metrics that are used to build several defect prediction models with Random Forests. The performance of the resulting models is comparable to previously published work. Furthermore, we develop a new evaluation measure based on the comparison to an optimal model.

On the Utility of a Defect Prediction Model during HW/SW Integration Testing: A Retrospective Case Study

Testing is an important and cost-intensive part of the software development life cycle. Defect prediction models try to identify error-prone components, so that these can be tested earlier or more in-depth, and thus improve the cost-effectiveness during testing. Such models have been researched extensively, but whether and when they are applicable in practice is still debated. The applicability depends on many factors, and we argue that it cannot be analyzed without a specific scenario in mind. In this paper, we therefore present an analysis of the utility for one case study, based on data collected during the hardware/software integration test of a system from the avionic domain. An analysis of all defects found during this phase reveals that more than half of them are not identifiable by a code-based defect prediction model. We then investigate the predictive performance of different prediction models for the remaining defects. The small ratio of defective instances results in relatively poor performance. Our analysis of the cost-effectiveness then shows that the prediction model is not able to outperform simple models, which order files either randomly or by lines of code. Hence, in our setup, the application of defect prediction models does not offer any advantage in practice.

ArQuE: Architecture-Centric Quality Engineering

The ArQuE project has developed an integrated and comprehensive method that enables goal-oriented,architecture-centric development and strategic quality engineering. The consolidated expertise from applying the ArQuE approach at different industry partners shows the applicability and scalability in the embedded systems domain. The approach leads to reduced maintenance effort and simplified evolution of the respective software systems. The experiences made and knowledge gained in the instantiations of the approach for the industrial systems document the overall project success and give evidence that the overall return on investment of the Argue approach is positive.