Hongmin Lu

The ability of object-oriented metrics to predict change-proneness: a meta-analysis

Many studies have investigated the relationships between object-oriented (OO) metrics and change-proneness and conclude that OO metrics are able to predict the extent of change of a class across the versions of a system. However, there is a need to re-examine this subject for two reasons. First, most studies only analyze a small number of OO metrics and, therefore, it is not clear whether this conclusion is applicable to most, if not all, OO metrics. Second, most studies only uses relatively few systems to investigate the relationships between OO metrics and change-proneness and, therefore, it is not clear whether this conclusion can be generalized to other systems. In this paper, based on 102 Java systems, we employ statistical meta-analysis techniques to investigate the ability of 62 OO metrics to predict change-proneness. In our context, a class which is changed in the next version of a system is called change-prone and not change-prone otherwise. The investigated OO metrics cover four metric dimensions, including 7 size metrics, 18 cohesion metrics, 20 coupling metrics, and 17 inheritance metrics. We use AUC (the area under a relative operating characteristic, ROC) to evaluate the predictive effectiveness of OO metrics. For each OO metric, we first compute AUCs and the corresponding variances for individual systems. Then, we employ a random-effect model to compute the average AUC over all systems. Finally, we perform a sensitivity analysis to investigate whether the AUC result from the random-effect model is robust to the data selection bias in this study. Our results from random-effect models reveal that: (1) size metrics exhibit moderate or almost moderate ability in discriminating between change-prone and not change-prone classes; (2) coupling and cohesion metrics generally have a lower predictive ability compared to size metrics; and (3) inheritance metrics have a poor ability to discriminate between change-prone and not change-prone classes. Our results from sensitivity analyses show that these conclusions reached are not substantially influenced by the data selection bias.

An improved accuracy measure for rough sets

In rough set theory, the accuracy measure is an important numerical characterization that quantifies the imprecision of a rough set caused by its boundary region. However, it does not take into consideration the granularity of the partition induced by an equivalence relation. Therefore, the imprecision of a rough set is not well characterized by the traditional accuracy measure. This paper first analyzes the limitations of the accuracy measure, and then improves it by information theory. Finally, four theorems show that the improved measure has some good properties and hence is more suitable to measure the imprecision of rough sets.

Test Case Prioritization for Web Service Regression Testing

Regression testing is necessary to assure the quality of service-oriented business applications in their evolutions. However, because of the constraint of testing resource, entire test suite may not run as a result. Therefore, test case prioritization technique is required to increase the efficiency of Web service application regression testing. In this paper, we propose a dependence analysis based test case prioritization technique. First, we analyze the dependence relationship using control and data flow information in an orchestration language: WS-BPEL. Then we construct a weighted graph and do impact analysis to identify modification-affected elements. After that, we prioritize test cases according to covering more modification-affected elements with the highest weight. Finally we conduct a case study to illustrate the applicability of our method.

DMC: a more precise cohesion measure for classes

Abstract In object-oriented systems, a single class consists of attributes and methods and its cohesion denotes the degree of relatedness among these elements. To quantify the cohesiveness of a class, a large number of measures that only depict method–attribute reference relationships have been proposed in last decade. However, the flow-dependence relationships among attributes, the direction of method–attribute references, and the potential dependence relationships among the elements in the class are ignored. To address this problem, this paper first depicts four types of explicit dependence relationships and uses a class element dependence graph to represent all dependencies among the elements in a class. Then, a dependence matrix that reflects the degree of direct dependence and indirect dependence among the elements in a class is computed. Finally, a more precise cohesion measure for classes is proposed.

Effort-aware just-in-time defect prediction: simple unsupervised models could be better than supervised models

Unsupervised models do not require the defect data to build the prediction models and hence incur a low building cost and gain a wide application range. Consequently, it would be more desirable for practitioners to apply unsupervised models in effort-aware just-in-time (JIT) defect prediction if they can predict defect-inducing changes well. However, little is currently known on their prediction effectiveness in this context. We aim to investigate the predictive power of simple unsupervised models in effort-aware JIT defect prediction, especially compared with the state-of-the-art supervised models in the recent literature. We first use the most commonly used change metrics to build simple unsupervised models. Then, we compare these unsupervised models with the state-of-the-art supervised models under cross-validation, time-wise-cross-validation, and across-project prediction settings to determine whether they are of practical value. The experimental results, from open-source software systems, show that many simple unsupervised models perform better than the state-of-the-art supervised models in effort-aware JIT defect prediction.

Are Slice-Based Cohesion Metrics Actually Useful in Effort-Aware Post-Release Fault-Proneness Prediction? An Empirical Study

Background. Slice-based cohesion metrics leverage program slices with respect to the output variables of a module to quantify the strength of functional relatedness of the elements within the module. Although slice-based cohesion metrics have been proposed for many years, few empirical studies have been conducted to examine their actual usefulness in predicting fault-proneness. Objective. We aim to provide an in-depth understanding of the ability of slice-based cohesion metrics in effort-aware post-release fault-proneness prediction, i.e. their effectiveness in helping practitioners find post-release faults when taking into account the effort needed to test or inspect the code. Method. We use the most commonly used code and process metrics, including size, structural complexity, Halsteads software science, and code churn metrics, as the baseline metrics. First, we employ principal component analysis to analyze the relationships between slice-based cohesion metrics and the baseline metrics. Then, we use univariate prediction models to investigate the correlations between slice-based cohesion metrics and post-release fault-proneness. Finally, we build multivariate prediction models to examine the effectiveness of slice-based cohesion metrics in effort-aware post-release fault-proneness prediction when used alone or used together with the baseline code and process metrics. Results. Based on open-source software systems, our results show that: 1) slice-based cohesion metrics are not redundant with respect to the baseline code and process metrics; 2) most slice-based cohesion metrics are significantly negatively related to post-release fault-proneness; 3) slice-based cohesion metrics in general do not outperform the baseline metrics when predicting post-release fault-proneness; and 4) when used with the baseline metrics together, however, slice-based cohesion metrics can produce a statistically significant and practically important improvement of the effectiveness in effort-aware post-release fault-proneness prediction. Conclusion. Slice-based cohesion metrics are complementary to the most commonly used code and process metrics and are of practical value in the context of effort-aware post-release fault-proneness prediction.

DRC: a dependence relationships based cohesion measure for classes

A large number of cohesion measures based on method-attribute references have been proposed. However, virtually no attention has been paid to the abstract representation that objectively depicts the relationships among the members of a class. Specially, the flow dependence relationship among attributes, the indirect and potential dependence relationships among class members, and the direction of method-attribute references are ignored. To address this problem, we first identifies four types of basic dependence relationships and uses a class member dependence graph to represent all dependences among the members of a class. Then, a dependence relationships based measure for measuring the class cohesiveness is proposed. Finally, we compare our class cohesion measure with typical cohesion measures.

An empirical analysis of package-modularization metrics: Implications for software fault-proneness

Abstract Context In a large object-oriented software system, packages play the role of modules which group related classes together to provide well-identified services to the rest of the system. In this context, it is widely believed that modularization has a large influence on the quality of packages. Recently, Sarkar, Kak, and Rama proposed a set of new metrics to characterize the modularization quality of packages from important perspectives such as inter-module call traffic, state access violations, fragile base-class design, programming to interface, and plugin pollution. These package-modularization metrics are quite different from traditional package-level metrics, which measure software quality mainly from size, extensibility, responsibility, independence, abstractness, and instability perspectives. As such, it is expected that these package-modularization metrics should be useful predictors for fault-proneness. However, little is currently known on their actual usefulness for fault-proneness prediction, especially compared with traditional package-level metrics. Objective In this paper, we examine the role of these new package-modularization metrics for determining software fault-proneness in object-oriented systems. Method We first use principal component analysis to analyze whether these new package-modularization metrics capture additional information compared with traditional package-level metrics. Second, we employ univariate prediction models to investigate how these new package-modularization metrics are related to fault-proneness. Finally, we build multivariate prediction models to examine the ability of these new package-modularization metrics for predicting fault-prone packages. Results Our results, based on six open-source object-oriented software systems, show that: (1) these new package-modularization metrics provide new and complementary views of software complexity compared with traditional package-level metrics; (2) most of these new package-modularization metrics have a significant association with fault-proneness in an expected direction; and (3) these new package-modularization metrics can substantially improve the effectiveness of fault-proneness prediction when used with traditional package-level metrics together. Conclusions The package-modularization metrics proposed by Sarkar, Kak, and Rama are useful for practitioners to develop quality software systems.

Predicting Vulnerable Components via Text Mining or Software Metrics? An Effort-Aware Perspective

In order to identify vulnerable software components, developers can take software metrics as predictors or use text mining techniques to build vulnerability prediction models. A recent study reported that text mining based models have higher recall than software metrics based models. However, this conclusion was drawn without considering the sizes of individual components which affects the code inspection effort to determine whether a component is vulnerable. In this paper, we investigate the predictive power of these two kinds of prediction models in the context of effort-aware vulnerability prediction. To this end, we use the same data sets, containing 223 vulnerabilities found in three web applications, to build vulnerability prediction models. The experimental results show that: (1) in the context of effort-aware ranking scenario, text mining based models only slightly outperform software metrics based models, (2) in the context of effort-aware classification scenario, text mining based models perform similarly to software metrics based models in most cases, and (3) most of the effect sizes (i.e. the magnitude of the differences) between these two kinds of models are trivial. These results suggest that, from the viewpoint of practical application, software metrics based models are comparable to text mining based models. Therefore, for developers, software metrics based models are practical choices for vulnerability prediction, as the cost to build and apply these models is much lower.

The Influence of Developer Quality on Software Fault-Proneness Prediction

Previous studies have shown that process metrics are useful for building fault-proneness prediction models. In particular, it has been found that those process metrics incorporating developer experience (defined as the percentage of the code a developer contributes) exhibit a good ability to predict fault-proneness. However, developer quality, which we strongly believe should have a great influence on software quality, is surprisingly ignored. In this paper, we first quantify the quality of a developer via the percentage of history bug-introduce commits over all his/her commits during the development process. Then, we leverage developer quality information to develop eight file quality metrics. Finally, we empirically study the usefulness of these eight file quality metrics for fault-proneness prediction. Based on eight open source software systems, our experiment results show that: 1) these proposed file quality metrics capture additional information compared with existing process metrics, 2) almost all the proposed file quality metrics have a significant association with fault-proneness in an expected direction, and 3) the proposed file quality metrics can in general improve the effectiveness of fault-proneness prediction models when together used with existing process metrics. These results suggest that developer quality has a strong influence on software quality and should be taken into account when predicting software fault-proneness.