Wendell D. Jones

Accuracy of software quality models over multiple releases

Many evolving mission‐critical systems must have high software reliability. However, it is often difficult to identify fault‐prone modules early enough in a development cycle to guide software enhancement efforts effectively and efficiently. Software quality models can yield timely predictions of membership in the fault‐prone class on a module‐by‐module basis, enabling one to target enhancement techniques. However, it is an open empirical question, “Can a software quality model remain useful over several releases?” Most prior software quality studies have examined only one release of a system, evaluating the model with modules from the same release. We conducted a case study of a large legacy telecommunications system where measurements on one software release were used to build models, and three subsequent releases of the same system were used to evaluate model accuracy. This is a realistic assessment of model accuracy, closely simulating actual use of a software quality model. A module was considered fault‐prone if any of its faults were discovered by customers. These faults are extremely expensive due to consequent loss of service and emergency repair efforts. We found that the model maintained useful accuracy over several releases. These findings are initial empirical evidence that software quality models can remain useful as a system is maintained by a stable software development process.

Application of a usage profile in software quality models

Faults discovered by customers are an important aspect of software quality. The working hypothesis of this paper is that variables derived from an execution profile can be useful in software quality models. An execution profile of a software system consists of the probability of execution of each module during operations. Execution represents opportunities for customers to discover faults. However, an execution profile over an entire customer-base can be difficult to measure directly. Deployment records of past releases can be a valuable source of data for calculating an approximation to the probability of execution. We analyze a metric derived from deployment records which is a practical surrogate for an execution profile in the context of a software quality model. We define usage as the proportion of systems in the field which have a module deployed. We present a case study of a very large legacy telecommunications system. We developed models using a standard statistical technique to predict whether software modules will have any faults discovered by customers on systems in the field. Static software product metrics and usage were independent variables. The significance levels of variables in logistic regression models were analyzed, and models with and without usage as an independent variable were compared. The case study was empirical evidence that usage can be a significant contributor to a software quality model.

Uncertain Classification of Fault-Prone Software Modules

Many development organizations try to minimize faults in software as a means for improving customer satisfaction. Assuring high software quality often entails time-consuming and costly development processes. A software quality model based on software metrics can be used to guide enhancement efforts by predicting which modules are fault-prone. This paper presents statistical techniques to determine which predictions by a classification tree should be considered uncertain. We conducted a case study of a large legacy telecommunications system. One release was the basis for the training dataset, and the subsequent release was the basis for the evaluation dataset. We built a classification tree using the TREEDISC algorithm, which is based on χ2 tests of contingency tables. The model predicted whether a module was likely to have faults discovered by customers, or not, based on software product, process, and execution metrics. We simulated practical use of the model by classifying the modules in the evaluation dataset. The model achieved useful accuracy, in spite of the very small proportion of fault-prone modules in the system. We assessed whether the classes assigned to the leaves were appropriate by statistical tests, and found sizable subsets of modules with uncertain classification. Discovering which modules have uncertain classifications allows sophisticated enhancement strategies to resolve uncertainties. Moreover, TREEDISC is especially well suited to identifying uncertain classifications.

USING CLASSIFICATION TREES FOR SOFTWARE QUALITY MODELS: LESSONS LEARNED

High software reliability is an important attribute of high-assurance systems. Software quality models yield timely predictions of quality indicators on a module-by-module basis, enabling one to focus on finding faults early in development. This paper introduces the Classification And Regression Trees (CART) a algorithm to practitioners in high-assurance systems engineering. This paper presents practical lessons learned on building classification trees for software quality modeling, including an innovative way to control the balance between misclassification rates. A case study of a very large telecommunications system used CART to build software quality models. The models predicted whether or not modules would have faults discovered by customers, based on various sets of software product and process metrics as independent variables. We found that a model based on two software product metrics had comparable accuracy to a model based on forty product and process metrics.

Using classification trees for software quality models: lessons learned

High software reliability is an important attribute of high-assurance systems. Software quality models yield timely predictions of reliability indicators on a module-by-module basis, enabling one to focus on finding faults early in development. This paper introduces the CART (Classification And Regression Trees) algorithm to practitioners in high-assurance systems engineering. This paper presents practical lessons learned in building classification trees for software quality modeling, including an innovative way to control the balance between misclassification rates. A case study of a very large telecommunications system used CART to build software quality models. The models predicted whether or not modules would have faults discovered by customers, based on various sets of software product and process metrics as independent variables. We found that a model based on two software product metrics had an accuracy that was comparable to a model based on 40 product and process metrics.

EMERALD: a case study in enhancing software reliability

As software grows in both size and complexity, reliability becomes a critical issue, particularly in the area of telecommunications. This case study examines a tool and procedures developed at a large telecommunications company for software risk assessment. EMERALD (Enhanced Measurement for Early Risk Assessment of Latent Defects) has been applied at all stages of the software life-cycle to indicate risk of future software problems. The tool provides information used for decision support during design, code inspection, testing and maintenance. EMERALD has been instrumental in Nortels efforts to increase efficiency, control costs, and enhance both product quality and customer satisfaction.

Cost-Benefit Analysis of Software Quality Models

Software reliability is increasingly important in todays marketplace. When traditional software development processes fail to deliver the level of reliability demanded by customers, radical changes in software development processes may be needed. Business process reengineering (BPR) is the popular term for comprehensive redesign of business processes. This paper focuses on the business processes that produce commercial software, and illustrates the central role that models have in implementation of BPR. Software metrics and software-quality modeling technology enable reengineering of software development processes, moving from a static process model to a dynamic one that adapts to the expected quality of each module. We present a method for cost-benefit analysis of BPR of software development processes as a function of model accuracy. The paper defines costs, benefits, profit, and return on investment from both short-term and long-term perspectives. The long-term perspective explicitly accounts for software maintenance efforts. A case study of a very large legacy telecommunications system illustrates the method. The dependent variable of the software-quality model was whether a module will have faults discovered by customers. The independent variables were software product and process metrics. In an example, the costs and benefits of using the model are compared to using random selection of modules for reliability enhancement. Such a cost-benefit analysis clarifies the implications of following model recommendations.

Preparing measurements of legacy software for predicting operational faults

Software quality modeling can be used by a software maintenance project to identify a limited set of software modules that probably need improvement. A models goal is to recommend a set of modules to receive special treatment. The purpose of the paper is to report our experiences modeling software quality with classification trees, including necessary preprocessing of data. We conducted a case study on two releases of a very large legacy telecommunications system. A module was considered fault-prone if any faults were discovered by customers, and not fault-prone otherwise. Software product, process, and execution metrics were the basis for predictors. The TREEDISC algorithm for building classification trees was investigated, because it emphasizes statistical significance. Numeric data, such as software metrics, are not suitable for TREEDISC. Consequently, we transformed measurements into discrete ordinal predictors by grouping. This case study investigated the sensitivity of modeling results to various groupings. We found that robustness, accuracy, and parsimony of the models were influenced by the maximum number of groups. Models based on two sets of candidate predictors had similar sensitivity.

Data Mining of Software Development Databases

Software quality models can predict which modules will have high risk, enabling developers to target enhancement activities to the most problematic modules. However, many find collection of the underlying software product and process metrics a daunting task.Many software development organizations routinely use very large databases for project management, configuration management, and problem reporting which record data on events during development. These large databases can be an unintrusive source of data for software quality modeling. However, multiplied by many releases of a legacy system or a broad product line, the amount of data can overwhelm manual analysis. The field of data mining is developing ways to find valuable bits of information in very large databases. This aptly describes our software quality modeling situation.This paper presents a case study that applied data mining techniques to software quality modeling of a very large legacy telecommunications software systems configuration management and problem reporting databases. The case study illustrates how useful models can be built and applied without interfering with development.

Assessing uncertain predictions of software quality

Many development organizations try to minimize faults in software as a means for improving customer satisfaction. Assuring high software quality often entails time-consuming and costly development processes. A software quality model based on software metrics can be used to guide enhancement efforts by predicting which modules are fault-prone. The paper presents a way to determine which predictions by a classification tree should be considered uncertain. We conducted a case study of a large legacy telecommunications system. One release was the basis for the training data set, and the subsequent release was the basis for the evaluation data set. We built a classification tree using the TREEDISC algorithm, which is based on chi-squared tests of contingency tables. The model predicted whether a module was likely to have faults discovered by customers, or not, based on software product, process, and execution metrics. We simulated practical use of the model by classifying the modules in the evaluation data set. The model achieved useful accuracy, in spite of the very small proportion of fault-prone modules in the system. We assessed whether the classes assigned to the leaves were appropriate by examining the details of the full tree, and found sizable subsets of modules with substantially uncertain classification. Discovering which modules have uncertain classifications allows sophisticated enhancement strategies to resolve uncertainties. Moreover, TREEDISC is especially well suited to identifying uncertain classifications.