John P. Hudepohl

On the value of static analysis for fault detection in software

No single software fault-detection technique is capable of addressing all fault-detection concerns. Similarly to software reviews and testing, static analysis tools (or automated static analysis) can be used to remove defects prior to release of a software product. To determine to what extent automated static analysis can help in the economic production of a high-quality product, we have analyzed static analysis faults and test and customer-reported failures for three large-scale industrial software systems developed at Nortel Networks. The data indicate that automated static analysis is an affordable means of software fault detection. Using the orthogonal defect classification scheme, we found that automated static analysis is effective at identifying assignment and checking faults, allowing the later software production phases to focus on more complex, functional, and algorithmic faults. A majority of the defects found by automated static analysis appear to be produced by a few key types of programmer errors and some of these types have the potential to cause security vulnerabilities. Statistical analysis results indicate the number of automated static analysis faults can be effective for identifying problem modules. Our results indicate static analysis tools are complementary to other fault-detection techniques for the economic production of a high-quality software product.

Assessing the benefits of incorporating function clone detection in a development process

The aim of the experiment presented in this paper is to present an insight into the evaluation of the potential benefits of introducing a function clone detection technology in an industrial software development process. To take advantage of function clone detection, two modifications to the software development process are presented. Our experiment consists of evaluating the impact that these proposed changes would have had on a specific software system if they had been applied over a 3 year period (involving 10000 person-months), where 6 subsequent versions of the software under study were released. The software under study is a large telecommunication system. In total 89 million lines of code have been analyzed. A first result showed that, against our expectations, a significant number of clones are being removed from the system over time. However, this effort is insufficient to prevent the growth of the overall number of clones in the system. In this context the first process change would have added value. We have also found that the second process change would have provided programmers with a significant number of opportunities for correcting problems before customers experienced them. This result shows a potential for improving the software system quality and customer satisfaction

Application of neural networks to software quality modeling of a very large telecommunications system

Society relies on telecommunications to such an extent that telecommunications software must have high reliability. Enhanced measurement for early risk assessment of latent defects (EMERALD) is a joint project of Nortel and Bell Canada for improving the reliability of telecommunications software products. This paper reports a case study of neural-network modeling techniques developed for the EMERALD system. The resulting neural network is currently in the prototype testing phase at Nortel. Neural-network models can be used to identify fault-prone modules for extra attention early in development, and thus reduce the risk of operational problems with those modules. We modeled a subset of modules representing over seven million lines of code from a very large telecommunications software system. The set consisted of those modules reused with changes from the previous release. The dependent variable was membership in the class of fault-prone modules. The independent variables were principal components of nine measures of software design attributes. We compared the neural-network model with a nonparametric discriminant model and found the neural-network model had better predictive accuracy.

EMERALD: software metrics and models on the desktop

As software becomes more and more sophisticated, industry has begun to place a premium on software reliability. The telecommunications industry is no exception. Consequently software reliability is a strategic business weapon in an increasingly competitive marketplace. In response to these concerns, BNR, Nortel, and Bell Canada developed the Enhanced Measurement for Early Risk Assessment of Latent Defects (Emerald), a decision support system designed to improve telecommunications software reliability. Emerald efficiently integrates software measurements, quality models, and delivery of results to the desktop of software developers. We have found that Emerald not only improves software reliability, but also facilitates the accurate correction of field problems. Our experiences developing Emerald have also taught us some valuable lessons about the implementation and adoption of this type of software tool.

Preliminary results on using static analysis tools for software inspection

Software inspection has been shown to be an effective defect removal practice, leading to higher quality software with lower field failures. Automated software inspection tools are emerging for identifying a subset of defects in a less labor-intensive manner than manual inspection. This paper investigates the use of automated inspection for a large-scale industrial software system at Nortel Networks. We propose and utilize a defect classification scheme for enumerating the types of defects that can be identified by automated inspections. Additionally, we demonstrate that automated code inspection faults can be used as efficient predictors of field failures and are effective for identifying fault-prone modules.

Accuracy of software quality models over multiple releases

Many evolving mission‐critical systems must have high software reliability. However, it is often difficult to identify fault‐prone modules early enough in a development cycle to guide software enhancement efforts effectively and efficiently. Software quality models can yield timely predictions of membership in the fault‐prone class on a module‐by‐module basis, enabling one to target enhancement techniques. However, it is an open empirical question, “Can a software quality model remain useful over several releases?” Most prior software quality studies have examined only one release of a system, evaluating the model with modules from the same release. We conducted a case study of a large legacy telecommunications system where measurements on one software release were used to build models, and three subsequent releases of the same system were used to evaluate model accuracy. This is a realistic assessment of model accuracy, closely simulating actual use of a software quality model. A module was considered fault‐prone if any of its faults were discovered by customers. These faults are extremely expensive due to consequent loss of service and emergency repair efforts. We found that the model maintained useful accuracy over several releases. These findings are initial empirical evidence that software quality models can remain useful as a system is maintained by a stable software development process.

Integrating metrics and models for software risk assessment

Enhanced Measurement for Early Risk Assessment of Latent Defects (EMERALD) is a decision support system for assessing reliability risk. It is used by software developers and managers to improve telecommunications software service quality as perceived by the customer and the end user. Risk models are based on static characteristics of source code. This paper shows how a system such as EMERALD can enhance software development, testing, and maintenance by integration of: a software quality improvement strategy; measurements and models; and delivery of results to the desktop of developers in a timely manner. This paper also summarizes empirical experiments with EMERALDs models using data from large industrial telecommunications software systems. EMERALD has been applied to a very large system with over 12 million lines of source code within procedures. Experience and lessons learned are also discussed.

Detection of fault-prone program modules in a very large telecommunications system

Telecommunications software is known for its high reliability. Society has become so accustomed to reliable telecommunications, that failures can cause major disruptions. This is an experience report on application of discriminant analysis based on 20 static software product metrics, to identify fault prone modules in a large telecommunications system, so that reliability may be improved. We analyzed a sample of 2000 modules representing about 1.3 million lines of code, drawn from a much larger system. Sample modules were randomly divided into a fit data set and a test data set. We simulated utilization of the fitted model with the test data set. We found that identifying new modules and changed modules mere significant components of the discriminant model, and improved its performance. The results demonstrate that data on module reuse is a valuable input to quality models and that discriminant analysis can be a useful tool in early identification of fault prone software modules in large telecommunications systems. Model results could be used to identify those modules that would probably benefit from extra attention, and thus, reduce the risk of unexpected problems with those modules.

Evolutionary neural networks: a robust approach to software reliability problems

In this empirical study, from a large data set of software metrics for program modules, thirty distinct partitions into training and validation sets are automatically generated with approximately equal distributions of fault prone and not fault prone modules. Thirty classification models are built for each of the two approaches considered-discriminant analysis and the evolutionary neural network (ENN) approach-and their performances on corresponding data sets are compared. The lower error proportions for ENNs on fault prone, not fault prone, and overall classification were found to be statistically significant. The robustness of ENNs follows from their superior performance on the range of data configurations used. It is suggested that ENNs can be effective in other software reliability problem domains, where they have been largely ignored.

Using the genetic algorithm to build optimal neural networks for fault-prone module detection

The genetic algorithm is applied to developing optimal or near optimal backpropagation neural networks for fault-prone/not-fault-prone classification of software modules. The algorithm considers each network in a population of neural networks as a potential solution to the optimal classification problem. Variables governing the learning and other parameters and network architecture are represented as substrings (genes) in a machine-level bit string (chromosome). When the population undergoes simulated evolution using genetic operators-selection based on a fitness function, crossover, and mutation-the average performance increases in successive generations. We found that, on the same data, compared with the best manually developed networks, evolved networks produced improved classifications in considerably less time, with no human effort, and with greater confidence in their optimality or near optimality. Strategies for devising a fitness function specific to the problem are explored and discussed.