Piotr Tomaszewski

Statistical models vs. expert estimation for fault prediction in modified code - an industrial case study

Statistical fault prediction models and expert estimations are two popular methods for deciding where to focus the fault detection efforts when the fault detection budget is limited. In this paper, we present a study in which we empirically compare the accuracy of fault prediction offered by statistical prediction models with the accuracy of expert estimations. The study is performed in an industrial setting. We invited eleven experts that are involved in the development of two large telecommunication systems. Our statistical prediction models are built on historical data describing one release of one of those systems. We compare the performance of these statistical fault prediction models with the performance of our experts when predicting faults in the latest releases of both systems. We show that the statistical methods clearly outperform the expert estimations. As the main reason for the superiority of the statistical models we see their ability to cope with large datasets. This makes it possible for statistical models to perform reliable predictions for all components in the system. This also enables prediction at a more fine-grain level, e.g., at the class instead of at the component level. We show that such a prediction is better both from the theoretical and from the practical perspective.

Software development productivity on a new platform: an industrial case study

The high non-functional requirements on mobile telecommunication applications call for new solutions. An example of such a solution can be a software platform that provides high performance and availability. The introduction of such a platform may, however, affect the development productivity. In this study, we present experiences from research carried out at Ericsson. The purpose of the research was productivity improvement and assessment when using the new platform. In this study, we quantify and evaluate the current productivity level by comparing it with UNIX development. The comparison is based on two large, commercially, available systems. We reveal a factor of four differences in productivity. Later, we decompose the problem into two issues: code writing speed and average amount of code necessary to deliver a certain functionality. We assess the impact of both these issues. We describe the nature of the problem by identifying factors that affect productivity and estimating their importance. To the issues identified we suggest a number of remedies. The main methods used in the study are interviews and historical data research.

The accuracy of fault prediction in modified code - statistical model vs. expert estimation

Fault prediction models still seem to be more popular in academia than in industry. In industry, expert estimations of fault proneness are the most popular methods of deciding where to focus the fault detection efforts. In this paper, we present a study in which we empirically evaluate the accuracy of fault prediction offered by statistical models as compared to expert estimations. The study is industry based. It involves a large telecommunication system and experts that were involved in the development of this system. Expert estimations are compared to simple prediction models built on another large system, also from the telecommunication domain. We show that the statistical methods clearly outperform the expert estimations. As the main reason for the superiority of the statistical models we see their ability to cope with large datasets, which results in their ability to perform reliable predictions for larger number of components in the system, as well as the ability to perform prediction at a more fine-grain level, e.g., at the class instead of at the component level

A Method for an Accurate Early Prediction of Faults in Modified Classes

In this paper we suggest and evaluate a method for predicting fault densities in modified classes early in the development process, i.e., before the modifications are implemented. We start by establishing methods that according to literature are considered the best for predicting fault densities of modified classes. We find that these methods can not be used until the system is implemented. We suggest our own methods, which are based on the same concept as the methods suggested in the literature, with the difference that our methods are applicable before the coding has started. We evaluate our methods using three large telecommunication systems produced by Ericsson. We find that our methods provide predictions that are of similar quality to the predictions based on metrics available after the code is implemented. Our predictions are, however, available much earlier in the development process. Therefore, they enable better planning of efficient fault prevention and fault detection activities

The increase of productivity over time : an industrial case study

Abstract Introducing new and specialized technology is often seen as a way of meeting increasing non-functional requirements. An example of such a technology is a software platform that provides high performance and availability. The novelty of such a platform and lack of related experience and competence among the staff may affect initial development productivity. The competence problems should disappear with time. In this paper, we present a study, which we conducted at Ericsson. The purpose of the study was to assess the impact of experience and maturity on productivity in software development on the specialized platform. We quantify the impact by comparing productivity of two projects. One represents an initial development stage while the other represents a subsequent and thus more matured development stage. Both projects resulted in large commercial products. We reveal a factor of four difference in productivity. The difference was caused by a higher code delivery rate and a lower number of code lines per functionality in the latter project. We assess the impact of both these issues on productivity and explain their nature. Based on our findings, we suggest a number of improvement suggestions and guidelines for the process of introducing a new technology.

Improving fault detection in modified code: a study from the telecommunication industry

Many software systems are developed in a number of consecutive releases. In each release not only new code is added but also existing code is often modified. In this study we show that the modified code can be an important source of faults. Faults are widely recognized as one of the major cost drivers in software projects. Therefore, we look for methods that improve the fault detection in the modified code. We propose and evaluate a number of prediction models that increase the efficiency of fault detection. To build and evaluate our models we use data collected from two large telecommunication systems produced by Ericsson. We evaluate the performance of our models by applying them both to a different release of the system than the one they are built on and to a different system. The performance of our models is compared to the performance of the theoretical best model, a simple model based on size, as well as to analyzing the code in a random order (not using any model). We find that the use of our models provides a significant improvement over not using any model at all and over using a simple model based on the class size. The gain offered by our models corresponds to 38–57% of the theoretical maximum gain.

Comparing the fault-proneness of new and modified code: an industrial case study

Faults are considered as one of the important factors affecting the cost of software development projects. To be able to efficiently handle faults, we must increase our understanding of the factors that make the code fault-prone. A majority of large software systems evolve during their lifetime. In each new release of the system the functionality can be added by writing new classes or/and by modifying already existing ones. In this study we compared the fault-proneness of new and modified classes in such systems. Our study is based on two releases of two large telecommunication systems developed at Ericsson. The major finding of the study is that the risk of introducing faults (the number of faults in the class /the number of new or modified lines of code in the class) is 20 to 40 times as high in modified classes compared to new ones. In the systems which we analyzed a small modification (a few percent) of the class resulted in as many faults as we would expect when the same class was written from scratch. Previous research on this relationship does not appear to exist. Partly in conflict with related research, we found that there is no statistically significant difference between the average number of faults in modified and new classes, and that the average fault-densities (the number of faults/the size of the entire class) in new and in modified classes are very similar. Finally, we also suggest how our findings can be used in practice.

Increasing the efficiency of fault detection in modified code

Many software systems are developed in a number of consecutive releases. Each new release does not only add new code but also modifies already existing one. In this study we have shown that the modified code can be an important source of faults. The faults are widely recognized as one of the major cost drivers in software projects. Therefore we look for methods of improving fault detection in the modified code. We suggest and evaluate a number of prediction models for increasing the efficiency of fault detection. We evaluate them against the theoretical best model, a simple model based on size, as well as against analyzing the code in a random order (not using any model). We find that using our models provides a significant improvement both over not using any model at all and using the simple model based on the class size. The gain offered by the models corresponds to 30% to 60% of the theoretical maximum.