István Siket

Empirical validation of object-oriented metrics on open source software for fault prediction

Open source software systems are becoming increasingly important these days. Many companies are investing in open source projects and lots of them are also using such software in their own work. But, because open source software is often developed with a different management style than the industrial ones, the quality and reliability of the code needs to be studied. Hence, the characteristics of the source code of these projects need to be measured to obtain more information about it. This paper describes how we calculated the object-oriented metrics given by Chidamber and Kemerer to illustrate how fault-proneness detection of the source code of the open source Web and e-mail suite called Mozilla can be carried out. We checked the values obtained against the number of bugs found in its bug database - called Bugzilla - using regression and machine learning methods to validate the usefulness of these metrics for fault-proneness prediction. We also compared the metrics of several versions of Mozilla to see how the predicted fault-proneness of the software system changed during its development cycle.

Extracting facts from open source software

Open source software systems are becoming increasingly important these days. Many companies are investing in open source projects and lots of them are also using such software in their own work. But because open source software is often developed without proper management, the quality and reliability of the code may be uncertain. The quality of the code needs to be measured and this can be done only with the help of proper tools. We describe a framework called Columbus with which we calculate the object oriented metrics validated by Basili et al. for illustrating how fault-proneness detection from the open source Web and e-mail suite called Mozilla can be done. We also compare the metrics of several versions of Mozilla to see how the predicted fault-proneness of the software system changed during its development. The Columbus framework has been further developed recently with a compiler wrapping technology that now gives us the possibility of automatically analyzing and extracting information from software systems without modifying any of the source code or makefiles. We also introduce our fact extraction process here to show what logic drives the various tools of the Columbus framework and what steps need to be taken to obtain the desired facts.

Test suite reduction for fault detection and localization: A combined approach

The relation of test suites and actual faults in a software is of critical importance for timely product release. There are two particularily critical properties of test suites to this end: fault localization capability, to characterize the effort of finding the actually defective program elements, and fault detection capability which measures how probable is their manifestation and detection in the first place. While there are well established methods to predict fault detection capability (by measuring code coverage, for instance), characterization of fault localization is an emerging research topic. In this work, we investigate the effect of different test reduction methods on the performance of fault localization and detection techniques. We also provide new combined methods that incorporate both localization and detection aspects. We empirically evaluate the methods first by measuring detection and localization metrics of test suites with various reduction sizes, followed by how reduced test suites perform with actual faults. We experiment with SIR programs traditionally used in fault localization research, and extend the case study with large industrial software systems including GCC and WebKit.

Qualitygate SourceAudit: A tool for assessing the technical quality of software

Software systems are evolving continuously in order to fulfill the ever-changing business needs. This endless modification, however, decreases the internal quality of the system over time. This phenomena is called software erosion, which results in higher development, testing, and operational costs. The SourceAudit tool presented in this paper helps managing the technical risks of software deterioration by allowing imme-diate, automatic, and objective assessment of software quality. By monitoring the high-level technical quality of systems it is possible to immediately perform the necessary steps needed to reduce the effects of software erosion, thus reaching higher maintainability and lower costs in the mid and long-term. The tool measures source code maintainability according to the ISO/IEC 25010 based probabilistic software maintainability model called ColumbusQM. It gives a holistic view on software quality and warns on source code maintainability decline.

Using Version Control History to Follow the Changes of Source Code Elements

Version control systems store the whole history of the source code. Since the source code of a system is organized into files and folders, the history tells us the concerned files and their changed lines only but software engineers are also interested in which source code elements (e.g. classes or methods) are affected by a change. Unfortunately, in most programming languages source code elements do not follow the file system hierarchy, which means that a file can contain more classes and methods and a class can be stored in more files, which makes it difficult to determine the changes of classes by using the changes of files. To solve this problem we developed an algorithm, which is able to follow the changes of the source code elements by using the changes of files and we successfully applied it on the Web Kit open source system.

Towards Building Method Level Maintainability Models Based on Expert Evaluations

The maintainability of software systems is getting more and more attention both from researchers and industrial experts. This is due to its direct impact on development costs and reliability of the software.

Source Meter Sonar Qube Plug-in

The SourceMeter Sonar Qube plug-in is an extension of Sonar Qube, an open-source platform for managing code quality made by Sonar Source S.A, Switzerland. The plug-in extends the built-in Java code analysis engine of Sonar Qube with Front End ARTs high-end Java code analysis engine. Most of Sonar Qubes original analysis results are replaced (including the detected source code duplications), while the range of available analyses is extended with a number of additional metrics and issue detectors. Additionally, the plug-in offers new GUI features on the Sonar Qube dashboard and drilldown views, making the Sonar Qube user experience more comfortable and the work with the tool more productive.

Effect of object oriented refactorings on testability, error proneness and other maintainability attributes

Refactoring (object-oriented) code aims to improve some of the quality attributes of the software system under maintenance. However, as any other changes to a system, refactoring actions may have side-effects too, which need to be taken carefully into account during their implementation. Consequences of refactoring are only moderately discussed in literature. In this work, we emphasize the importance of documenting such consequences, by reviewing relevant literature and giving our views on the topic. We do this in three steps: first, we investigate how high level quality attributes (including testability and other maintainability aspects based on the ISO 9126 standard and fault proneness) can be estimated based on measurable metrics in the code, like complexity, size, and coupling. In the second step, we examine what effect of individual refactoring techniques can have on such metrics. Finally, we combine these findings to get a view on how refactoring can influence high level quality characteristics. With this work, we want to foster discussion in this important topic, rather than giving a solution to the problem, as it requires a vast amount of further research by the testing and software quality communities.

Prediction models for performance, power, and energy efficiency of software executed on heterogeneous hardware

Heterogeneous computer environments are becoming commonplace so it is increasingly important to understand how and where we could execute a given algorithm the most efficiently. In this paper we propose a methodology that uses both static source code metrics, and dynamic execution time, power, and energy measurements to build gain ratio prediction models. These models are trained on special benchmarks that have both sequential and parallel implementations and can be executed on various computing elements, e.g., on CPUs, GPUs, or FPGAs. After they are built, however, they can be applied to a new system using only the system’s static source code metrics which are much more easily computable than any dynamic measurement. We found that while estimating a continuous gain ratio is a much harder problem, we could predict the gain category (e.g., “slight improvement” or “large deterioration”) of porting to a specific configuration significantly more accurately than a random choice, using static information alone. We also conclude based on our benchmarks that parallelized implementations are less maintainable, thereby supporting the need for automatic transformations.

A Public Unified Bug Dataset for Java

Background: Bug datasets have been created and used by many researchers to build bug prediction models. Aims: In this work we collected existing public bug datasets and unified their contents. Method: We considered 5 public datasets which adhered to all of our criteria. We also downloaded the corresponding source code for each system in the datasets and performed their source code analysis to obtain a common set of source code metrics. This way we produced a unified bug dataset at class and file level that is suitable for further research (e.g. to be used in the building of new bug prediction models). Furthermore, we compared the metric definitions and values of the different bug datasets. Results: We found that (i) the same metric abbreviation can have different definitions or metrics calculated in the same way can have different names, (ii) in some cases different tools give different values even if the metric definitions coincide because (iii) one tool works on source code while the other calculates metrics on bytecode, or (iv) in several cases the downloaded source code contained more files which influenced the afferent metric values significantly. Conclusions: Apart from all these imprecisions, we think that having a common metric set can help in building better bug prediction models and deducing more general conclusions. We made the unified dataset publicly available for everyone. By using a public dataset as an input for different bug prediction related investigations, researchers can make their studies reproducible, thus able to be validated and verified.