Romain Robbes

An extensive comparison of bug prediction approaches

Reliably predicting software defects is one of software engineerings holy grails. Researchers have devised and implemented a plethora of bug prediction approaches varying in terms of accuracy, complexity and the input data they require. However, the absence of an established benchmark makes it hard, if not impossible, to compare approaches.

Evaluating defect prediction approaches: a benchmark and an extensive comparison

Reliably predicting software defects is one of the holy grails of software engineering. Researchers have devised and implemented a plethora of defect/bug prediction approaches varying in terms of accuracy, complexity and the input data they require. However, the absence of an established benchmark makes it hard, if not impossible, to compare approaches. We present a benchmark for defect prediction, in the form of a publicly available dataset consisting of several software systems, and provide an extensive comparison of well-known bug prediction approaches, together with novel approaches we devised. We evaluate the performance of the approaches using different performance indicators: classification of entities as defect-prone or not, ranking of the entities, with and without taking into account the effort to review an entity. We performed three sets of experiments aimed at (1) comparing the approaches across different systems, (2) testing whether the differences in performance are statistically significant, and (3) investigating the stability of approaches across different learners. Our results indicate that, while some approaches perform better than others in a statistically significant manner, external validity in defect prediction is still an open problem, as generalizing results to different contexts/learners proved to be a partially unsuccessful endeavor.

Software systems as cities: a controlled experiment

Software visualization is a popular program comprehension technique used in the context of software maintenance, reverse engineering, and software evolution analysis. While there is a broad range of software visualization approaches, only few have been empirically evaluated. This is detrimental to the acceptance of software visualization in both the academic and the industrial world. We present a controlled experiment for the empirical evaluation of a 3D software visualization approach based on a city metaphor and implemented in a tool called CodeCity. The goal is to provide experimental evidence of the viability of our approach in the context of program comprehension by having subjects perform tasks related to program comprehension. We designed our experiment based on lessons extracted from the current body of research. We conducted the experiment in four locations across three countries, involving 41 participants from both academia and industry. The experiment shows that CodeCity leads to a statistically significant increase in terms of task correctness and decrease in task completion time. We detail the experiment we performed, discuss its results and reflect on the many lessons learned.

Linking e-mails and source code artifacts

E-mails concerning the development issues of a system constitute an important source of information about high-level design decisions, low-level implementation concerns, and the social structure of developers. Establishing links between e-mails and the software artifacts they discuss is a non-trivial problem, due to the inherently informal nature of human communication. Different approaches can be brought into play to tackle this trace-ability issue, but the question of how they can be evaluated remains unaddressed, as there is no recognized benchmark against which they can be compared. In this article we present such a benchmark, which we created through the manual inspection of a statistically significant number of e-mails pertaining to six unrelated software systems. We then use our benchmark to measure the effectiveness of a number of approaches, ranging from lightweight approaches based on regular expressions to full-fledged information retrieval approaches.

A Change-based Approach to Software Evolution

Software evolution research is limited by the amount of information available to researchers: Current version control tools do not store all the information generated by developers. They do not record every intermediate version of the system issued, but only snapshots taken when a developer commits source code into the repository. Additionally, most software evolution analysis tools are not a part of the day-to-day programming activities, because analysis tools are resource intensive and not integrated in development environments. We propose to model development information as change operations that we retrieve directly from the programming environment the developers are using, while they are effecting changes to the system. This accurate and incremental information opens new ways for both developers and researchers to explore and evolve complex systems.

How Program History Can Improve Code Completion

Code completion is a widely used productivity tool. It takes away the burden of remembering and typing the exact names of methods or classes: As a developer starts typing a name, it provides a progressively refined list of candidates matching the name. However, the candidate list always comes in alphabetic order, i.e., the environment is only second-guessing the name based on pattern matching. Finding the correct candidate can be cumbersome or slower than typing the full name. We present an approach to improve code completion with program history. We define a benchmark measuring the accuracy and usefulness of a code completion engine. Further, we use the change history data to also improve the results offered by code completion tools. Finally, we propose an alternative interface for completion tools.

On the Relationship Between Change Coupling and Software Defects

Change coupling is the implicit relationship between two or more software artifacts that have been observed to frequently change together during the evolution of a software system. Researchers have studied this dependency and have observed that it points to design issues such as architectural decay. It is still unknown whether change coupling correlates with a tangible effect of design issues, i.e., software defects.In this paper we analyze the relationship between change coupling and software defects on three large software systems. We investigate whether change coupling correlates with defects, and if the performance of bug prediction models based on software metrics can be improved with change coupling information.

Recovering inter-project dependencies in software ecosystems

In large software systems, knowing the dependencies between modules or components is critical to assess the impact of changes. To recover the dependencies, fact extractors analyze the system as a whole and build the dependency graph, parsing the system down to the statement level. At the level of software ecosystems, which are collections of software projects, the dependencies that need to be recovered reside not only within the individual systems, but also between the libraries, frameworks, and entire software systems that make up the complete ecosystem; scaling issues arise. In this paper we present and evaluate several variants of a lightweight and scalable approach to recover dependencies between the software projects of an ecosystem. We evaluate our recovery algorithms on the Squeak 3.10 Universe, an ecosystem containing more than 200 software projects.

The Small Project Observatory: Visualizing software ecosystems

Software evolution research has focused mostly on analyzing the evolution of single software systems. However, it is rarely the case that a project exists as standalone, independent of others. Rather, projects exist in parallel within larger contexts in companies, research groups or even the open-source communities. We call these contexts software ecosystems. In this paper, we present the Small Project Observatory, a prototype tool which aims to support the analysis of software ecosystems through interactive visualization and exploration. We present a case study of exploring an ecosystem using our tool, we describe the architecture of the tool, and we distill lessons learned during the tool-building experience.

SpyWare: a change-aware development toolset

Our research is driven by the motivation that change must be put in the center, if one wants to understand the complex processes of software evolution. We built a toolset named SpyWare which, using a monitoring plug-in for integrated development environments (IDEs), tracks the changes that a developer performs on a program as they happen. SpyWare stores these first-class changes in a change repository and offers a plethora of productivity-enhancing IDE extensions to exploit the recorded information.