Michaela Greiler

Characteristics of useful code reviews: an empirical study at Microsoft

Over the past decade, both open source and commercial software projects have adopted contemporary peer code review practices as a quality control mechanism. Prior research has shown that developers spend a large amount of time and effort performing code reviews. Therefore, identifying factors that lead to useful code reviews can benefit projects by increasing code review effectiveness and quality. In a three-stage mixed research study, we qualitatively investigated what aspects of code reviews make them useful to developers, used our findings to build and verify a classification model that can distinguish between useful and not useful code review feedback, and finally we used this classifier to classify review comments enabling us to empirically investigate factors that lead to more effective code review feedback. In total, we analyzed 1.5 millions review comments from five Microsoft projects and uncovered many factors that affect the usefulness of review feedback. For example, we found that the proportion of useful comments made by a reviewer increases dramatically in the first year that he or she is at Microsoft but tends to plateau afterwards. In contrast, we found that the more files that are in a change, the lower the proportion of comments in the code review that will be of value to the author of the change. Based on our findings, we provide recommendations for practitioners to improve effectiveness of code reviews.

The art of testing less without sacrificing quality

Testing is a key element of software development processes for the management and assessment of product quality. In most development environments, the software engineers are responsible for ensuring the functional correctness of code. However, for large complex software products, there is an additional need to check that changes do not negatively impact other parts of the software and they comply with system constraints such as backward compatibility, performance, security etc. Ensuring these system constraints may require complex verification infrastructure and test procedures. Although such tests are time consuming and expensive and rarely find defects they act as an insurance process to ensure the software is compliant. However, long lasting tests increasingly conflict with strategic aims to shorten release cycles. To decrease production costs and to improve development agility, we created a generic test selection strategy called THEO that accelerates test processes without sacrificing product quality. THEO is based on a cost model, which dynamically skips tests when the expected cost of running the test exceeds the expected cost of removing it. We replayed past development periods of three major Microsoft products resulting in a reduction of 50% of test executions, saving millions of dollars per year, while maintaining product quality.

Code ownership and software quality: a replication study

In a traditional sense, ownership determines rights and duties in regard to an object, for example a property. The owner of source code usually refers to the person that invented the code. However, larger code artifacts, such as files, are usually composed by multiple engineers contributing to the entity over time through a series of changes. Frequently, the person with the highest contribution, e.g. The most number of code changes, is defined as the code owner and takes responsibility for it. Thus, code ownership relates to the knowledge engineers have about code. Lacking responsibility and knowledge about code can reduce code quality. In an earlier study, Bird et al. [1] showed that Windows binaries that lacked clear code ownership were more likely to be defect prone. However recommendations for large artifacts such as binaries are usually not actionable. E.g. Changing the concept of binaries and refactoring them to ensure strong ownership would violate system architecture principles. A recent replication study by Foucault et al. [2] on open source software replicate the original results and lead to doubts about the general concept of ownership impacting code quality. In this paper, we replicated and extended the previous two ownership studies [1, 2] and reflect on their findings. Further, we define several new ownership metrics to investigate the dependency between ownership and code quality on file and directory level for 4 major Microsoft products. The results confirm the original findings by Bird et al. [1] that code ownership correlates with code quality. Using new and refined code ownership metrics we were able to classify source files that contained at least one bug with a median precision of 0.74 and a median recall of 0.38. On directory level, we achieve a precision of 0.76 and a recall of 0.60.

Code reviews do not find bugs: how the current code review best practice slows us down

Because of its many uses and benefits, code reviews are a standard part of the modern software engineering workflow. Since they require involvement of people, code reviewing is often the longest part of the code integration activities. Using experience gained at Microsoft and with support of data, we posit (1) that code reviews often do not find functionality issues that should block a code submission; (2) that effective code reviews should be performed by people with specific set of skills; and (3) that the social aspect of code reviews cannot be ignored. We find that we need to be more sophisticated with our guidelines for the code review workflow. We show how our findings from code reviewing practice influence our code review tools at Microsoft. Finally, we assert that, due to its costs, code reviewing practice is a topic deserving to be better understood, systematized and applied to software engineering workflow with more precision than the best practice currently prescribes.

Lessons learned from building and deploying a code review analytics platform

Tool-based code review is growing in popularity and has become a standard part of the development process at Mi-crosoft. Adoption of these tools makes it possible to mine data from code reviews and provide access to it. In this paper, we pre-sent an experience report for CodeFlow Analytics, a system that collects code review data, generates metrics from this data, and provides a number of ways for development teams to access the metrics and data. We discuss the design, design decisions and chal-lenges that we encountered when building CodeFlow Analytics. We contacted teams that used CodeFlow Analytics over the past two years and discuss what prompted them to use CodeFlow Ana-lytics, how they have used it, and what the impact has been. Fur-ther, we survey research that has been enabled by using the Code-Flow Analytics platform. We provide a series of lessons learned from this experience to help others embarking on a task of building an analytics platform in an enterprise setting.

Code Reviewing in the Trenches: Understanding Challenges and Best Practices

Code review has been widely adopted by and adapted to open source and industrial projects. Code review practices have undergone extensive research, with most studies relying on trace data from tool reviews, sometimes augmented by surveys and interviews. Several recent industrial research studies, along with blog posts and white papers, have revealed additional insights on code reviewing “from the trenches.” Unfortunately, the lessons learned about code reviewing are widely dispersed and poorly summarized by the existing literature. In particular, practitioners wishing to adopt or reflect on an existing or new code review process might have difficulty determining what challenges to expect and which best practices to adopt for their development context. Building on the existing literature, this article adds insights from a recent large-scale study of Microsoft developers to summarize the challenges that code-change authors and reviewers face, suggest best code-reviewing practices, and discuss tradeoffs that practitioners should consider. This article is part of a theme issue on Process Improvement.

Introduction to the special issue on source code analysis and manipulation

Research in the area of source code analysis is concerned with the main and most essential artifact and building block in the software engineering process. Those building blocks include any fully executable description of the software system starting from very low level expressions such as machine code to high level descriptions of the system using high level languages or even graphical representations. It is the source code where we can find answers to many questions one might have about the software system. And it is also the source code that represents the essential truth about the behavior and execution of a system. Therefore, it is an important and crucial requisite to analyze, manipulate and learn from it. In this special issue, we have selected four excellent papers that look at different aspects of source code analysis. Those articles have been selected from the accepted submissions to 16th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM 2016). SCAM brings together researchers and practitioners to enhance theory, techniques, and applications that focus on the analysis and manipulation of source code. To be included in this special issue the papers have been substantially enhanced and revised and went through another thorough reviewing process undergoing multiple review rounds. The selected studies tackle topic such as change impact analysis, information retrieval in software engineering, comparison of mutation testing and evaluation of similarity analysis techniques and tools. A more detailed description of each of the four articles included can be found below.