Moran Shochat

Advanced code coverage analysis using substring holes

Code coverage is a common aid in the testing process. It is generally used for marking the source code segments that were executed and, more importantly, those that were not executed. Many code coverage tools exist, supporting a variety of languages and operating systems. Unfortunately, these tools provide little or no assistance when code coverage data is voluminous. Such quantities are typical of system tests and even for earlier testing phases. Drill-down capabilities that look at different granularities of the data, starting with directories and going through files to functions and lines of source code, are insufficient. Such capabilities make the assumption that the coverage issues themselves follow the code hierarchy. We argue that this is not the case for much of the uncovered code. Two notable examples are error handling code and platform-specific constructs. Both tend to be spread throughout the source in many files, even though the related coverage, or lack thereof, is highly dependent. To make the task more manageable, and therefore more likely to be performed by users, we developed a hole analysis algorithm and tool that is based on common substrings in the names of functions. We tested its effectiveness using two large IBM software systems. In both of them, we asked domain experts to judge the results of several hole-ranking heuristics. They found that 57% - 87% of the 30 top-ranked holes identified by the effective heuristics are relevant. Moreover, these holes are often unexpected. This is especially impressive because substring hole analysis relies only on the names of functions, whereas domain experts have a broad and deep understanding of the system. We grounded our results in a theoretical framework that states desirable mathematical properties of hole ranking heuristics. The empirical results show that heuristics with these properties tend to perform better, and do so more consistently, than heuristics lacking them.

Automated substring hole analysis

Code coverage is a common measure for quantitatively assessing the quality of software testing. Code coverage indicates the fraction of code that is actually executed by tests in a test suite. While code coverage has been around since the 60s there has been little work on how to effectively analyze code coverage data measured in system tests. Raw data of this magnitude, containing millions of data records, is often impossible for a human user to comprehend and analyze. Even drill-down capabilities that enable looking at different granularities starting with directories and going through files to lines of source code are not enough. Substring hole analysis is a novel method for viewing the coverage of huge data sets. We have implemented a tool that enables automatic substring hole analysis. We used this tool to analyze coverage data of several large and complex IBM software systems. The tool identified coverage holes that suggested interesting scenarios that were untested.

SeeCode --- A Code Review Plug-in for Eclipse

It is well known that code reviews are among the most effective techniques for finding bugs [2, 3, 4]. In this paper, we describe a code review tool, SeeCode, which supports the code review process. SeeCode is an Eclipse plug-in and thus naturally integrates into the developers working environment. It supports a distributed review environment and the various roles used in a review meeting. Reviewers can review the code at the same time, either through a virtual or a face-to-face meeting, or at different times. Review comments and Author navigation through the code are visible to all reviewers. Review comments are associated with line numbers, and the association is maintained when the code is changed by the developer. The integration with the Eclipse [8] Integrated Development Environment (IDE) enables easy code navigation, which is required especially when object-oriented code is reviewed. SeeCode also supports a quantitative feedback mechanism that reports the effectiveness of the ongoing review effort. This feedback is updated as the review progresses, and can be utilized by the review moderator to keep the review process on track. SeeCode has been piloted by several IBM groups with good feedback. The distributed review feature and integration with the IDE are particularly noted by users as key features.