Yoram Adler

Advanced code coverage analysis using substring holes

Code coverage is a common aid in the testing process. It is generally used for marking the source code segments that were executed and, more importantly, those that were not executed. Many code coverage tools exist, supporting a variety of languages and operating systems. Unfortunately, these tools provide little or no assistance when code coverage data is voluminous. Such quantities are typical of system tests and even for earlier testing phases. Drill-down capabilities that look at different granularities of the data, starting with directories and going through files to functions and lines of source code, are insufficient. Such capabilities make the assumption that the coverage issues themselves follow the code hierarchy. We argue that this is not the case for much of the uncovered code. Two notable examples are error handling code and platform-specific constructs. Both tend to be spread throughout the source in many files, even though the related coverage, or lack thereof, is highly dependent. To make the task more manageable, and therefore more likely to be performed by users, we developed a hole analysis algorithm and tool that is based on common substrings in the names of functions. We tested its effectiveness using two large IBM software systems. In both of them, we asked domain experts to judge the results of several hole-ranking heuristics. They found that 57% - 87% of the 30 top-ranked holes identified by the effective heuristics are relevant. Moreover, these holes are often unexpected. This is especially impressive because substring hole analysis relies only on the names of functions, whereas domain experts have a broad and deep understanding of the system. We grounded our results in a theoretical framework that states desirable mathematical properties of hole ranking heuristics. The empirical results show that heuristics with these properties tend to perform better, and do so more consistently, than heuristics lacking them.

Evaluating workloads using comparative functional coverage

We introduce comparative functional coverage—a technique for comparing the coverage of multiple workloads—and the tool in which it was implemented, FoCuS. The need to compare workloads and the use of functional coverage as a technique to explore data are not new. However, the use of functional coverage for comparing workloads has not been addressed as a method that responds to this long unanswered need. We describe our work in augmenting a functional coverage tool so it can handle multiple data sources. We present the data and include an experiment that shows the usefulness of this method.

Automated substring hole analysis

Code coverage is a common measure for quantitatively assessing the quality of software testing. Code coverage indicates the fraction of code that is actually executed by tests in a test suite. While code coverage has been around since the 60s there has been little work on how to effectively analyze code coverage data measured in system tests. Raw data of this magnitude, containing millions of data records, is often impossible for a human user to comprehend and analyze. Even drill-down capabilities that enable looking at different granularities starting with directories and going through files to lines of source code are not enough. Substring hole analysis is a novel method for viewing the coverage of huge data sets. We have implemented a tool that enables automatic substring hole analysis. We used this tool to analyze coverage data of several large and complex IBM software systems. The tool identified coverage holes that suggested interesting scenarios that were untested.

Evaluating workloads using multi-comparative functional coverage

In this paper we present a technique for comparing multiple tests and workloads. We show how to automatically determine what each test does uniquely and how to present the information as succinctly as possible. This technology has a number of uses, including in understanding the contribution of tests in regression buckets, especially of legacy systems, and in evaluating what is missing in tests compared to customer usage. We also show that the technology used in the analysis is superior to previous technology in that it can automatically find holes that were previously only found manually.