Dale E. Blue

Interaction-based test-suite minimization

Combinatorial Test Design (CTD) is an effective test planning technique that reveals faults resulting from feature interactions in a system. The standard application of CTD requires manual modeling of the test space, including a precise definition of restrictions between the test space parameters, and produces a test suite that corresponds to new test cases to be implemented from scratch. In this work, we propose to use Interaction-based Test-Suite Minimization (ITSM) as a complementary approach to standard CTD. ITSM reduces a given test suite without impacting its coverage of feature interactions. ITSM requires much less modeling effort, and does not require a definition of restrictions. It is appealing where there has been a significant investment in an existing test suite, where creating new tests is expensive, and where restrictions are very complex. We discuss the tradeoffs between standard CTD and ITSM, and suggest an efficient algorithm for solving the latter. We also discuss the challenges and additional requirements that arise when applying ITSM to real-life test suites. We introduce solutions to these challenges and demonstrate them through two real-life case studies.

Evaluating workloads using comparative functional coverage

We introduce comparative functional coverage—a technique for comparing the coverage of multiple workloads—and the tool in which it was implemented, FoCuS. The need to compare workloads and the use of functional coverage as a technique to explore data are not new. However, the use of functional coverage for comparing workloads has not been addressed as a method that responds to this long unanswered need. We describe our work in augmenting a functional coverage tool so it can handle multiple data sources. We present the data and include an experiment that shows the usefulness of this method.

Evaluating workloads using multi-comparative functional coverage

In this paper we present a technique for comparing multiple tests and workloads. We show how to automatically determine what each test does uniquely and how to present the information as succinctly as possible. This technology has a number of uses, including in understanding the contribution of tests in regression buckets, especially of legacy systems, and in evaluating what is missing in tests compared to customer usage. We also show that the technology used in the analysis is superior to previous technology in that it can automatically find holes that were previously only found manually.

Proactive and pervasive combinatorial testing

Combinatorial testing (CT) is a well-known technique for improving the quality of test plans while reducing testing costs. Traditionally, CT is used by testers at testing phase to design a test plan based on a manual definition of the test space. In this work, we extend the traditional use of CT to other parts of the development life cycle. We use CT at early design phase to improve design quality. We also use CT after test cases have been created and executed, in order to find gaps between design and test. For the latter use case we deploy a novel technique for a semi-automated definition of the test space, which significantly reduces the effort associated with manual test space definition. We report on our practical experience in applying CT for these use cases to three large and heavily deployed industrial products. We demonstrate the value gained from extending the use of CT by (1) discovering latent design flaws with high potential impact, and (2) correlating CT-uncovered gaps between design and test with field reported problems.