Rachel Tzoref-Brill

Interaction-based test-suite minimization

Combinatorial Test Design (CTD) is an effective test planning technique that reveals faults resulting from feature interactions in a system. The standard application of CTD requires manual modeling of the test space, including a precise definition of restrictions between the test space parameters, and produces a test suite that corresponds to new test cases to be implemented from scratch. In this work, we propose to use Interaction-based Test-Suite Minimization (ITSM) as a complementary approach to standard CTD. ITSM reduces a given test suite without impacting its coverage of feature interactions. ITSM requires much less modeling effort, and does not require a definition of restrictions. It is appealing where there has been a significant investment in an existing test suite, where creating new tests is expensive, and where restrictions are very complex. We discuss the tradeoffs between standard CTD and ITSM, and suggest an efficient algorithm for solving the latter. We also discuss the challenges and additional requirements that arise when applying ITSM to real-life test suites. We introduce solutions to these challenges and demonstrate them through two real-life case studies.

System Level Combinatorial Testing in Practice -- The Concurrent Maintenance Case Study

Combinatorial test design (CTD) is an effective test design technique that reveals faults resulting from parameter interactions in a system. CTD requires a test space definition in the form of a set of parameters, their respective values, and restrictions on the value combinations. Though CTD is considered an industry best practice, there is only a small body of work on the practical application of CTD to industrial systems, and some key elements of the CTD application process are under-explored. Specifically, little consideration has been given to the process for identifying the parameters of the test space and their interrelations, how to validate the test space definition remains an open question, the application of CTD in reported work concentrates mostly on function or interface level testing and is hardly expanded to other levels of testing, and there is a significant lack of evaluation of the degree to which CTD helped improve the quality of the system under test. In this work, we analyze the continuous application of CTD in system test of two large industrial systems: IBM® POWER7® and IBM® System z®. For POWER7, CTD was used to design test cases for server concurrent maintenance. The application of CTD was in direct response to inconsistent reliability of those features on the prior POWER6® servers, and resulted in noteworthy quality improvements on POWER7. Success with POWER7 led to application of the methods to System z Enhanced Driver Maintenance testing, also featured in this work. To the best of our knowledge, this is the first comprehensive analysis of CTD usage in system test of industrial products, and the first analysis of long term use of CTD. We describe the methodology that we followed to define the combinatorial test space, while answering some unique challenges rising from the use of CTD to design functional test cases at system level rather than at interface or function level. We also describe our methodology for evaluating the test space definition and continuously improving it over time. In addition, we describe advanced CTD features that we found helpful for achieving an effective yet affordable test plan. Finally, we quantitatively and qualitatively evaluate the overall effectiveness of CTD usage, and show that it resulted in significantly improved server concurrent maintenance features.Combinatorial test design (CTD) is an effective test design technique that reveals faults resulting from parameter interactions in a system. CTD requires a test space definition in the form of a set of parameters, their respective values, and restrictions on the value combinations. Though CTD is considered an industry best practice, there is only a small body of work on the practical application of CTD to industrial systems, and some key elements of the CTD application process are under-explored. Specifically, little consideration has been given to the process for identifying the parameters of the test space and their interrelations, how to validate the test space definition remains an open question, the application of CTD in reported work concentrates mostly on function or interface level testing and is hardly expanded to other levels of testing, and there is a significant lack of evaluation of the degree to which CTD helped improve the quality of the system under test. In this work, we analyze the continuous application of CTD in system test of two large industrial systems: IBM® POWER7® and IBM® System z®. For POWER7, CTD was used to design test cases for server concurrent maintenance. The application of CTD was in direct response to inconsistent reliability of those features on the prior POWER6® servers, and resulted in noteworthy quality improvements on POWER7. Success with POWER7 led to application of the methods to System z Enhanced Driver Maintenance testing, also featured in this work. To the best of our knowledge, this is the first comprehensive analysis of CTD usage in system test of industrial products, and the first analysis of long term use of CTD. We describe the methodology that we followed to define the combinatorial test space, while answering some unique challenges rising from the use of CTD to design functional test cases at system level rather than at interface or function level. We also describe our methodology for evaluating the test space definition and continuously improving it over time. In addition, we describe advanced CTD features that we found helpful for achieving an effective yet affordable test plan. Finally, we quantitatively and qualitatively evaluate the overall effectiveness of CTD usage, and show that it resulted in significantly improved server concurrent maintenance features.

Simplified Modeling of Combinatorial Test Spaces

Combinatorial test design (CTD) is an effective test planning technique that reveals faults that result from feature interactions in a system. The test space is manually modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is usually a user input. In many real-life testing problems, the relationships between the different test parameters are complex. Thus, precisely capturing them by restrictions in the CTD model might be a very challenging and time consuming task. From our experience, this is one of the main obstacles in applying CTD to a wide range of testing problems. In this paper, we introduce two new constructs to the CTD model, counters and value properties, that considerably reduce the complexity of the modeling task, allowing one to easily model testing problems that were practically impossible to model before. We demonstrate the impact of these constructs on two real-life case studies.

Forcing small models of conditions on program interleaving for detection of concurrent bugs

Is it possible to efficiently reveal concurrency bugs by forcing a well selected set of conditions on program interleaving? To study this general question, we defined two simple models of conditions on program interleaving targeted at the insufficient-synchronization-scope bug pattern. We analyzed these models with respect to several buggy programs. We also implemented an algorithm that tries to force one of these models. The analysis of these models shows that relatively small models can detect insufficient-synchronization-scope bugs. The experiments with the forcing algorithm demonstrated the ability of finding the bug with high efficiency: the average testing time till the bug is detected was improved by factors of 7 and 73 compared with the average time required by a dynamic exploration that did not incorporate the forcing algorithm.

Combinatorial Testing with Order Requirements

Combinatorial Test Design, CTD, does not easily lend itself to the modeling of ordered parameter-values. Such modeling is much needed in practice, e.g. For the testing of sequences of API calls or parameterized events. We extend the CTD paradigm to address this need. We define a test as an ordered tuple of the parameter-values of the model, and introduce the concepts of ordered restrictions and ordered interaction coverage requirements. We develop an efficient algorithm for generating a small set of tests that satisfy the ordered and unordered interaction coverage requirements and evaluate it on several real-life examples.

Lattice-Based Semantics for Combinatorial Model Evolution

Combinatorial test design (CTD) is an effective test design technique, considered to be a testing best practice. CTD provides automatic test plan generation, but it requires a manual definition of the test space in the form of a combinatorial model. As the system under test evolves, e.g., due to iterative development processes and bug fixing, so does the test space, and thus, in the context of CTD, evolution translates into frequent manual model definition updates.

Cluster-based test suite functional analysis

A common industrial challenge is that of analyzing large legacy free text test suites in order to comprehend their functional content. The analysis results are used for different purposes, such as dividing the test suite into disjoint functional parts for automation and management purposes, identifying redundant test cases, and extracting models for combinatorial test generation while reusing the legacy test suite. Currently the analysis is performed manually, which hinders the ability to analyze many such large test suites due to time and resource constraints. We report on our practical experience in automated analysis of real-world free text test suites from six different industrial companies. Our novel, cluster-based approach provides significant time savings for the analysis of the test suites, varying from a reduction of 35% to 97% compared to the human time required, thus enabling functional analysis in many cases where manual analysis is infeasible in practice.

Visualization of combinatorial models and test plans

Combinatorial test design (CTD) is an effective and widely used test design technique. CTD provides automatic test plan generation, but it requires a manual definition of the test space in the form of a combinatorial model. One challenge for successful application of CTD in practice relates to this manual model definition and maintenance process. Another challenge relates to the comprehension and use of the test plan generated by CTD for prioritization purposes. In this work we introduce the use of visualizations as a means to address these challenges. We apply three different forms of visualization, matrices, graphs, and treemaps, to visualize the relationships between the different elements of the model, and to visualize the strength of each test in the test plan and the relationships between the different tests in terms of combinatorial coverage. We evaluate our visualizations via a user survey with 19 CTD practitioners, as well as via two industrial projects in which our visualization was used and allowed test designers to get vital insight into their models and into the coverage provided through CTD generated test plans.

Combining minimization and generation for combinatorial testing

Combinatorial Test Design (CTD) is an effective test planning technique that reveals faults resulting from feature interactions in a system. The standard application of CTD requires manual modeling of the test space, including a precise definition of restrictions between the test space parameters, and produces a test suite that corresponds to new test cases to be implemented from scratch. Interaction-based Test-Suite Minimization (ITSM) is a complementary approach to standard CTD, which reduces a given test suite without impacting its coverage of feature interactions. ITSM requires much less modeling effort, and does not require a definition of restrictions or generation of new test data. On the other hand, it does not improve the coverage obtained by the given test suite. In this work, we introduce Minimization Generation CTD (MG-CTD). MG-CTD is a combination of CTD with ITSM for addressing situations in which CTD is impractical, and ITSM is insufficient. In MG-CTD, one can define a subset of the parameters that can be freely assigned, as in CTD. The other parameter combinations must be selected from an existing set, as in ITSM. MG-CTD is suitable when for some parts of the test space it is easy to specify restrictions and generate new test data, while for others it is not. MG-CTD can be viewed as an enhancement of ITSM, and always achieves better interaction coverage than ITSM. We discuss the trade-offs between CTD, ITSM and MG-CTD, and present an efficient implementation which is based on binary decision diagrams. We then present some of the measures that one should take when implementing such an approach, in order to achieve the best possible coverage in the final result. Finally, we demonstrate MG-CTD on three real-life case studies.

Feedback-driven combinatorial test design and execution

This work introduces a novel approach for online design and execution of load tests on Cloud applications. Our approach utilizes a Combinatorial Test Design (CTD) engine in order to exercise combinations of levels of resource utilization on the target systems subcomponents. In order to cope with the unpredictability and uncontrollability of Cloud environments, and to align with agile and DevOps paradigms, it designs and executes tests in an iterative online fashion. During test execution, monitoring information is collected from the Cloud, and leveraged for driving and adjusting the subsequent test scenarios. In this work we introduce the overall approach and the algorithms behind it, and demonstrate it on an example setting consisting of three sub-components comprising a typical installation of a web blogging application.