Aviad Zlotnick

Interaction-based test-suite minimization

Combinatorial Test Design (CTD) is an effective test planning technique that reveals faults resulting from feature interactions in a system. The standard application of CTD requires manual modeling of the test space, including a precise definition of restrictions between the test space parameters, and produces a test suite that corresponds to new test cases to be implemented from scratch. In this work, we propose to use Interaction-based Test-Suite Minimization (ITSM) as a complementary approach to standard CTD. ITSM reduces a given test suite without impacting its coverage of feature interactions. ITSM requires much less modeling effort, and does not require a definition of restrictions. It is appealing where there has been a significant investment in an existing test suite, where creating new tests is expensive, and where restrictions are very complex. We discuss the tradeoffs between standard CTD and ITSM, and suggest an efficient algorithm for solving the latter. We also discuss the challenges and additional requirements that arise when applying ITSM to real-life test suites. We introduce solutions to these challenges and demonstrate them through two real-life case studies.

Simplified Modeling of Combinatorial Test Spaces

Combinatorial test design (CTD) is an effective test planning technique that reveals faults that result from feature interactions in a system. The test space is manually modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is usually a user input. In many real-life testing problems, the relationships between the different test parameters are complex. Thus, precisely capturing them by restrictions in the CTD model might be a very challenging and time consuming task. From our experience, this is one of the main obstacles in applying CTD to a wide range of testing problems. In this paper, we introduce two new constructs to the CTD model, counters and value properties, that considerably reduce the complexity of the modeling task, allowing one to easily model testing problems that were practically impossible to model before. We demonstrate the impact of these constructs on two real-life case studies.

Forcing small models of conditions on program interleaving for detection of concurrent bugs

Is it possible to efficiently reveal concurrency bugs by forcing a well selected set of conditions on program interleaving? To study this general question, we defined two simple models of conditions on program interleaving targeted at the insufficient-synchronization-scope bug pattern. We analyzed these models with respect to several buggy programs. We also implemented an algorithm that tries to force one of these models. The analysis of these models shows that relatively small models can detect insufficient-synchronization-scope bugs. The experiments with the forcing algorithm demonstrated the ability of finding the bug with high efficiency: the average testing time till the bug is detected was improved by factors of 7 and 73 compared with the average time required by a dynamic exploration that did not incorporate the forcing algorithm.

Advanced code coverage analysis using substring holes

Code coverage is a common aid in the testing process. It is generally used for marking the source code segments that were executed and, more importantly, those that were not executed. Many code coverage tools exist, supporting a variety of languages and operating systems. Unfortunately, these tools provide little or no assistance when code coverage data is voluminous. Such quantities are typical of system tests and even for earlier testing phases. Drill-down capabilities that look at different granularities of the data, starting with directories and going through files to functions and lines of source code, are insufficient. Such capabilities make the assumption that the coverage issues themselves follow the code hierarchy. We argue that this is not the case for much of the uncovered code. Two notable examples are error handling code and platform-specific constructs. Both tend to be spread throughout the source in many files, even though the related coverage, or lack thereof, is highly dependent. To make the task more manageable, and therefore more likely to be performed by users, we developed a hole analysis algorithm and tool that is based on common substrings in the names of functions. We tested its effectiveness using two large IBM software systems. In both of them, we asked domain experts to judge the results of several hole-ranking heuristics. They found that 57% - 87% of the 30 top-ranked holes identified by the effective heuristics are relevant. Moreover, these holes are often unexpected. This is especially impressive because substring hole analysis relies only on the names of functions, whereas domain experts have a broad and deep understanding of the system. We grounded our results in a theoretical framework that states desirable mathematical properties of hole ranking heuristics. The empirical results show that heuristics with these properties tend to perform better, and do so more consistently, than heuristics lacking them.

Combinatorial Testing with Order Requirements

Combinatorial Test Design, CTD, does not easily lend itself to the modeling of ordered parameter-values. Such modeling is much needed in practice, e.g. For the testing of sequences of API calls or parameterized events. We extend the CTD paradigm to address this need. We define a test as an ordered tuple of the parameter-values of the model, and introduce the concepts of ordered restrictions and ordered interaction coverage requirements. We develop an efficient algorithm for generating a small set of tests that satisfy the ordered and unordered interaction coverage requirements and evaluate it on several real-life examples.

Automated substring hole analysis

Code coverage is a common measure for quantitatively assessing the quality of software testing. Code coverage indicates the fraction of code that is actually executed by tests in a test suite. While code coverage has been around since the 60s there has been little work on how to effectively analyze code coverage data measured in system tests. Raw data of this magnitude, containing millions of data records, is often impossible for a human user to comprehend and analyze. Even drill-down capabilities that enable looking at different granularities starting with directories and going through files to lines of source code are not enough. Substring hole analysis is a novel method for viewing the coverage of huge data sets. We have implemented a tool that enables automatic substring hole analysis. We used this tool to analyze coverage data of several large and complex IBM software systems. The tool identified coverage holes that suggested interesting scenarios that were untested.

A weakly labeled approach for breast tissue segmentation and breast density estimation in digital mammography

Breast tissue segmentation is a fundamental task in digital mammography. Commonly, this segmentation is applied prior to breast density estimation. However, observations show a strong correlation between the segmentation parameters and the breast density, resulting in a chicken and egg problem. This paper presents a new method for breast segmentation, based on training with weakly labeled data, namely breast density categories. To this end, a Fuzzy-logic module is employed computing an adaptive parameter for segmentation. The suggested scheme consists of a feedback stage where a preliminary segmentation is used to allow extracting domain specific features from an early estimation of the tissue regions. Selected features are then fed into a fuzzy logic module to yield an updated threshold for segmentation. Our evaluation is based on 50 fibroglandular delineated images and on breast density classification, obtained on a large data set of 1243 full-field digital mammograms. The data set contained images from different devices. The proposed analysis provided an average Jaccard spatial similarity coefficient of 0.4 with improvement of this measure in 70% of cases where the suggested module was applied. In breast density classification, average classification accuracy of 75% was obtained, which significantly improved the baseline method (67.4%). Major improvement is obtained in low breast densities where higher threshold levels rejects false positive regions. These results show a promise for the clinical application of this method in breast segmentation, without the need for laborious tissue annotation.

Testing large-scale cloud management

Testing for correctness and reliability is a major challenge in the development and deployment of cloud computing platforms. Testing a cloud composed of hundreds to thousands of servers is often cost-prohibitive because of the extensive amount of hardware required. Simulation and emulation, i.e., traditional alternatives to hardware, are too abstract or too slow for testing production code in environments with many servers. We propose a testing approach that combines simulation and emulation in a cloud simulator that runs on a single processor yet enables testing of cloud management software as if the software were managing hundreds of servers and thousands of virtual machine instances. This approach alleviates a significant obstacle on the path to high-quality cloud computing systems.

Automatic Dual-View Mass Detection in Full-Field Digital Mammograms

Mammography is the first-line modality for screening and diagnosis of breast cancer. Following the common practice of radiologists to examine two mammography views, we propose a fully automated dual-view analysis framework for breast mass detection in mammograms. The framework combines unsupervised segmentation and random-forest classification to detect and rank candidate masses in cranial-caudal (CC) and mediolateral-oblique (MLO) views. Subsequently, it estimates correspondences between pairs of candidates in the two views. The performance of the method was evaluated using a publicly available full-field digital mammography database (INbreast). Dual-view analysis provided area under the ROC curve of 0.94, with detection sensitivity of 87% at specificity of 90%, which significantly improved single-view performance (72% sensitivity at 90% specificity, 78% specificity at 87% sensitivity, P<0.05). One-to-one mapping of candidate masses from two views facilitated correct estimation of the breast quadrant in 77% of the cases. The proposed method may assist radiologists to efficiently identify and classify breast masses.

Combining minimization and generation for combinatorial testing

Combinatorial Test Design (CTD) is an effective test planning technique that reveals faults resulting from feature interactions in a system. The standard application of CTD requires manual modeling of the test space, including a precise definition of restrictions between the test space parameters, and produces a test suite that corresponds to new test cases to be implemented from scratch. Interaction-based Test-Suite Minimization (ITSM) is a complementary approach to standard CTD, which reduces a given test suite without impacting its coverage of feature interactions. ITSM requires much less modeling effort, and does not require a definition of restrictions or generation of new test data. On the other hand, it does not improve the coverage obtained by the given test suite. In this work, we introduce Minimization Generation CTD (MG-CTD). MG-CTD is a combination of CTD with ITSM for addressing situations in which CTD is impractical, and ITSM is insufficient. In MG-CTD, one can define a subset of the parameters that can be freely assigned, as in CTD. The other parameter combinations must be selected from an existing set, as in ITSM. MG-CTD is suitable when for some parts of the test space it is easy to specify restrictions and generate new test data, while for others it is not. MG-CTD can be viewed as an enhancement of ITSM, and always achieves better interaction coverage than ITSM. We discuss the trade-offs between CTD, ITSM and MG-CTD, and present an efficient implementation which is based on binary decision diagrams. We then present some of the measures that one should take when implementing such an approach, in order to achieve the best possible coverage in the final result. Finally, we demonstrate MG-CTD on three real-life case studies.