Mustafa Al-Hajjaji

Similarity-based prioritization in software product-line testing

Exhaustively testing every product of a software product line (SPL) is a difficult task due to the combinatorial explosion of the number of products. Combinatorial interaction testing is a technique to reduce the number of products under test. However, it is typically up-to the tester in which order these products are tested. We propose a similarity-based prioritization to be applied on these products before they are generated. The proposed approach does not guarantee to find more errors than sampling approaches, but it aims at increasing interaction coverage of an SPL under test as fast as possible over time. This is especially beneficial since usually the time budget for testing is limited. We implemented similarity-based prioritization in FeatureIDE and evaluated it by comparing its outcome to the default outcome of three sampling algorithms as well as to random orders. The experiment results indicate that the order with similarity-based prioritization is better than random orders and often better than the default order of existing sampling algorithms.

IncLing: efficient product-line testing using incremental pairwise sampling

A software product line comprises a family of software products that share a common set of features. It enables customers to compose software systems from a managed set of features. Testing every product of a product line individually is often infeasible due to the exponential number of possible products in the number of features. Several approaches have been proposed to restrict the number of products to be tested by sampling a subset of products achieving sufficient combinatorial interaction coverage. However, existing sampling algorithms do not scale well to large product lines, as they require a considerable amount of time to generate the samples. Moreover, samples are not available until a sampling algorithm completely terminates. As testing time is usually limited, we propose an incremental approach of product sampling for pairwise interaction testing (called IncLing), which enables developers to generate samples on demand in a step-wise manner. Furthermore, IncLing uses heuristics to efficiently achieve pairwise interaction coverage with a reasonable number of products. We evaluated IncLing by comparing it against existing sampling algorithms using feature models of different sizes. The results of our approach indicate efficiency improvements for product-line testing.

Tool demo: testing configurable systems with FeatureIDE

Most software systems are designed to provide custom functionality using configuration options. Testing such systems is challenging as running tests of a single configuration is often not sufficient, because defects may appear in other configurations. Ideally, all configurations of a software system should be tested, which is usually not applicable in practice due to the combinatorial explosion with respect to the configuration options. Multiple sampling strategies aim to reduce the set of tested configurations to a feasible amount, such as T-wise sampling, random configurations, and user-defined configurations. However, these strategies are often not applied in practice as they require manual effort or a specialized testing framework. Within our tool FeatureIDE, we integrate all aforementioned strategies and reduce the manual effort by automating the process of generating and testing configurations. Furthermore, we provide support for unit testing to avoid redundant test executions and for variability-aware testing. With this extension of FeatureIDE, we aim to make recent testing techniques for configurable systems applicable in practice.

Effective product-line testing using similarity-based product prioritization

A software product line comprises a family of software products that share a common set of features. Testing an entire product-line product-by-product is infeasible due to the potentially exponential number of products in the number of features. Accordingly, several sampling approaches have been proposed to select a presumably minimal, yet sufficient number of products to be tested. Since the time budget for testing is limited or even a priori unknown, the order in which products are tested is crucial for effective product-line testing. Prioritizing products is required to increase the probability of detecting faults faster. In this article, we propose similarity-based prioritization, which can be efficiently applied on product samples. In our approach, we incrementally select the most diverse product in terms of features to be tested next in order to increase feature interaction coverage as fast as possible during product-by-product testing. We evaluate the gain in the effectiveness of similarity-based prioritization on three product lines with real faults. Furthermore, we compare similarity-based prioritization to random orders, an interaction-based approach, and the default orders produced by existing sampling algorithms considering feature models of various sizes. The results show that our approach potentially increases effectiveness in terms of fault detection ratio concerning faults within real-world product-line implementations as well as synthetically seeded faults. Moreover, we show that the default orders of recent sampling algorithms already show promising results, which, however, can still be improved in many cases using similarity-based prioritization.

Mutation Operators for Preprocessor-Based Variability

Mutation testing has emerged as one of the most promising techniques to increase the quality of software-intensive systems. In mutation testing, random faults based on a predefined set of mutation operators are automatically injected into a program to evaluate test suites. The effectiveness of mutation testing strongly depends on the representativeness of the mutation operators. Existing operators are not sufficient to represent typical faults caused by variability. Thus, we propose a set of mutation operators for software with preprocessor-based variability. We derive the operators systematically based on a taxonomy of variability-related faults and evaluate them by investigating their applicability to real-world faults that have previously been identified in research on configurable software systems. Our goal is to leverage mutation testing to highly-variable software for its practical application and to enable empirical evaluation of testing techniques.

Optimizing product orders using graph algorithms for improving incremental product-line analysis

The individual analysis of each product of a software product line (SPL) leads to redundant analysis steps due to the inherent commonality. Therefore, incremental SPL analyses exploit commonalities and focus on the differences between products to reduce the analysis effort. However, existing techniques are influenced by the order in which products are analyzed. The more similar subsequently analyzed products are, the greater is the potential reduction of the overall analysis effort as similar products imply less differences to be analyzed. Hence, an order of products, where the total number of differences is minimized, facilitates incremental SPL analyses. In this paper, we apply graph algorithms to determine optimized product orders. We capture products as nodes in a graph, where solution-space information defines edge weights between product nodes. We adopt existing heuristics for finding an optimal solution of the traveling salesperson problem to determine a path in the product graph with minimal costs. A path represents an optimized product order w.r.t. minimized differences between all products. We realize a prototype of our approach and evaluate its applicability and performance showing a significant optimization compared to standard and random orders.

Fine-grained test case prioritization for integration testing of delta-oriented software product lines

Software product line (SPL) testing is a challenging task, due to the huge number of variants sharing common functionalities to be taken into account for efficient testing. By adopting the concept of regression testing, incremental SPL testing strategies cope with this challenge by exploiting the reuse potential of test artifacts between subsequent variants under test. In previous work, we proposed delta-oriented test case prioritization for incremental SPL integration testing, where differences between architecture test model variants allow for reasoning about the order of reusable test cases to be executed. However, the prioritization left two issues open, namely (1) changes to component behavior are ignored, which may also influence component interactions and, (2) the weighting and ordering of similar test cases result in an unintended clustering of test cases. In this paper, we extend the test case prioritization technique by (1) incorporating changes to component behavior allowing for a more fine-grained analysis and (2) defining a dissimilarity measure to avoid clustered test case orders. We prototyped our test case prioritization technique and evaluated its applicability and effectiveness by means of a case study from the automotive domain showing positive results.

Delta-oriented product prioritization for similarity-based product-line testing

Testing every product of a software product line (SPL) is often not feasible due to the exponential number of products in the number of features. Thus, the order in which products are tested matters, because it can increase the early rate of fault detection. Several approaches have been proposed to prioritize products based on configuration similarity. However, current approaches are oblivious to solution-space differences among products, because they consider only problem-space information. With delta modeling, we incorporate solution-space information in product prioritization to improve the effectiveness of SPL testing. Deltas capture the differences between products facilitating the reasoning about product similarity. As a result, we select the most dissimilar product to the previously tested ones, in terms of deltas, to be tested next. We evaluate the effectiveness of our approach using an SPL from the automotive domain showing an improvement in the effectiveness of SPL testing.

Equivalent Mutants in Configurable Systems: An Empirical Study

Mutation testing is a program-transformation technique that evaluates the quality of test cases by assessing their capability to detect injected artificial faults. The costs of using mutation testing are usually high, hindering its use in industry. Previous research has reported that roughly one-third of the mutants generated in single systems are equivalents. In configurable systems, a set of mutation operators that focus on preprocessor directives (e.g., #ifdef) has been proposed. However, there is a lack of existing studies that investigate whether equivalent mutants do occur with these operators. To perform this investigation, we provide a tool that implements the aforementioned mutation operators and we conduct an empirical study using 20 C files of four industrial-scale systems. In particular, we provide examples of equivalent mutants and detailed information, such as which mutation operators generate these mutants and how often they occur. Our preliminary results show that nearly 40% of the generated mutants are equivalent. Hence, testing costs can be drastically reduced if the community comes up with efficient techniques to avoid these equivalent mutants.

Efficient product-line testing using cluster-based product prioritization

A software product-line comprises a set of products that share a common set of features. These features can be reused to customize a product to satisfy specific needs of certain customers or markets. As the number of possible products increases exponentially for new features, testing all products is infeasible. Existing testing approaches reduce their effort by restricting the number of products (sampling) and improve their effectiveness by considering the order of tests (prioritization). In this paper, we propose a cluster-based prioritization technique to sample similar products with respect to the feature selection. We evaluate our approach using feature models of different sizes and show that cluster-based prioritization can enhance the effectiveness of product-line testing.