Sascha Lity

Incremental model-based testing of delta-oriented software product lines

Software product line (SPL) engineering provides a promising approach for developing variant-rich software systems. But, testing of every product variant in isolation to ensure its correctness is in general not feasible due to the large number of product variants. Hence, a systematic approach that applies SPL reuse principles also to testing of SPLs in a safe and efficient way is essential. To address this issue, we propose a novel, model-based SPL testing framework that is based on a delta-oriented SPL test model and regression-based test artifact derivations. Test artifacts are incrementally constructed for every product variant by explicitly considering commonality and variability between two consecutive products under test. The resulting SPL testing process is proven to guarantee stable test coverage for every product variant and allows the derivation of redundancy-reduced, yet reliable retesting obligations. We compare our approach with an alternative SPL testing strategy by means of a case study from the automotive domain.

Delta-oriented model-based integration testing of large-scale systems

Software architecture specifications are of growing importance for coping with the complexity of large-scale systems. They provide an abstract view on the high-level structural system entities together with their explicit dependencies and build the basis for ensuring behavioral conformance of component implementations and interactions, e.g., using model-based integration testing. The increasing inherent diversity of such large-scale variant-rich systems further complicates quality assurance. In this article, we present a combination of architecture-driven model-based testing principles and regression-inspired testing strategies for efficient, yet comprehensive variability-aware conformance testing of variant-rich systems. We propose an integrated delta-oriented architectural test modeling and testing approach for component as well as integration testing that allows the generation and reuse of test artifacts among different system variants. Furthermore, an automated derivation of retesting obligations based on accurate delta-oriented architectural change impact analysis is provided. Based on a formal conceptual framework that guarantees stable test coverage for every system variant, we present a sample implementation of our approach and an evaluation of the validity and efficiency by means of a case study from the automotive domain.

Multi-objective Test Suite Optimization for Incremental Product Family Testing

The design of an adequate test suite is usually guided by identifying test requirements which should be satisfied by the selected set of test cases. To reduce testing costs, test suite minimization heuristics aim at eliminating redundancy from existing test suites. However, recent test suite minimization approaches lack (1) to handle test suites commonly derived for families of similar software variants under test, and (2) to incorporate fine-grained information concerning cost/profit goals for test case selection. In this paper, we propose a formal framework to optimize test suites designed for sets of software variants under test w.r.t. multiple conflicting cost/profit objectives. The problem representation is independent of the concrete testing methodology. We apply integer linear programming (ILP) to approximate optimal solutions. We further develop an efficient incremental heuristic for deriving a sequence of representative software variants to be tested for approaching optimal profits under reduced costs. We evaluated the algorithm by comparing its outcome to the optimal solution.

Staged configuration of dynamic software product lines with complex binding time constraints

Dynamic software product lines (DSPL) constitute a promising approach for developing highly-configurable, runtime-adaptive systems in a feature-oriented way. A DSPL integrates both variability in time and space in a unified conceptual framework. For this, domain features are equipped with additional binding time information to distinguish between static configuration parameters and dynamically (re-) configurable features. Until now, little support exists to specify and validate staged (re-)configuration semantics for DSPLs in a concise way. In this paper, we propose conservative extensions to domain feature models comprising variable feature binding times together with different kinds of binding time constraints. Those extensions are motivated by a real-world industrial case study from the automation engineering domain. Our implementation performs a model transformation into plain feature models treatable by corresponding state-of-the-art analysis tools. We conducted an evaluation of our approach concerning the case study.

Applying Model-based Software Product Line Testing Approaches to the Automation Engineering Domain

Abstract The software constitutes a major part of nowadays automation systems being responsible for conducting complex control tasks. Machines and plants are often unique in some industrial branches; hence, they become mechatronic products configured individually. The inherent software variability of those highly-configurable systems makes efficient, yet accurate quality assurance a challenging task. This article presents a comprehensive approach for applying model-based software product line testing techniques to the automation engineering domain. Existing approaches for variability modeling are adapted to domain specific modeling languages to allow for variability-aware test case generation and execution. The implementation of the approach is evaluated by means of a sample automation system product line.

Applying Incremental Model Slicing to Product-Line Regression Testing

One crucial activity in software product linei¾źSPL testing is the detection of erroneous artifact interactions when combined for a variant. This detection is similar to the retest test-case selection problem in regression testing, where change impact analysis is applied to reason about changed dependencies to be retested. In this paper, we propose automated change impact analysis based on incremental model slicing for incremental SPL testing. Incremental slicing allows for a slice computation by adapting a previous slice with explicit derivation of their differences by taking model changes into account. We apply incremental slicing to determine the impact of applied model changes and to reason about their potential retest. Based on our novel retest coverage criterion, each slice change specifies a retest test goal to be covered by existing test cases selected for retesting. We prototypically implemented our approach and evaluated its applicability and effectiveness by means of four SPLs.

Delta-oriented model-based SPL regression testing

Testing software product lines by considering each product variant in isolation is impracticable due to the high number of potential product configurations. Therefore, applying SPL reuse principles also to test artifacts in a concise way is essential. We address this open issue by a novel, model-based SPL testing framework based on reusable delta-oriented state machine test models and regression-based test suite evolution. Therein, SPL test artifacts are incrementally evolved for every product variant by explicitly considering commonality and variability between two subsequent products under test. Our approach guarantees for every product configuration stable test coverage and allows the derivation of redundancy-reduced, yet reliable retesting obligations. We illustrate our framework by means of an automotive case study and compare our experimental results with alternative SPL testing strategies w.r.t. efficiency improvements.

Delta-oriented test case prioritization for integration testing of software product lines

Software product lines have potential to allow for mass customization of products. Unfortunately, the resulting, vast amount of possible product variants with commonalities and differences leads to new challenges in software testing. Ideally, every product variant should be tested, especially in safety-critical systems. However, due to the exponentially increasing number of product variants, testing every product variant is not feasible. Thus, new concepts and techniques are required to provide efficient SPL testing strategies exploiting the commonalities of software artifacts between product variants to reduce redundancy in testing. In this paper, we present an efficient integration testing approach for SPLs based on delta modeling. We focus on test case prioritization. As a result, only the most important test cases for every product variant are tested, reducing the number of executed test cases significantly, as testing can stop at any given point because of resource constraints while ensuring that the most important test cases have been covered. We present the general concept and our evaluation results. The results show a measurable reduction of executed test cases compared to single-software testing approaches.

Towards incremental model slicing for delta-oriented software product lines

The analysis of nowadays software systems for supporting, e.g., testing, verification or debugging is becoming more challenging due to their increasing complexity. Model slicing is a promising analysis technique to tackle this issue by abstracting from those parts not influencing the current point of interest. In the context of software product lines, applying model slicing separately for each variant is in general infeasible. Delta modeling allows exploiting the explicit specification of commonality and variability within deltas and enables the reuse of artifacts and already obtained results to reduce the modeling and analysis efforts. In this paper, we propose a novel approach for incremental model slicing for delta-oriented software product lines. Based on the specification of model changes between variants by means of model regression deltas, an incremental adaptation of variant-specific dependency graphs as well as an incremental slice computation is achieved. The slice computation further allows for the derivation of differences between slices for the same point of interest enhancing, e.g., change impact analysis. We provide details of our incremental approach, discuss benefits and present future work.

Optimizing product orders using graph algorithms for improving incremental product-line analysis

The individual analysis of each product of a software product line (SPL) leads to redundant analysis steps due to the inherent commonality. Therefore, incremental SPL analyses exploit commonalities and focus on the differences between products to reduce the analysis effort. However, existing techniques are influenced by the order in which products are analyzed. The more similar subsequently analyzed products are, the greater is the potential reduction of the overall analysis effort as similar products imply less differences to be analyzed. Hence, an order of products, where the total number of differences is minimized, facilitates incremental SPL analyses. In this paper, we apply graph algorithms to determine optimized product orders. We capture products as nodes in a graph, where solution-space information defines edge weights between product nodes. We adopt existing heuristics for finding an optimal solution of the traveling salesperson problem to determine a path in the product graph with minimal costs. A path represents an optimized product order w.r.t. minimized differences between all products. We realize a prototype of our approach and evaluate its applicability and performance showing a significant optimization compared to standard and random orders.