Malte Lochau

Incremental model-based testing of delta-oriented software product lines

Software product line (SPL) engineering provides a promising approach for developing variant-rich software systems. But, testing of every product variant in isolation to ensure its correctness is in general not feasible due to the large number of product variants. Hence, a systematic approach that applies SPL reuse principles also to testing of SPLs in a safe and efficient way is essential. To address this issue, we propose a novel, model-based SPL testing framework that is based on a delta-oriented SPL test model and regression-based test artifact derivations. Test artifacts are incrementally constructed for every product variant by explicitly considering commonality and variability between two consecutive products under test. The resulting SPL testing process is proven to guarantee stable test coverage for every product variant and allows the derivation of redundancy-reduced, yet reliable retesting obligations. We compare our approach with an alternative SPL testing strategy by means of a case study from the automotive domain.

Model-based pairwise testing for feature interaction coverage in software product line engineering

Testing software product lines (SPLs) is very challenging due to a high degree of variability leading to an enormous number of possible products. The vast majority of today’s testing approaches for SPLs validate products individually using different kinds of reuse techniques for testing. Because of their reusability and adaptability capabilities, model-based approaches are suitable to describe variability and are therefore frequently used for implementation and testing purposes of SPLs. Due to the enormous number of possible products, individual product testing becomes more and more infeasible. Pairwise testing offers one possibility to test a subset of all possible products. However, according to the best of our knowledge, there is no contribution discussing and rating this approach in the SPL context. In this contribution, we provide a mapping between feature models describing the common and variable parts of an SPL and a reusable test model in the form of statecharts. Thereby, we interrelate feature model-based coverage criteria and test model-based coverage criteria such as control and data flow coverage and are therefore able to discuss the potentials and limitations of pairwise testing. We pay particular attention to test requirements for feature interactions constituting a major challenge in SPL engineering. We give a concise definition of feature dependencies and feature interactions from a testing point of view, and we discuss adequacy criteria for SPL coverage under pairwise feature interaction testing and give a generalization to the T-wise case. The concept and implementation of our approach are evaluated by means of a case study from the automotive domain.

MoSo-PoLiTe: tool support for pairwise and model-based software product line testing

Testing Software Product Lines is a very challenging task and approaches like combinatorial testing and model-based testing are frequently used to reduce the effort of testing Software Product Lines and to reuse test artifacts. In this contribution we present a tool chain realizing our MoSo-PoLiTe concept combining combinatorial and model-based testing. Our tool chain contains a pairwise configuration selection component on the basis of a feature model. This component implements an heuristic finding a minimal subset of configurations covering 100% pairwise interaction. Additionally, our tool chain allows the model-based test case generation for each configuration within this generated subset. This tool chain is based on commercial tools since it was developed within industrial cooperations. A non-commercial implementation of pairwise configuration selection is available and an integration with an Open Source model-based testing tool is under development.

Dynamic configuration management of cloud-based applications

Cloud-based applications are multi-tenant aware, whereas customers (i.e., tenants) share hardware and software resources. Offering highly configurable applications to thousands of tenants in a shared cloud environment demands for scalable configuration management. Based on an example scenario taken from the Indenica project, we identify requirements for applying methods from software product line (SPL) engineering to configure cloud-based multi-tenant aware applications. Using an extended feature model (EFM) to express variability of functionality and service qualities, we propose a concept for dynamic configuration management to address the identified requirements. Our proposed configuration management includes an adaptive staged configuration process that is capable of adding and removing stakeholders dynamically and that allows for reconfiguration of variants as stakeholders objectives change.

Similarity-based prioritization in software product-line testing

Exhaustively testing every product of a software product line (SPL) is a difficult task due to the combinatorial explosion of the number of products. Combinatorial interaction testing is a technique to reduce the number of products under test. However, it is typically up-to the tester in which order these products are tested. We propose a similarity-based prioritization to be applied on these products before they are generated. The proposed approach does not guarantee to find more errors than sampling approaches, but it aims at increasing interaction coverage of an SPL under test as fast as possible over time. This is especially beneficial since usually the time budget for testing is limited. We implemented similarity-based prioritization in FeatureIDE and evaluated it by comparing its outcome to the default outcome of three sampling algorithms as well as to random orders. The experiment results indicate that the order with similarity-based prioritization is better than random orders and often better than the default order of existing sampling algorithms.

Model-based coverage-driven test suite generation for software product lines

Software Product Line (SPL) engineering is a popular approach for the systematic reuse of software artifacts across a large number of similar products. Unfortunately, testing each product of an SPL separately is often unfeasible. Consequently, SPL engineering is in conflict with standards like ISO 26262, which require each installed software configuration of safety-critical SPLs to be tested using a model-based approach with well-defined coverage criteria.n In this paper we address this dilemma and present a new SPL test suite generation algorithm that uses model-based testing techniques to derive a small test suite from one variable 150% test model of the SPL such that a given coverage criterion is satisfied for the test model of every product. Furthermore, our algorithm simplifies the subsequent selection of a small, representative set of products (w.r.t. the given coverage criterion) on which the generated test suite can be executed.

Pairwise feature-interaction testing for SPLs: potentials and limitations

A fundamental problem of testing Software Product Lines (SPLs) is that variability enables the production of a large number of instances and it is difficult to construct and run test cases even for SPLs with a small number of variable features. Interacting features is a foundation of a fault model for SPLs, where faults are likely to be revealed at execution points where features exchange information with other features or influence one another. Therefore, a test adequacy criterion is to cover as many interactions among different features as possible, thus increasing the probability of finding bugs. Our approach combines a combinatorial designs algorithm for pairwise feature generation with model-based testing to reduce the size of the SPL required for comprehensive coverage of interacting features. We implemented our approach and applied it to an SPL from the automotive domain provided by one of our industrial partners. The results suggest that with our approach higher coverage of feature interactions is achieved at a fraction of cost when compared with a baseline approach of testing all feature interactions.

IncLing: efficient product-line testing using incremental pairwise sampling

A software product line comprises a family of software products that share a common set of features. It enables customers to compose software systems from a managed set of features. Testing every product of a product line individually is often infeasible due to the exponential number of possible products in the number of features. Several approaches have been proposed to restrict the number of products to be tested by sampling a subset of products achieving sufficient combinatorial interaction coverage. However, existing sampling algorithms do not scale well to large product lines, as they require a considerable amount of time to generate the samples. Moreover, samples are not available until a sampling algorithm completely terminates. As testing time is usually limited, we propose an incremental approach of product sampling for pairwise interaction testing (called IncLing), which enables developers to generate samples on demand in a step-wise manner. Furthermore, IncLing uses heuristics to efficiently achieve pairwise interaction coverage with a reasonable number of products. We evaluated IncLing by comparing it against existing sampling algorithms using feature models of different sizes. The results of our approach indicate efficiency improvements for product-line testing.

Multi-perspectives on feature models

Domain feature models concisely express commonality and variability among variants of a software product line. For supporting separation of concerns, e.g., due to legal restrictions, technical considerations and business requirements, multi-view approaches restrict the configuration choices on feature models for different stakeholders. However, recent approaches lack a formalization for precise, yet flexible specifications of views that ensure every derivable configuration perspective to obey feature model semantics. Here, we introduce a novel approach for preconfiguring feature models to create multi-perspectives. Such customized perspectives result from composition of various concern-relevant views. A structured view model is used to organize features in view groups, wherein a feature may be contained in multiple views. We provide formalizations for view composition and guaranteed consistency of perspectives w.r.t. feature model semantics. Thereupon, an efficient algorithm to verify consistency for entire multi-perspectives is provided. We present an implementation and evaluate our concepts by means of various experiments.

Reducing feature models to improve runtime adaptivity on resource limited devices

Mobile devices like smartphones are getting increasingly important in our daily lifes. They are used in various environments and have to dynamically adapt themselves accordingly in order to provide an optimal runtime behavior. Naturally, adapting to continuously changing environmental conditions is a challenging task because mobile devices are always limited in their resources and have to adapt in real-time. In this paper, we introduce an approach that enables resource limited devices to adapt to changing conditions using dynamic software product lines techniques. Therefore, feature models are reduced to a specific hardware context before installing the adaptive mobile application on the device. This reduces the amount of possible configurations that are compatible with the device and, thereby, minimizes the costs and the duration of an adaptation during runtime.