Sebastian Oster

Automated incremental pairwise testing of software product lines

Testing Software Product Lines is very challenging due to a high degree of variability leading to an enormous number of possible products. The vast majority of todays testing approaches for Software Product Lines validate products individually using different kinds of reuse techniques for testing. Due to the enormous number of possible products, individual product testing becomes more and more unfeasible. Combinatorial testing offers one possibility to test a subset of all possible products. In this contribution we provide a detailed description of a methodology to apply combinatorial testing to a feature model of a Software Product Line. We combine graph transformation, combinatorial testing, and forward checking for that purpose. Additionally, our approach considers predefined sets of products.

Pairwise testing for software product lines: comparison of two approaches

Software Product Lines (SPL) are difficult to validate due to combinatorics induced by variability, which in turn leads to combinatorial explosion of the number of derivable products. Exhaustive testing in such a large products space is hardly feasible. Hence, one possible option is to test SPLs by generating test configurations that cover all possible t feature interactions (t-wise). It dramatically reduces the number of test products while ensuring reasonable SPL coverage. In this paper, we report our experience on applying t-wise techniques for SPL with two independent toolsets developed by the authors. One focuses on generality and splits the generation problem according to strategies. The other emphasizes providing efficient generation. To evaluate the respective merits of the approaches, measures such as the number of generated test configurations and the similarity between them are provided. By applying these measures, we were able to derive useful insights for pairwise and t-wise testing of product lines.

Model-based pairwise testing for feature interaction coverage in software product line engineering

Testing software product lines (SPLs) is very challenging due to a high degree of variability leading to an enormous number of possible products. The vast majority of today’s testing approaches for SPLs validate products individually using different kinds of reuse techniques for testing. Because of their reusability and adaptability capabilities, model-based approaches are suitable to describe variability and are therefore frequently used for implementation and testing purposes of SPLs. Due to the enormous number of possible products, individual product testing becomes more and more infeasible. Pairwise testing offers one possibility to test a subset of all possible products. However, according to the best of our knowledge, there is no contribution discussing and rating this approach in the SPL context. In this contribution, we provide a mapping between feature models describing the common and variable parts of an SPL and a reusable test model in the form of statecharts. Thereby, we interrelate feature model-based coverage criteria and test model-based coverage criteria such as control and data flow coverage and are therefore able to discuss the potentials and limitations of pairwise testing. We pay particular attention to test requirements for feature interactions constituting a major challenge in SPL engineering. We give a concise definition of feature dependencies and feature interactions from a testing point of view, and we discuss adequacy criteria for SPL coverage under pairwise feature interaction testing and give a generalization to the T-wise case. The concept and implementation of our approach are evaluated by means of a case study from the automotive domain.

MoSo-PoLiTe: tool support for pairwise and model-based software product line testing

Testing Software Product Lines is a very challenging task and approaches like combinatorial testing and model-based testing are frequently used to reduce the effort of testing Software Product Lines and to reuse test artifacts. In this contribution we present a tool chain realizing our MoSo-PoLiTe concept combining combinatorial and model-based testing. Our tool chain contains a pairwise configuration selection component on the basis of a feature model. This component implements an heuristic finding a minimal subset of configurations covering 100% pairwise interaction. Additionally, our tool chain allows the model-based test case generation for each configuration within this generated subset. This tool chain is based on commercial tools since it was developed within industrial cooperations. A non-commercial implementation of pairwise configuration selection is available and an integration with an Open Source model-based testing tool is under development.

Model-based coverage-driven test suite generation for software product lines

Software Product Line (SPL) engineering is a popular approach for the systematic reuse of software artifacts across a large number of similar products. Unfortunately, testing each product of an SPL separately is often unfeasible. Consequently, SPL engineering is in conflict with standards like ISO 26262, which require each installed software configuration of safety-critical SPLs to be tested using a model-based approach with well-defined coverage criteria. In this paper we address this dilemma and present a new SPL test suite generation algorithm that uses model-based testing techniques to derive a small test suite from one variable 150% test model of the SPL such that a given coverage criterion is satisfied for the test model of every product. Furthermore, our algorithm simplifies the subsequent selection of a small, representative set of products (w.r.t. the given coverage criterion) on which the generated test suite can be executed.

Pairwise feature-interaction testing for SPLs: potentials and limitations

A fundamental problem of testing Software Product Lines (SPLs) is that variability enables the production of a large number of instances and it is difficult to construct and run test cases even for SPLs with a small number of variable features. Interacting features is a foundation of a fault model for SPLs, where faults are likely to be revealed at execution points where features exchange information with other features or influence one another. Therefore, a test adequacy criterion is to cover as many interactions among different features as possible, thus increasing the probability of finding bugs. Our approach combines a combinatorial designs algorithm for pairwise feature generation with model-based testing to reduce the size of the SPL required for comprehensive coverage of interacting features. We implemented our approach and applied it to an SPL from the automotive domain provided by one of our industrial partners. The results suggest that with our approach higher coverage of feature interactions is achieved at a fraction of cost when compared with a baseline approach of testing all feature interactions.

Reducing feature models to improve runtime adaptivity on resource limited devices

Mobile devices like smartphones are getting increasingly important in our daily lifes. They are used in various environments and have to dynamically adapt themselves accordingly in order to provide an optimal runtime behavior. Naturally, adapting to continuously changing environmental conditions is a challenging task because mobile devices are always limited in their resources and have to adapt in real-time. In this paper, we introduce an approach that enables resource limited devices to adapt to changing conditions using dynamic software product lines techniques. Therefore, feature models are reduced to a specific hardware context before installing the adaptive mobile application on the device. This reduces the amount of possible configurations that are compatible with the device and, thereby, minimizes the costs and the duration of an adaptation during runtime.

Industrial evaluation of pairwise SPL testing with MoSo-PoLiTe

Testing Software Product Lines is a very challenging task due to variability. Frequently, approaches such as combinatorial testing are used to generate representative sets of products for testing purposes instead of testing each individual product of the SPL under test. In this contribution we present the results of applying the MoSo-PoLiTe framework at Danfoss Power Electronics A/S to calculate a representative set of product configurations for black box testing purposes. Within this evaluation we use MoSo-PoLiTes pairwise configuration selection component on the basis of a feature model. This component implements a heuristics finding a minimal subset of configurations covering 100% T-wise feature interaction. According to the best of our knowledge, this is the first publication providing industrial results about pairwise SPL testing.

Model-Driven Software Product Line Testing: An Integrated Approach

Software Product Line engineering is a popular approach which improves reusability of software in a large number of products that share a common set of features. Feature Models (FMs) are often used to model commonalities and variabilities within a Software Product Line (SPL). Due to their variability, testing SPLs is very challenging and many different approaches exist. Classification Trees (CTs) are a well-known and in practice popular black-box approach for the systematic derivation of a set of test cases of a single software system instance. In this paper, we explore the relations and similarities between FMs and CTs. Our contribution is the introduction of an integrated approach Feature Model for Testing (FMT) marrying properties and abilities of CTs and FMs.

Model-driven rapid prototyping with programmed graph transformations

Modern software systems are constantly increasing in complexity and supporting the rapid prototyping of such systems has become crucial to check the feasibility of extensions and optimizations, thereby reducing risks and, consequently, the cost of development. As modern software systems are also expected to be reused, extended, and adapted over a much longer lifetime than ever before, ensuring the maintainability of such systems is equally gaining relevance. In this paper, we present the development, optimization and maintenance of MoSo-PoLiTe, a framework for Software Product Line (SPL) testing, as a novel case study for rapid prototyping via metamodelling and programmed graph transformations. The first part of the case study evaluates the use of programmed graph transformations for optimizing an existing, hand-written system (MoSo-PoLiTe) via rapid prototyping of various strategies. In the second part, we present a complete re-engineering of the hand-written system with programmed graph transformations and provide a critical comparison of both implementations. Our results and conclusions indicate that metamodelling and programmed graph transformation are not only suitable techniques for rapid prototyping, but also lead to more maintainable systems.