Remo Lachmann

Delta-oriented model-based integration testing of large-scale systems

Software architecture specifications are of growing importance for coping with the complexity of large-scale systems. They provide an abstract view on the high-level structural system entities together with their explicit dependencies and build the basis for ensuring behavioral conformance of component implementations and interactions, e.g., using model-based integration testing. The increasing inherent diversity of such large-scale variant-rich systems further complicates quality assurance. In this article, we present a combination of architecture-driven model-based testing principles and regression-inspired testing strategies for efficient, yet comprehensive variability-aware conformance testing of variant-rich systems. We propose an integrated delta-oriented architectural test modeling and testing approach for component as well as integration testing that allows the generation and reuse of test artifacts among different system variants. Furthermore, an automated derivation of retesting obligations based on accurate delta-oriented architectural change impact analysis is provided. Based on a formal conceptual framework that guarantees stable test coverage for every system variant, we present a sample implementation of our approach and an evaluation of the validity and efficiency by means of a case study from the automotive domain.

Delta-oriented test case prioritization for integration testing of software product lines

Software product lines have potential to allow for mass customization of products. Unfortunately, the resulting, vast amount of possible product variants with commonalities and differences leads to new challenges in software testing. Ideally, every product variant should be tested, especially in safety-critical systems. However, due to the exponentially increasing number of product variants, testing every product variant is not feasible. Thus, new concepts and techniques are required to provide efficient SPL testing strategies exploiting the commonalities of software artifacts between product variants to reduce redundancy in testing. In this paper, we present an efficient integration testing approach for SPLs based on delta modeling. We focus on test case prioritization. As a result, only the most important test cases for every product variant are tested, reducing the number of executed test cases significantly, as testing can stop at any given point because of resource constraints while ensuring that the most important test cases have been covered. We present the general concept and our evaluation results. The results show a measurable reduction of executed test cases compared to single-software testing approaches.

Fine-grained test case prioritization for integration testing of delta-oriented software product lines

Software product line (SPL) testing is a challenging task, due to the huge number of variants sharing common functionalities to be taken into account for efficient testing. By adopting the concept of regression testing, incremental SPL testing strategies cope with this challenge by exploiting the reuse potential of test artifacts between subsequent variants under test. In previous work, we proposed delta-oriented test case prioritization for incremental SPL integration testing, where differences between architecture test model variants allow for reasoning about the order of reusable test cases to be executed. However, the prioritization left two issues open, namely (1) changes to component behavior are ignored, which may also influence component interactions and, (2) the weighting and ordering of similar test cases result in an unintended clustering of test cases. In this paper, we extend the test case prioritization technique by (1) incorporating changes to component behavior allowing for a more fine-grained analysis and (2) defining a dissimilarity measure to avoid clustered test case orders. We prototyped our test case prioritization technique and evaluated its applicability and effectiveness by means of a case study from the automotive domain showing positive results.

Requirements-based delta-oriented SPL testing

Variability of modern software systems increases potential sources of errors and demands appropriate quality assurance strategies. In order to reduce the test effort when testing software product lines, incremental model-based testing strategies have been proposed, based on the conceptual ideas of delta modeling. It requires executable system specifications to derive and classify test cases. However, in industrial practice such system models rarely exist, but requirements and test cases are captured in natural language. In order to make delta-oriented testing strategies applicable in this context, we transfer them to the requirements level and show how a delta-oriented classification of requirements and associated test cases reduce test effort in these less formal domains.

Delta-oriented product prioritization for similarity-based product-line testing

Testing every product of a software product line (SPL) is often not feasible due to the exponential number of products in the number of features. Thus, the order in which products are tested matters, because it can increase the early rate of fault detection. Several approaches have been proposed to prioritize products based on configuration similarity. However, current approaches are oblivious to solution-space differences among products, because they consider only problem-space information. With delta modeling, we incorporate solution-space information in product prioritization to improve the effectiveness of SPL testing. Deltas capture the differences between products facilitating the reasoning about product similarity. As a result, we select the most dissimilar product to the previously tested ones, in terms of deltas, to be tested next. We evaluate the effectiveness of our approach using an SPL from the automotive domain showing an improvement in the effectiveness of SPL testing.

Risk-based integration testing of software product lines

Software product lines (SPL) capture commonalities and variabilities of product families and, thus, enable mass customization of product variants according to customers desired configurations. However, they introduce new challenges to software testing due to a potentially large number of variants. While each variant should be tested, testing resources are limited and, thus, a retest of all, partially redundant, test cases for each variant is not feasible in SPL testing. Coping with these issues has been a major research focus in recent years, leading to different testing approaches. However, risk-based testing has not gained much attention in the SPL domain while being a successful approach for single-software systems. In this paper, we propose a novel risk-based testing approach for SPL integration testing. We incrementally test SPLs by stepping from one variant to the next. For each variant, we automatically compute failure probabilities and failure impacts for its architectural components. To avoid a computational overhead of generating and analyzing each variant, we exploit the variability between variants defined as deltas to focus on important changes. We evaluate our approach using an automotive case study, showing that the risk-based technique leads to positive results compared to random and delta-oriented testing.

Multi-objective black-box test case selection for system testing

Testing is a fundamental task to ensure software quality. Regression testing aims to ensure that changes to software do not introduce new failures. As resources are often limited and testing comprises a vast amount of test cases, different regression strategies have been proposed to reduce testing effort by selecting or prioritizing important test cases, e.g., code coverage (to ensure a sufficient testing depth). However, in system testing, source code is often not available creating a black-box system. In this paper, we introduce an automated, multi-objective test case selection technique in black-box systems using genetic algorithms. We define seven different objectives, based on meta-data, allowing a flexible test case selection for a variety of systems. For evaluation, we apply our technique on two different subject systems assessing the feasibility and suitability of our test case selection approach. Results indicate that our approach is applicable based on different data available and is able to outperform random test case selection and retest-all.

System-Level Test Case Prioritization Using Machine Learning

Regression testing is the common task of retesting software that has been changed or extended (e.g., by new features) during software evolution. As retesting the whole program is not feasible with reasonable time and cost, usually only a subset of all test cases is executed for regression testing, e.g., by executing test cases according to test case prioritization. Although a vast amount of methods for test case prioritization exist, they mostly require access to source code (i.e., white-box). However, in industrial practice, system-level testing is an important task that usually grants no access to source code (i.e., black-box). Hence, for an effective regression testing process, other information has to be employed. In this paper, we introduce a novel technique for test case prioritization for manual system-level regression testing based on supervised machine learning. Our approach considers black-box meta-data, such as test case history, as well as natural language test case descriptions for prioritization. We use the machine learning algorithm SVM Rank to evaluate our approach by means of two subject systems and measure the prioritization quality. Our results imply that our technique improves the failure detection rate significantly compared to a random order. In addition, we are able to outperform a test case order given by a test expert. Moreover, using natural language descriptions improves the failure finding rate.