Paolo Vavassori

Generating Tests for Detecting Faults in Feature Models

We present a novel fault-based approach for testing feature models (FMs). We identify several fault classes that represent possible mistakes one can make during feature modeling. We introduce the concept of distinguishing configuration, i.e., a configuration that is able to detect a given fault. Starting from this definition, we devise a technique, based on the use of a logic solver, able either to find distinguishing configurations to be used as tests or to prove that a mutation produces an equivalent feature model. Compact test suites can be produced by exploiting an SMT solver. The experiments show that our methodology is viable and produces reasonable sized test suites in a short time. W.r.t. the approaches that use only the products, our approach has a better fault detection capability and requires fewer tests.

Combinatorial Interaction Testing with CITLAB

In this paper the CITLAB tool for Combinatorial Interaction Testing is presented. The tool allows importing/exporting models of combinatorial problems from/to different application domains, by means of a common interchange syntax notation and a corresponding interoperable semantic metamodel. Moreover, the tool is a framework allowing embedding and transparent invocation of multiple, different implementations of combinatorial algorithms. CITLAB has been designed tightly integrated with the Eclipse IDE framework, by means of its plug-in extension mechanism. It is intended to easy the spread of CIT testing both in industrial practice and in academic research, by allowing users and researchers to apply multiple test suite generation algorithms, each with its peculiarities, on the same problem models, and let them compare the results in order to select the one that best fits their needs, while alleviating from the pain of knowing all the different details and notations of the underlying CIT tools.

CITLAB: A Laboratory for Combinatorial Interaction Testing

Although the research community around combinatorial interaction testing has been very active for several years, it has failed to find common solutions on some issues. First of all, there is not a common abstract nor concrete language to express combinatorial problems. Combinatorial testing generator tools are strongly decoupled making difficult their interoperability and the exchange of models and data. In this paper, we propose an abstract and concrete specific language for combinatorial problems. It features and formally defines the concepts of parameters and types, constraints, seeds, and test goals. The language is defined by means of XTEXT, a framework for the definition of domain-specific languages. XTEXT is used to derive a powerful editor integrated with eclipse and with all the expected features of a modern editor. Eclipse is also used to build an extensible framework in which test generators, importers, and exporters can be easily added as plugins.

Combinatorial Testing for Feature Models Using CitLab

Feature models are commonly used to represent product lines and systems with a set of features interrelated each others. Test generation from feature models, i.e. generating a valid and representative subset of all the possible product configurations, is still an open challenge. A common approach is to build combinatorial interaction test suites, for instance achieving pairwise coverage among the features. In this paper we show how standard feature models can be translated to combinatorial interaction models in our framework CITLAB, with all the advantages of having a combinatorial testing environment (in terms of a clear semantics, editing facilities, language for seeds and test goals, and generation algorithms). We present our translation which gives a precise semantics to feature models and it tries to minimize the number of parameter and constraints while preserving the original semantics of the feature model. Experiments show the advantages of our approach.

Automatic Detection and Removal of Conformance Faults in Feature Models

Building a feature model for an existing SPL can improve the automatic analysis of the SPL and reduce the effort in maintenance. However, developing a feature model can be error prone, and checking that it correctly identifies each actual product of the SPL may be unfeasible due to the huge number of possible configurations. We apply mutation analysis and propose a method to detect and remove conformance faults by selecting special configurations that distinguish a feature model from its mutants. We propose a technique that, by iterating this process, is able to repair a faulty model. We devise several variations of a simple hill climbing algorithm for automatic fault removal and we compare them by a series of experiments on three different sets of feature models. We find that our technique is able to improve the conformance of around 90% of the models and find the correct model in around 40% of the cases.

Validation of Models and Tests for Constrained Combinatorial Interaction Testing

In Combinatorial Interaction Testing, models specify a set of parameters with associated domains, and some constraints over the parameters. Test generation tools produce, starting from these models, test suites for achieving some given coverage criteria. The validation of both the models and the produced test suites is a worthwhile activity. Validating the models permits to early discover possible defects in them, to feed the test generations tools with good inputs and, possibly, to improve the quality of the testing process. Validating the produced test suites, instead, permits to check if the test generation tools are correct and to judge their quality. This paper proposes to validate the models by checking that the constraints are consistent, that there is no constraint implied by the other constraints, and that the parameters and their values are really necessary. The proposed test suite validation, instead, consists in checking that the tests respect the type definitions and the constraints, that all the test requirements are covered, and that all the tests in the test suite are valid and necessary. For every error we propose a possible technique able to identify the potential causes and to suggest fixes for those problems. Experiments show that the targeted defects are widespread both in benchmark and real-life models.

Efficient Combinatorial Test Generation Based on Multivalued Decision Diagrams

Combinatorial interaction testing (CIT) is an emerging testing technique that has proved to be effective in finding faults due to the interaction among inputs. Efficient test generation for CIT is still an open problem especially when applied to real models having meaningful size and containing many constraints among inputs. In this paper we present a novel technique for the automatic generation of compact test suites starting from models containing constraints given in general form. It is based on the use of Multivalued Decision Diagrams (MDDs) which prove to be suitable to efficiently support CIT. We devise and experiment several optimizations including a novel variation of the classical greedy policy normally used in similar algorithms. The results of a thorough comparison with other similar techniques are presented and show that our approach can provide several advantages in terms of applicability, test suite size, generation time, and cost.

Validation of Constraints Among Configuration Parameters Using Search-Based Combinatorial Interaction Testing

The appeal of highly-configurable software systems lies in their adaptability to users’ needs. Search-based Combinatorial Interaction Testing (CIT) techniques have been specifically developed to drive the systematic testing of such highly-configurable systems. In order to apply these, it is paramount to devise a model of parameter configurations which conforms to the software implementation. This is a non-trivial task. Therefore, we extend traditional search-based CIT by devising 4 new testing policies able to check if the model correctly identifies constraints among the various software parameters. Our experiments show that one of our new policies is able to detect faults both in the model and the software implementation that are missed by the standard approaches.

Rehabilitating equivalent mutants as static anomaly detectors in software artifacts

In mutation analysis a mutant is said equivalent if it leaves the semantics of the program or the model unchanged. Equivalent mutants are usually seen as an inconvenience; for example, in software testing they cannot be detected by a test and, therefore, they fictitiously reduce the mutation score of a test suite. In this paper, instead, equivalent mutants are seen as an opportunity, since they can be used to find some static anomalies of software artifacts, i.e., anomalies that can be removed without affecting the artifact semantics. The proposal is applicable to different kinds of software artifacts as source code, Boolean expressions, and feature models.

A novel use of equivalent mutants for static anomaly detection in software artifacts

Equivalent mutants are usually seen as an inconvenience in mutation analysis.We claim that equivalent mutants can be useful to detect and remove static anomalies.A process for detecting static anomalies is proposed.The process is based on mutation, equivalence checking, and quality measurement.The process is applicable to different kinds of software artifacts. Context: In mutation analysis, a mutant of a software artifact, either a program or a model, is said equivalent if it leaves the artifact meaning unchanged. Equivalent mutants are usually seen as an inconvenience and they reduce the applicability of mutation analysis.Objective: Instead, we here claim that equivalent mutants can be useful to define, detect, and remove static anomalies, i.e., deficiencies of given qualities: If an equivalent mutant has a better quality value than the original artifact, then an anomaly has been found and removed.Method: We present a process for detecting static anomalies based on mutation, equivalence checking, and quality measurement.Results: Our proposal and the originating technique are applicable to different kinds of software artifacts. We present anomalies and conduct several experiments in different contexts, at specification, design, and implementation level.Conclusion: We claim that in mutation analysis a new research direction should be followed, in which equivalent mutants and operators generating them are welcome.