Sagar Sen

Pairwise testing for software product lines: comparison of two approaches

Software Product Lines (SPL) are difficult to validate due to combinatorics induced by variability, which in turn leads to combinatorial explosion of the number of derivable products. Exhaustive testing in such a large products space is hardly feasible. Hence, one possible option is to test SPLs by generating test configurations that cover all possible t feature interactions (t-wise). It dramatically reduces the number of test products while ensuring reasonable SPL coverage. In this paper, we report our experience on applying t-wise techniques for SPL with two independent toolsets developed by the authors. One focuses on generality and splits the generation problem according to strategies. The other emphasizes providing efficient generation. To evaluate the respective merits of the approaches, measures such as the number of generated test configurations and the similarity between them are provided. By applying these measures, we were able to derive useful insights for pairwise and t-wise testing of product lines.

Automatic Model Generation Strategies for Model Transformation Testing

Testing model transformations requires input models which are graphs of inter-connected objects that must conform to a meta-model and meta-constraints from heterogeneous sources such as well-formedness rules, transformation pre-conditions, and test strategies. Manually specifying such models is tedious since models must simultaneously conform to several meta-constraints. We propose automatic model generation via constraint satisfaction using our tool Cartier for model transformation testing. Due to the virtually infinite number of models in the input domain we compare strategies based on input domain partitioning to guide model generation. We qualify the effectiveness of these strategies by performing mutation analysis on the transformation using generated sets of models. The test sets obtained using partitioning strategies gives mutation scores of up to 87% vs. 72% in the case of unguided/random generation. These scores are based on analysis of 360 automatically generated test models for the representative transformation of UML class diagram models to RDBMS models.

Meta-model Pruning

Large and complex meta-models such as those of Uml and its profiles are growing due to modelling and inter-operability needs of numerous stakeholders. The complexity of such meta-models has led to coining of the term meta-muddle . Individual users often exercise only a small view of a meta-muddle for tasks ranging from model creation to construction of model transformations. What is the effective meta-model that represents this view? We present a flexible meta-model pruning algorithm and tool to extract effective meta-models from a meta-muddle. We use the notion of model typing for meta-models to verify that the algorithm generates a super-type of the large meta-model representing the meta-muddle. This implies that all programs written using the effective meta-model will work for the meta-muddle hence preserving backward compatibility. All instances of the effective meta-model are also instances of the meta-muddle. We illustrate how pruning the original Uml meta-model produces different effective meta-models.

On Combining Multi-formalism Knowledge to Select Models for Model Transformation Testing

Testing remains a major challenge for model transformation development. Test models that are used as test data for model transformations, are constrained by various sources of knowledge that is expressed in different formalisms. Thus, in order to automatically generate test models it is necessary to interpret these different sources of knowledge and combine them into a consistent set of information that can be used for model synthesis. In this paper, we identify sources of testing knowledge and present our tool Cartier that uses Alloy as the first-order relational logic language to represent combined knowledge in the form of constraints. The constraints are solved leading to a selection of qualified test models from the input domain of a model transformation. We illustrate our approach using the Unified Modeling Language class diagram to relational database management systems transformation as a running example.

Reusable model transformations

Model transformations written for an input metamodel may often apply to other metamodels that share similar concepts. For example, a transformation written to refactor Java models can be applicable to refactoring UML class diagrams as both languages share concepts such as classes, methods, attributes, and inheritance. Deriving motivation from this example, we present an approach to make model transformations reusable such that they function correctly across several similar metamodels. Our approach relies on these principal steps: (1) We analyze a transformation to obtain an effective subset of used concepts. We prune the input metamodel of the transformation to obtain an effective input metamodel containing the effective subset. The effective input metamodel represents the true input domain of transformation. (2) We adapt a target input metamodel by weaving it with aspects such as properties derived from the effective input metamodel. This adaptation makes the target metamodel a subtype of the effective input metamodel. The subtype property ensures that the transformation can process models conforming to the target input metamodel without any change in the transformation itself. We validate our approach by adapting well known refactoring transformations (Encapsulate Field, Move Method, and Pull Up Method) written for an in-house domain-specific modeling language (DSML) to three different industry standard metamodels (Java, MOF, and UML).

Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study

Regression testing in continuous integration environment is bounded by tight time constraints. To satisfy time constraints and achieve testing goals, test cases must be efficiently ordered in execution. Prioritization techniques are commonly used to order test cases to reflect their importance according to one or more criteria. Reduced time to test or high fault detection rate are such important criteria. In this paper, we present a case study of a test prioritization approach ROCKET (Prioritization for Continuous Regression Testing) to improve the efficiency of continuous regression testing of industrial video conferencing software. ROCKET orders test cases based on historical failure data, test execution time and domain-specific heuristics. It uses a weighted function to compute test priority. The weights are higher if tests uncover regression faults in recent iterations of software testing and reduce time to detection of faults. The results of the study show that the test cases prioritized using ROCKET (1) provide faster fault detection, and (2) increase regression fault detection rate, revealing 30% more faults for 20% of the test suite executed, comparing to manually prioritized test cases.

Towards Domain-specific Model Editors with Automatic Model Completion

Integrated development environments such as Eclipse allow users to write programs quickly by presenting a set of recommendations for code completion. Similarly, word processing tools such as Microsoft Word present corrections for grammatical errors in sentences. Both of these existing structure editors use a set of constraints expressed in the form of a natural language grammar to restrict/correct the user ( syntax-directed editing) or formal grammar (language-directed editing ) to aid document completion. Taking this idea further, in this paper we present an integrated software system capable of generating recommendations for model completion of partial models built in editors for domain-specific modeling languages. We present a methodology to synthesize model editors equipped with automatic completion from a modeling language’s declarative specification consisting of a meta-model with a visual syntax. This meta-model directed completion feature is powered by a first-order relational logic engine implemented in ALLOY. We incorporate automatic completion in the generative tool AToM3. We use the finite state machines modeling language as a concise running example. Our approach leverages a correct by construction philosophy that renders subsequent simulation of models considerably less error-prone.

Practical pairwise testing for software product lines

One key challenge for software product lines is efficiently managing variability throughout their lifecycle. In this paper, we address the problem of variability in software product lines testing. We (1) identify a set of issues that must be addressed to make software product line testing work in practice and (2) provide a framework that combines a set of techniques to solve these issues. The framework integrates feature modelling, combinatorial interaction testing and constraint programming techniques. First, we extract variability in a software product line as a feature model with specified feature interdependencies. We then employ an algorithm that generates a minimal set of valid test cases covering all 2-way feature interactions for a given time interval. Furthermore, we evaluate the framework on an industrial SPL and show that using the framework saves time and provides better test coverage. In particular, our experiments show that the framework improves industrial testing practice in terms of (i) 17% smaller set of test cases that are (a) valid and (b) guarantee all 2-way feature coverage (as opposite to 19.2% 2-way feature coverage in the hand made test set), and (ii) full flexibility and adjustment of test generation to available testing time.

Static Analysis of Model Transformations for Effective Test Generation

Model transformations are an integral part of several computing systems that manipulate interconnected graphs of objects called models in an input domain specified by a metamodel and a set of invariants. Test models are used to look for faults in a transformation. A test model contains a specific set of objects, their interconnections and values for their attributes. Can we automatically generate an effective set of test models using knowledge from the transformation? We present a white-box testing approach that uses static analysis to guide the automatic generation of test inputs for transformations. Our static analysis uncovers knowledge about how the input model elements are accessed by transformation operations. This information is called the input metamodel footprint due to the transformation. We transform footprint, input metamodel, its invariants, and transformation pre-conditions to a constraint satisfaction problem in Alloy. We solve the problem to generate sets of test models containing traces of the footprint. Are these test models effective? With the help of a case study transformation we evaluate the effectiveness of these test inputs. We use mutation analysis to show that the test models generated from footprints are more effective (97.62% avg. mutation score) in detecting faults than previously developed approaches based on input domain coverage criteria (89.9% avg.) and unguided generation (70.1% avg.).

Using models of partial knowledge to test model transformations

Testers often use partial knowledge to build test models. This knowledge comes from sources such as requirements, known faults, existing inputs, and execution traces. In Model-Driven Engineering, test inputs are models executed by model transformations. Modelers build them using partial knowledge while meticulously satisfying several well-formedness rules imposed by the modelling language. This manual process is tedious and language constraints can force users to create complex models even for representing simple knowledge. In this paper, we want to simplify the development of test models by presenting an integrated methodology and semi-automated tool that allow users to build only small partial test models directly representing their testing intent. We argue that partial models are more readable and maintainable and can be automatically completed to full input models while considering language constraints. We validate this approach by evaluating the size and fault-detecting effectiveness of partial models compared to traditionally-built test models. We show that they can detect the same bugs/faults with a greatly reduced development effort.