Benoit Baudry

Improving test suites for efficient fault localization

The need for testing-for-diagnosis strategies has been identified for a long time, but the explicit link from testing to diagnosis (fault localization) is rare. Analyzing the type of information needed for efficient fault localization, we identify the attribute (called Dynamic Basic Block) that restricts the accuracy of a diagnosis algorithm. Based on this attribute, a test-for-diagnosis criterion is proposed and validated through rigorous case studies: it shows that a test suite can be improved to reach a high level of diagnosis accuracy. So, the dilemma between a reduced testing effort (with as few test cases as possible) and the diagnosis accuracy (that needs as much test cases as possible to get more information) is partly solved by selecting test cases that are dedicated to diagnosis.

Software Reliability Modeling with Test Coverage: Experimentation and Measurement with A Fault-Tolerant Software Project

As the key factor in software quality, software reliability quantifies software failures. Traditional software reliability growth models use the execution time during testing for reliability estimation. Although testing time is an important factor in reliability, it is likely that the prediction accuracy of such models can be further improved by adding other parameters which affect the final software quality. Meanwhile, in software testing, test coverage has been regarded as an indicator for testing completeness and effectiveness in the literature. In this paper, we propose a novel method to integrate time and test coverage measurements together to predict the reliability. The key idea is that failure detection is not only related to the time that the software experiences under testing, but also to what fraction of the code has been executed by the testing. This is the first time that execution time and test coverage are incorporated together into one single mathematical form to estimate the reliability achieved. We further extend this method to predict the reliability of fault- tolerant software systems. The experimental results with multi-version software show that our reliability model achieves a substantial estimation improvement compared with existing reliability models.

A Generic Approach for Automatic Model Composition

Analyzing and modelling a software system with separate views is a good practice to deal with complexity and maintainability. When adopting such a modular approach for modelling, it is necessary to have the ability to automatically compose models to build a global view of the system. In this paper we propose a generic framework for composition that is independent from a modelling language. We define a process for adapting this framework to particular modelling language (defined with a metamodel) and illustrate how the generic composition has been specialized for class diagrams.

Barriers to systematic model transformation testing

Introduction Model Driven Engineering (MDE) techniques support extensive use of models in order to manage the increasing complexity of software systems. Appropriate abstractions of software system elements can ease reasoning and understanding and thus limit the risk of errors in large systems. Automatic model transformations play a critical role in MDE since they automate complex, tedious, error-prone, and recurrent software development tasks. Airbus uses automatic code synthesis from SCADE models to generate the code for embedded controllers in the Airbus A380. Commercial tools for model transformations exist. Objecteering and Together from Borland are tools that can automatically add design patterns in a UML class model. Esterel Technologies have a tool for automatic code synthesis for safety critical systems. Other examples of transformations are refinement of a design model by adding details pertaining to a particular target platform, refactoring a model by changing its structure to enhance design quality, or reverse engineering code to obtain an abstract model. These software development tasks are critical and thus the model transformations that automate them must be validated. A fault in a transformation can introduce a fault in the transformed model, which if undetected and not removed, can propagate to other models in successive development steps. As a fault propagates across transformations, it becomes more difficult to detect and isolate. Since model transformations are meant to be reused, faults present in them may result in many faulty models. Model transformations constitute a class of programs with unique characteristics that make testing them challenging. The complexity of input and output data, lack of model management tools, and the heterogeneity of transformation languages pose special problems to testers of transformations. In this paper we identify current model transformation characteristics that contribute to the difficulty of systematically testing transformations. We present promising solutions and propose possible ways to overcome these barriers.

Validation in model-driven engineering: testing model transformations

The OMGs model-driven architecture is quickly attracting attention as a method of constructing systems that offers advantages over traditional approaches in terms of reliability, consistency, and maintainability. The key concepts in the MDA are models that are related by model transformations. However, for the MDA to provide an adequate alternative to existing approaches, it must offer comparable support for software engineering processes such as requirements analysis, design and testing. This paper attempts to explore the application of the last of these processes, testing, to the most novel part of the MDA, that of model transformation. We present a general view of the roles of testing in the different stages of model-driven development, and a more detailed exploration of approaches to testing model transformations. Based on this, we highlight the particular issues for the different testing tasks, including adequacy criteria, test oracles and automatic test data generation. We also propose possible approaches for the testing tasks, and show how existing functional and structural testing techniques can be adapted for use in this new development context.

Pairwise testing for software product lines: comparison of two approaches

Software Product Lines (SPL) are difficult to validate due to combinatorics induced by variability, which in turn leads to combinatorial explosion of the number of derivable products. Exhaustive testing in such a large products space is hardly feasible. Hence, one possible option is to test SPLs by generating test configurations that cover all possible t feature interactions (t-wise). It dramatically reduces the number of test products while ensuring reasonable SPL coverage. In this paper, we report our experience on applying t-wise techniques for SPL with two independent toolsets developed by the authors. One focuses on generality and splits the generation problem according to strategies. The other emphasizes providing efficient generation. To evaluate the respective merits of the approaches, measures such as the number of generated test configurations and the similarity between them are provided. By applying these measures, we were able to derive useful insights for pairwise and t-wise testing of product lines.

Mutation analysis testing for model transformations

In MDE, model transformations should be efficiently tested so that it may be used and reused safely. Mutation analysis is an efficient technique to evaluate the quality of test data, and has been extensively studied both for procedural and object-oriented languages. In this paper, we study how it can be adapted to model oriented programming. Since no model transformation language has been widely accepted today, we propose generic fault models that are related to the model transformation process. First, we identify abstract operations that constitute this process: model navigation, models elements filtering, output model creation and input model modification. Then, we propose a set of specific mutation operators which are directly inspired from these operations. We believe that these operators are meaningful since a large part of the errors in a transformation are due to the manipulation of complex models regardless of the concrete implementation language.

Automatic Model Generation Strategies for Model Transformation Testing

Testing model transformations requires input models which are graphs of inter-connected objects that must conform to a meta-model and meta-constraints from heterogeneous sources such as well-formedness rules, transformation pre-conditions, and test strategies. Manually specifying such models is tedious since models must simultaneously conform to several meta-constraints. We propose automatic model generation via constraint satisfaction using our tool Cartier for model transformation testing. Due to the virtually infinite number of models in the input domain we compare strategies based on input domain partitioning to guide model generation. We qualify the effectiveness of these strategies by performing mutation analysis on the transformation using generated sets of models. The test sets obtained using partitioning strategies gives mutation scores of up to 87% vs. 72% in the case of unguided/random generation. These scores are based on analysis of 360 automatically generated test models for the representative transformation of UML class diagram models to RDBMS models.

Meta-model Pruning

Large and complex meta-models such as those of Uml and its profiles are growing due to modelling and inter-operability needs of numerous stakeholders. The complexity of such meta-models has led to coining of the term meta-muddle . Individual users often exercise only a small view of a meta-muddle for tasks ranging from model creation to construction of model transformations. What is the effective meta-model that represents this view? We present a flexible meta-model pruning algorithm and tool to extract effective meta-models from a meta-muddle. We use the notion of model typing for meta-models to verify that the algorithm generates a super-type of the large meta-model representing the meta-muddle. This implies that all programs written using the effective meta-model will work for the meta-muddle hence preserving backward compatibility. All instances of the effective meta-model are also instances of the meta-muddle. We illustrate how pruning the original Uml meta-model produces different effective meta-models.

Providing Support for Model Composition in Metamodels

In aspect-oriented modeling (AOM), a design is described using a set of design views. It is sometimes necessary to compose the views to obtain an integrated view that can be analyzed by tools. Analysis can uncover conflicts and interactions that give rise to undesirable emergent behavior. Design models tend to have complex structures and thus manual model composition can be arduous and error- prone. Tools that automate significant parts of model composition are needed if AOM is to gain industrial acceptance. One way of providing automated support for composing models written in a particular language is to define model composition behavior in the metamodel defining the language. In this paper we show how this can be done by extending the UML metamodel with behavior describing symmetric, signature-based composition of UML model elements. We also describe an implementation of the metamodel that supports systematic composition of UML class models.