Axel Hollmann

Test generation and minimization with "basic" statecharts

Model-based testing as a black-box testing technique has grown in importance. The models used represent the relevant features of the system under consideration (SUC), and can also be used as a basis for generating test case sets. In this work we introduce a novel representation of state-charts which subsumes common features of different state-chart variants. Based on this model and well-defined test criteria, efficient algorithms are introduced for generating test case sets. Those test case sets are minimized to cover both the model of SUC and its inversion, i.e., the complementary model.

Model-based mutation testing-Approach and case studies

This paper rigorously introduces the concept of model-based mutation testing (MBMT) and positions it in the landscape of mutation testing. Two elementary mutation operators, insertion and omission, are exemplarily applied to a hierarchy of graph-based models of increasing expressive power including directed graphs, event sequence graphs, finite-state machines and statecharts. Test cases generated based on the mutated models (mutants) are used to determine not only whether each mutant can be killed but also whether there are any faults in the corresponding system under consideration (SUC) developed based on the original model. Novelties of our approach are: (1) evaluation of the fault detection capability (in terms of revealing faults in the SUC) of test sets generated based on the mutated models, and (2) superseding of the great variety of existing mutation operators by iterations and combinations of the two proposed elementary operators. Three case studies were conducted on industrial and commercial real-life systems to demonstrate the feasibility of using the proposed MBMT approach in detecting faults in SUC, and to analyze its characteristic features. Our experimental data suggest that test sets generated based on the mutated models created by insertion operators are more effective in revealing faults in SUC than those generated by omission operators. Worth noting is that test sets following the MBMT approach were able to detect faults in the systems that were tested by manufacturers and independent testing organizations before they were released.

Modeling, analysis and testing of safety issues: an event-based approach and case study

This paper proposes an event-based approach with an intuitive simple graphical representation of the system and its environment for designing, analysis and testing safety-critical systems. The events are user actions and system responses, and are ordered according to the threats posed by the resulting system states. This ordering is an integral aspect of the graphical representation, making it possible to directly identify the risks associated with each and every functionally desirable, and undesirable, event relative to one another. Tests that target safety requirements are devised by examining possible traces of these events, represented compactly by regular expressions, exhibiting particular risk patterns such as human error and system failures.

Model-Based Higher-Order Mutation Analysis

Mutation analysis is widely used as an implementation-oriented method for software testing and test adequacy assessment. It is based on creating different versions of the software by seeding faults into its source code and constructing test cases to reveal these changes. However, in case that source code of software is not available, mutation analysis is not applicable. In such cases, the approach introduced in this paper suggests the alternative use of a model of the software under test. The objectives of this approach are (i) introduction of a new technique for first-order and higher-order mutation analysis using two basic mutation operators on graph-based models, (ii) comparison of the fault detection ability of first-order and higher-order mutants, and (iii) validity assessment of the coupling effect.

Towards Scalable Robustness Testing

Several approaches have been developed to assess the robustness of a system. We propose a model-based approach to scalable testing the robustness of a software system using event sequence graphs (ESG) and decision tables (DT). Elementary modification operators are introduced to manipulate ESGs and DTs resulting in faulty models. Test cases generated from these faulty models are applied to the system under consideration to check its robustness. Thus, the approach enables the quantification of robustness with respect to a universe of erroneous inputs.

Communication Sequence Graphs for Mutation-Oriented Integration Testing

Integration testing (IT) plays an important role for validation of the communication between different software components to ensure their optimal cooperation. In this paper we (i) introduce communication sequence graphs (CSG) for integration testing representing the communication between software components on a meta-level, (ii) define coverage criteria based on these graphs, and (iii) give hints how to extend CSG notion by Boolean algebra in order to represent complex data structures of data exchanged between components to be integrated. A case study borrowed from a robot controlling system illustrates CSG as a new integration testing approach.

A Graph-Model-Based Testing Method Compared with the Classification Tree Method for Test Case Generation

In automotive industry, mechanic control units are more and more replaced by electronic devices that are often aggregated in electronic control units (ECU). Systematic testing is one of the preferred industrial validation methods to ensure functionality of those ECUs. It is imperative to reduce the costs and improve the effectiveness of testing by automating the testing process. This paper introduces a model-based testing method using event sequence graphs and compares this approach with the classification tree method which is popular in automotive industry. A case study applies both methods to the formal specification of an adaptive cruise control unit for generation and selection of test cases. To enable a meaningful comparison, test costs and number of faults revealed by both methods will be compared.

Structural Feature Extraction for GUI Test Enhancement

The challenging goal of designing interactive systems is to prevent faults caused by the control unit that could potentially lead to failures. When designing interactive systems, it is the behavior that is dominant in the process of modeling. Nevertheless, structural features in software application programming and their organization are seldom taken into account for test derivation. The consequence is a reduction in fault coverage with regard to find structural defects in such systems. This paper introduces an approach to reveal structural features of such systems that expose further test opportunities. The approach uses a graphical, event-based model of which structural features are extracted by simple transformations resulting into an increased fault coverage.

Statistical Evaluation of Test Sets Using Mutation Analysis

Evaluation of the ability of test sets for fault detection, and indirectly also evaluation of the quality of test techniques that generate those test sets, have become more of an issue in software testing. Based on mutation analysis, this paper evaluates and compares fault detection ability of test sets using statistical techniques. In this process also different mutant types (and indirectly different fault types) are considered. A case study, drawn from a large commercial web-based system, validates the approach and analyzes its characteristics.