Mutlu Beyazit

Testing is an Event-Centric Activity

Recent advances in techniques for testing graphical user interfaces (GUIs) enabled to develop workflow models and successfully employ them to generate large numbers of test cases by defining new test adequacy criteria and optimizing test suites for increasing the test efficiency. The key to the success of these event-focused techniques, especially event flow graphs and event sequence graphs, is that they primarily focus on the input space, and model the workflow in simple terms. If necessary, they can also be augmented to model more complex systems and processes to adapt to the needs of test engineers. We now posit that we can extend these techniques to also domains other than GUIs to create a general event-driven paradigm for testing.

A Formal Framework for Mutation Testing

Model-based approaches, especially based on directed graphs (DG), are becoming popular for mutation testing as they enable definition of simple, nevertheless powerful, mutation operators and effective coverage criteria. However, these models easily become intractable if the system under consideration is too complex or large. Moreover, existing DG-based algorithms for test generation and optimization are rare and rather in an initial stage. Finally, DG models fail to represent languages beyond type-3 (regular). This paper proposes a grammar-based mutation testing framework, together with effective mutation operators, coverage concepts and algorithms for test sequence generation. The objective is to establish a formal framework for model-based mutation testing which enables complementary or alternative use of regular grammars, depending on the preferences of the test engineer. A case study validates the approach and analyzes its characteristic issues.

Event-Based GUI Testing and Reliability Assessment Techniques -- An Experimental Insight and Preliminary Results

It is widely accepted that graphical user interfaces (GUIs) highly affect -- positive or negative -- the quality and reliability of human-machine systems. In spite of this fact, quantitative assessment of the reliability of GUIs is a relatively young research field. Existing software reliability assessment techniques attempt to statistically describe the software testing process and to determine and thus predict the reliability of the system under consideration (SUC). These techniques model the reliability of the SUC based on particular assumptions and preconditions on probability distribution of cumulative number of failures, failure data observed, and form of the failure intensity function, etc. We expect that the methods used for modeling a GUI and related frameworks used for testing it also affect the factors mentioned above, especially failure data to be observed and prerequisites to be met. Thus, the quality of the reliability assessment process, and ultimately also the reliability of the GUI, depends on the methods used for modeling and testing the SUC. This paper attempts to gain some experimental insight into this problem. GUI testing frameworks based on event sequence graphs and event flow graphs were chosen as examples. A case study drawn from a large commercial web-based system is used to carry out the experiments and discuss the results.

Mutation Testing of "Go-Back" Functions Based on Pushdown Automata

A go-back (GB) function for canceling recent user or system operations and going back to and resuming of previous state(s) is very often used regardless of the application domain. Therefore, faulty handling of them can cause severe damages in those applications. This paper proposes a mutation-based approach to testing GB functions modeled by pushdown automata. Novel mutation operators, recent coverage criteria, and a new algorithm for test case generation are introduced. A case study validates the approach and discusses its characteristics.

Event-Oriented, Model-Based GUI Testing and Reliability Assessment—Approach and Case Study

Abstract It is widely accepted that graphical user interfaces (GUIs) highly affect—positive or negative—the quality and reliability of human-machine systems. However, quantitative assessment of the reliability of GUIs is a relatively young research field. Based on probability theory and statistics, the existing software reliability models describe the behavior of software failures and attempt to predict the reliability of the system under consideration (SUC). They operate on particular assumptions about the probability distribution of the cumulative number of failures, the observed failure data, and the form of the failure intensity function, etc. Bad news is that there is no single, universal model that can be used in all cases. To select an appropriate model, or a set of models, the failure data should be considered and analyzed carefully. We expect that the methods used for modeling and testing a GUI also affect its reliability. Consequently, the quality of the reliability assessment process, and, ultimately, the reliability of the GUI depend on the approaches used for modeling and testing the SUC. This chapter critically reviews existing software reliability models and attempts to gain experimental insight into this problem. Two different event-based GUI testing frameworks are chosen as examples. A case study drawn from a large commercial web-based system is used to carry out the experiments; results are analyzed and discussed to come to recommendations for practical usage of the approach.

Exploiting Model Morphology for Event-Based Testing

Model-based testing employs models for testing. Model-based mutation testing (MBMT) additionally involves fault models, called mutants, by applying mutation operators to the original model. A problem encountered with MBMT is the elimination of equivalent mutants and multiple mutants modeling the same faults. Another problem is the need to compare a mutant to the original model for test generation. This paper proposes an event-based approach to MBMT that is not fixed on single events and a single model but rather operates on sequences of events of length k ≥ 1 and invokes a sequence of models that are derived from the original one by varying its morphology based on k. The approach employs formal grammars, related mutation operators, and algorithms to generate test cases, enabling the following: (1) the exclusion of equivalent mutants and multiple mutants; (2) the generation of a test case in linear time to kill a selected mutant without comparing it to the original model; (3) the analysis of morphologically different models enabling the systematic generation of mutants, thereby extending the set of fault models studied in related literature. Three case studies validate the approach and analyze its characteristics in comparison to random testing and another MBMT approach.

Mutation of Directed Graphs -- Corresponding Regular Expressions and Complexity of Their Generation

Directed graphs (DG), interpreted as state transition diag rams, are traditionally used to represent finite-state automata (FSA). In the context of formal langua ges, both FSA and regular expressions (RE) are equivalent in that they accept and generate, respectively, type-3 (regular) languages. Based on our previous work, this paper analyzes effects of graph manipulations on corresponding RE. In this present, starting stage we assume that the DG under consideration contains no cycles. Graph manipulation is performed by deleting or inserting of nodes or arcs. Combined and/or multiple application of these basic operators enable a great variety of transformations of DG (and corresponding RE) that can be seen as mutants of the original DG (and corresponding RE). DG are popular for modeling complex systems; however they easily become intractable if the system under consideration is complex and/or large. In such situations, we propose to switch to corresponding RE in order to benefit from their compact format for modeling and algebra ic operations for analysis. The results of the study are of great potential interest to mutation test ing.

Using regular grammars for event-based testing

Model-based testing involves formal models for test generation. This paper suggests regular grammars for event-based modeling. This model, represented in BNF, will then be systematically modified by well-defined mutation operators in order to generate fault models, called mutants. Specific algorithms apply to both the model of the system under consideration and the mutants to generate test cases. While existing methods focus on single events the approach introduced in this paper suggests considering event sequences of length k≥1, that is, k-sequences. The approach also enables to cope with a tough problem encountered in mutation-oriented testing: the elimination of mutants that are equivalent to the original model, and mutants that model the same faults multiple times. These mutants lead to unproductive test suites that cause wasting of resources. The approach proposed devises strategies to exclude the mentioned mutants in that they will not be generated at all.

Event-Based Mutation Testing vs. State-Based Mutation Testing - An Experimental Comparison

Model-based testing (MBT) focuses on relevant, mostly user-centric features of the system under consideration (SUC) and enables test case generation without requiring source code. Depending on these features and the preferences of the tester, modeling can be event-based or state-based. This paper compares both techniques using mutation testing, which is originally code-based, but has recently been extended to enable also MBT. For the comparison, the paper introduces frameworks that are composed of a set of models, a set of mutation operators, a set of coverage criteria, and a set of test generation algorithms. The introduced concepts and notions are demonstrated over a case study based on a large web-based commercial portal. Analysis of the experimental data yields results on the discussed frameworks reviewing benefits and drawbacks of event-based and state-based testing.

Fault domain-based testing in imperfect situations: a heuristic approach and case studies

Model-based testing (MBT) involves creating an abstraction, called a model, to represent the system and automatically deriving test cases from this model. MBT can be performed using various approaches that generally employ certain assumptions or requirements affecting the test performance in practice. Here, we consider the harmonized state identifiers (HSI) method, which is based on finite state machine (FSM) models and generates test sets that cover all faults in a given domain under certain conditions. We are interested in the application of the HSI method in practical scenarios where some conditions do not hold or are not straightforward to satisfy. Thus, we propose a heuristic extension to the HSI method, called heuristic HSI (HHSI), to consider imperfect situations as they often occur in practice. To analyze the characteristics of HHSI, we empirically compare it to random testing and coverage-based testing using non-trivial case studies. The experiments include model-based mutation analyses over several FSM models.