Juan José Domínguez-Jiménez

Evolutionary mutation testing

Abstract Context Mutation testing is a testing technique that has been applied successfully to several programming languages. However, it is often regarded as computationally expensive, so several refinements have been proposed to reduce its cost. Moreover, WS-BPEL compositions are being widely adopted by developers, but present new challenges for testing, since they can take much longer to run than traditional programs of the same size. Therefore, it is interesting to reduce the number of mutants required. Objective We present Evolutionary Mutation Testing (EMT), a novel mutant reduction technique for finding mutants that help derive new test cases that improve the quality of the initial test suite. It uses evolutionary algorithms to reduce the number of mutants that are generated and executed with respect to the exhaustive execution of all possible mutants, keeping as many difficult to kill and potentially equivalent mutants (strong mutants) as possible in the reduced set. Method To evaluate EMT we have developed GAmera, a mutation testing system powered by a co-evolutive genetic algorithm. We have applied this system to three WS-BPEL compositions to estimate its effectiveness, comparing it with random selection. Results The results obtained experimentally show that EMT can select all strong mutants generating 15% less mutants than random selection in over 20% less time for complex compositions. When generating a percentage of all mutants, EMT finds on average more strong mutants than random selection. This has been confirmed to be statistically significant within a 99.9% confidence interval. Conclusions EMT has reduced for the three tested compositions the number of mutants required to select those which are useful to derive new test cases that improve the quality of the test suite. The directed search performed by EMT makes it more effective than random selection, especially as compositions become more complex and the search space widens.

GAmera: An Automatic Mutant Generation System for WS-BPEL Compositions

Business processes based in WS-BPEL compositions are rapidly becoming commonplace in recent years, so it is important to pay special attention to testing in this context. Mutation testing is a white-box testing technique that has been applied successfully to programs written in several languages. In order to apply it we need a system for mutant generation. This paper presents the first results of applying a new system, GAmera, for automatic mutant generation for WS-BPEL compositions. It is composed by three different elements: an analyzer, a mutant generator and a system that executes and evaluates the mutants. GAmera is based in genetic algorithms and attempts to minimize the number of generated mutants, independently of the number and type of mutation operators, without losing relevant information. It can also detect potentially equivalent mutants allowing to improve the quality of the test suite.

A Framework for Mutant Genetic Generation for WS-BPEL

The rise of Web Services and their WS-BPEL compositions in recent years makes necessary to pay special attention to testing in this context. Mutation testing is a white box testing technique that has been applied successfully to programs written in different languages. In order to apply it we need a set of mutation operators and a system for mutant generation. This paper introduces a set of mutation operators for the WS-BPEL 2.0 language and a framework, based in genetic algorithms, for automatic mutant generation without rendering all possible mutants. This framework can also detect potentially equivalent mutants.

Quality metrics for mutation testing with applications to WS-BPEL compositions

Mutation testing is a successful testing technique based on fault injection. However, it can be very costly, and several cost‐reduction techniques for reducing the number of mutants have been proposed in the literature. Cost reduction can be aided by an analysis of mutation operators, but this requires the definition of specialized metrics. Several metrics have been proposed before, although their effectiveness and relative merits are not easy to assess. A step ahead in the evaluation of mutation‐reduction techniques would be a better metric to determine objectively the quality of a set of mutants with respect to a given test suite. This work introduces such a metric, which is naturally extended to mutation operators and may be used to reduce the number of mutants, particularly of equivalent mutants. Finally, a firm mutation analysis tool for WS‐BPEL service compositions is presented, and experimental results obtained by comparing different metrics on several compositions are presented. Copyright

Assessment of class mutation operators for C + + with the MuCPP mutation system

Context: Mutation testing has been mainly analyzed regarding traditional mutation operators involving structured programming constructs common in mainstream languages, but mutations at the class level have not been assessed to the same extent. This fact is noteworthy in the case of C + + , despite being one of the most relevant languages including object-oriented features. Objective: This paper provides a complete evaluation of class operators for the C + + programming language. MuCPP, a new system devoted to the application of mutation testing to this language, was developed to this end. This mutation system implements class mutation operators in a robust way, dealing with the inherent complexity of the language. Method:MuCPP generates the mutants by traversing the abstract syntax tree of each translation unit with the Clang API, and stores mutants as branches in the Git version control system. The tool is able to detect duplicate mutants, avoid system headers, and drive the compilation process. Then, MuCPP is used to conduct experiments with several open-source C + + programs. Results: The improvement rules listed in this paper to reduce unproductive class mutants have a significant impact in the computational cost of the technique. We also calculate the quantity and distribution of mutants generated with class operators, which generate far fewer mutants than their traditional counterparts. Conclusions: We show that the tests accompanying these programs cannot detect faults related to particular object-oriented features of C + + . In order to increase the mutation score, we create new test scenarios to kill the surviving class mutants for all the applications. The results confirm that, while traditional mutation operators are still needed, class operators can complement them and help testers further improve the test suite.

Mutation Testing of Event Processing Queries

Event processing queries are intended to process continuous event streams. These queries are partially similar to traditional SQL queries, but provide the facilities to express rich features (e.g., pattern expression, sliding window of length and time). An error while implementing a query may result in abnormal program behaviors and lost business opportunities. Moreover, queries can be generated with unsanitized inputs and the structure of intended queries might be altered. Thus, a tester needs to test the behavior of queries in presence of malicious inputs. Mutation testing has been found to be effective to assess test suites quality and generating new test cases. Unfortunately, there is no effort to perform mutation testing of event processing queries. In this work, we propose mutation-based testing of event processing queries. We choose Event Processing Language (EPL) as our case study and develop necessary mutation operators and killing criteria to generate high quality event streams and malicious inputs. Our proposed operators modify different features of EPL queries (pattern expression, windows of length and time, batch processing of events). We develop an architecture to generate mutants for EPL and perform mutation analysis. We evaluate our proposed EPL mutation testing approach with a set of developed benchmark containing diverse types EPL queries. The evaluation results indicate that the proposed operators and mutant killing criteria are effective to generate test cases capable of revealing anomalous program behaviors (e.g., event notification failure, delay of event reporting, unexpected event), and SQL injection attacks. Moreover, the approach incurs less manual effort and can complement other testing approach such as random testing.

GAmera: a Tool for WS-BPEL composition testing using mutation analysis

This paper shows a novel tool, GAmera, the first mutant generation tool for testing Web Service compositions written in the WS-BPEL language. After several improvements and the development of a graphical interface, we consider GAmera to be a mature tool that implements an optimization technique to reduce the number of generated mutants without significant loss of testing effectiveness. A genetic algorithm is used for generating and selecting a subset of high-quality mutants. This selection reduces the computational cost of mutation testing. The subset of mutants generated with this tool allows the user to improve the quality of the initial test suite.

A Framework for Genetic Test-Case Generation for WS-BPEL Compositions

Search-based testing generates test cases by encoding an adequacy criterion as the fitness function that drives a search-based optimization algorithm. Genetic algorithms have been successfully applied in search-based testing: while most of them use adequacy criteria based on the structure of the program, some try to maximize the mutation score of the test suite. This work presents a genetic algorithm for generating a test suite for mutation testing. The algorithm adopts several features from existing bacteriological algorithms, using single test cases as individuals and keeping generated individuals in a memory. The algorithm can optionally use automated seeding when producing the first population, by taking into account interesting constants in the source code. We have implemented this algorithm in a framework and we have applied it to a WS-BPEL composition, measuring to which extent the genetic algorithm improves the initial random test suite. We compare our genetic algorithm, with and without automated seeding, to random testing.

IoT–TEG: Test event generator system

Abstract Internet of Things (IoT) has been paid increasingly attention by the government, academe and industry all over the world. One of the main drawbacks of the IoT systems is the amount of information they have to handle. This information arrives as events that need to be processed in real time in order to make correct decisions. Given that processing the data is crucial, testing the IoT systems that will manage that information is required. In order to test IoT systems, it is necessary to generate a huge number of events with specific structures and values to test the functionalities required by these systems. As this task is very hard and very prone to error if done by hand, this paper addresses the automated generation of appropriate events for testing. For this purpose, a general specification to define event types and its representation are proposed and an event generator is developed based on this definition. Thanks to the adaptability of the proposed specification, the event generator can generate events of an event type, or events which combine the relevant attributes of several event types. Results from experiments and real-world tests show that the developed system meets the demanded requirements.

Evaluation of EPL mutation operators with the MuEPL mutation system

Abstract The expert systems (ESs) have been developed to facilitate the users their tasks, to enhance the productivity and reduce losses. In order to replicate the behaviour of a human expert, they generate output using their stored knowledge base. In a ES, the accumulation of knowledge from different sources is a very important factor. Nowadays, we are living in a world where two crucial processes need to be perform quickly: decision-making and problem solving. The complexity of the decisions and problems lays on the different factors, situations and data that are involved. The Internet of Things (IoT) has been created to address these situations and helps the users to make correct decisions in real time according to the received data. In the concept of IoT, daily life objects are connected to each other so they can transfer data over the internet without a human to human interaction. The combination of IoT and ESs is a step further for the decision making problem, the received data from the IoT system will be sent to the ES, then the ES will process the information and send the results or decisions to the user. Given that correct decisions-making and problem solving are critical processed, these complex systems need to be tested. Mutation testing, which is a technique used in fault testing, has been examined in a range of studies where different programming languages have been used as well as in IoT expert systems evaluations. However, this technique has not been applied to an IoT programming language, which is noteworthy in the case of event processing languages (EPLs), that have been designed to address the main problems of IoT systems. Among the existing EPLs, the EPL of EsperTech is used the most often. In this paper, we apply mutation testing using MuEPL to EPL of EsperTech programs in order to simulate the common errors of the developers and to avoid the wrong decisions before moving out to ES in the IoT network.