Andreas Windisch

Applying particle swarm optimization to software testing

Evolutionary structural testing is an approach to automatically generating test cases that achieve high structural code coverage. It typically uses genetic algorithms (GAs) to search for relevant test cases. In recent investigations particle swarm optimization (PSO), an alternative search technique, often outperformed GAs when applied to various problems. This raises the question of how PSO competes with GAs in the context of evolutionary structural testing.In order to contribute to an answer to this question, we performed experiments with 25 small artificial test objects and 13 more complex industrial test objects taken from various development projects. The results show that PSO outperforms GAs for most code elements to be covered in terms of effectiveness and efficiency.

Evolutionary functional black-box testing in an industrial setting

During the past years, evolutionary testing research has reported encouraging results for automated functional (i.e. black-box) testing. However, despite promising results, these techniques have hardly been applied to complex, real-world systems and as such, little is known about their scalability, applicability, and acceptability in industry. In this paper, we describe the empirical setup used to study the use of evolutionary functional testing in industry through two case studies, drawn from serial production development environments at Daimler and Berner & Mattner Systemtechnik, respectively. Results of the case studies are presented, and research questions are assessed based on them. In summary, the results indicate that evolutionary functional testing in an industrial setting is both scalable and applicable. However, the creation of fitness functions is time-consuming. Although in some cases, this is compensated by the results, it is still a significant factor preventing functional evolutionary testing from more widespread use in industry.

Industrial Scaled Automated Structural Testing with the Evolutionary Testing Tool

Evolutionary testing has been researched and promising results have been presented. However, evolutionary testing has remained predominately a research-based activity not practiced within industry. Although attempts have been made, such as Daimlers Evolutionary Structural Test (EST) prototype, until now, no such tool has been suitable for industrial adoption. The European project EvoTest (IST-33472) team has been working from 2006 till2009 to improve this situation. This paper describes the final version of the Evolutionary Testing Framework (ETF) resulting from the EvoTest project. In specific we will present the EvoTest Structural Testing tool for fully automatic structural testing that has been demonstrated to be suitable within an industrial setting. The paper concentrates on how to use it and interpret the results. The paper starts with introducing the concepts of Evolutionary Testing in general and Structural Testing in specific. Subsequently, the ETF and the EvoTest Structural Testing tool built on-top of it will be described. We will concentrate on the usage, the architecture, and remaining limitations of the tool. The paper concludes describing the results of using the EvoTest Structural Testing tool in practice on real-world systems in an industrial setting.

Integrating Model-Based Testing with Evolutionary Functional Testing

Evolutionary Functional Testing (EFT) is a relatively recent approach to automating the testing process. The research presented in this paper aims at increasing the acceptability of EFT in industrial settings. An approach suitable for efficiently and effectively testing complex continuous control systems is introduced. The main focus is on generating realistic test stimuli, enabling interactivity between test driver and test object, and facilitating the process of designing a suitable fitness function. This is accomplished by integrating EFT with model-based testing methodologies resulting in an intuitive testing approach that enables even testers not familiar with search based testing to achieve good results with limited effort. A test environment optimized for deployment in the industrial domain is introduced. Features of the test environment include the capability of automatically generating realistic continuous test data sets, interacting with the system under test during test execution, and automatically executing and evaluating large numbers of tests. A thorough case study using an adaptive cruise control system from the automotive domain is performed to assess the approach. Results of this work indicate high usability, efficiency, and effectiveness of the proposed method for testing complex embedded systems.

Search-based test data generation from stateflow statecharts

This paper proposes an effective method for automating the test data generation process aiming at structurally covering Stateflow statecharts, while assuring the generation of suitable and - most notably - realistic and meaningful system inputs. For this purpose the principles of evolutionary structural testing have been adapted both for the application to statecharts and for the consideration of continuous signals. The approach is evaluated using a complex industrial case study in comparison to random testing. The results demonstrate the value of this approach in industrial settings due to both its search effectiveness and its high degree of automation, potentially contributing to an improvement in quality assurance of embedded software systems.

Signal Generation for Search-Based Testing of Continuous Systems

Test case generation constitutes a critical activity in software testing that is cost-intensive, time-consuming and error-prone when done manually. Hence, an automation of this process is required. One automation approach is search-based testing for which the task of generating test data is transformed into an optimization problem which is solved using metaheuristic search techniques. However, only little work has been done so far applying search-based testing techniques to systems that depend on continuous input signals.This paper proposes two novel approaches to generating input signals from within search-based testing techniques for continuous systems. These approaches are then shown to be very effective when experimentally applied to the problem of approximating a set of realistic signals.

Search-based testing of complex simulink models containing stateflow diagrams

Model-based software design is constantly becoming more important and thus requiring systematic model testing. Test case generation constitutes a critical activity that is cost-intensive, time-consuming and error-prone when done manually. Hence, an automation of this process is required. One automation approach is search-based testing for which the task of generating test data is transformed into an optimization problem which is solved using metaheuristic search techniques. However, only little work has been done so far applying search-based testing techniques to continuous functional models, such as SIMULINK STATEFLOW models. This paper presents the current state of my thesis developing a new approach for automatically generating continuous test data sets achieving high structural model coverage for SIMULINK models containing STATEFLOW diagrams using search-based testing. The expected contribution of this work is to demonstrate how search-based testing techniques can be applied successfully to continuous functional models and how to cope with the arising problems such as generating and optimizing continuous signals, covering structural model elements and dealing with the complexity of the models.

A Search-Based Approach to Functional Hardware-in-the-Loop Testing

The potential of applying search-based testing principles to functional testing has been demonstrated in various cases. The focus was mainly on simulating the system under test using a model or compiled source code in order to evaluate test cases. However, in many cases only the final hardware unit is available for testing. This research presents an approach in which evolutionary functional testing is performed using an actual electronic control unit for test case evaluation. A test environment designed to be used for large-scale industrial systems is introduced. An extensive case study has been carried out to assess its capabilities. Results indicate that the approach proposed in this work is suitable for automated functional testing of embedded control systems within a Hardware-in the-Loop test environment.

Considering Signal Constraints in Search-Based Testing of Continuous Systems

This paper presents a technique to consider constraints on input signals of continuous systems when applying search-based testing. The signal constraints are described using a logic based on Signal Temporal Logic. We developed a distance-oriented evaluation technique for these constraints that provides an exact rating of the amount of constraint violation, thus allowing a ranking of the generated solutions in terms of constraint violation. An adaptive penalty function is then used to incorporate the evaluation results into the optimization. Finally, the overall method is shown to be capable of considering signal constraints appropriately when experimentally applied to search-based black-box testing of a MATLAB SIMULINK model of an automatic transmission controller.

Temporal White-Box Testing Using Evolutionary Algorithms

Embedded computer systems should fulfill real-time requirements which need to be checked in order to assure system quality. This paper stands to propose some ideas for testing the temporal behavior of real-time systems. The goal is to achieve white-box temporal testing using evolutionary techniques to detect system failures in reasonable time and little effort.