Harmen Sthamer

Evolutionary test environment for automatic structural testing

Abstract Testing is the most significant analytic quality assurance measure for software. Systematic design of test cases is crucial for the test quality. Structure-oriented test methods, which define test cases on the basis of the internal program structures, are widely used. A promising approach for the automation of structural test case design is evolutionary testing. Evolutionary testing searches test data that fulfil a given structural test criteria by means of evolutionary computation. In this work, an evolutionary test environment has been developed that performs fully automatic test data generation for most structural test methods. The introduction of an approximation level for fitness evaluation of generated test data and the definition of an efficient test strategy for processing test goals, increases the performance of evolutionary testing considerably.

Testability transformation

A testability transformation is a source-to-source transformation that aims to improve the ability of a given test generation method to generate test data for the original program. We introduce testability transformation, demonstrating that it differs from traditional transformation, both theoretically and practically, while still allowing many traditional transformation rules to be applied. We illustrate the theory of testability transformation with an example application to evolutionary testing. An algorithm for flag removal is defined and results are presented from an empirical study which show how the algorithm improves both the performance of evolutionary test data generation and the adequacy level of the test data so-generated.

Testing real-time systems using genetic algorithms

The development of real-time systems is an essential industrial activity whose importance is increasing. The most important analytical method to assure the quality of real-time systems is dynamic testing. Testing is the only method which examines the actual run-time behaviour of real-time software, based on an execution in the real application environment. Dynamic aspects like the duration of computations, the memory actually needed, or the synchronization of parallel processes are of major importance for the correct function of real-time systems and have to be tested. A comprehensive investigation of existing software test methods shows that they mostly concentrate on testing for functional correctness. They are not suited for an examination of temporal correctness which is essential to real-time systems. Very small systems show a wide range of different execution times. Therefore, existing test procedures must be supplemented by new methods, which concentrate on determining whether the system violates its specified timing constraints. In general, this means that outputs are produced too early or their computation takes too long. The task of the tester is to find the inputs with the longest or shortest execution times to check whether they produce a temporal error. If the search for such inputs is interpreted as a problem of optimization, genetic algorithms can be used to find the inputs with the longest or shortest execution times automatically. The fitness function is the execution time measured in processor cycles. Experiments using genetic algorithms on a number of programs with up to 1511 LOC and 843 integer input parameters have successfully identified new longer and shorter paths than had been found using random testing or systematic testing. Genetic algorithms are able therefore to check large programs and they show considerable promise in establishing the validity of the temporal behaviour of real-time software.

Evolutionary testing of flag conditions

Evolutionary Testing (ET) has been shown to be very successful in testing real world applications [16]. However, it has been pointed out [11], that further research is necessary if flag variables appear in program expressions. The problems increase when ET is used to test state-based applications where the encoding of states hinders successful evolutionary tests. This is because the ET performance is reduced to a random test in case of the use of flag variables or variables that encode an enumeration type. The authors have developed an ET System to provide easy access to automatic testing. An extensive set of programs has been tested using this system [4], [16]. This system is extended for new areas of software testing and research has been carried out to improve its performance. This paper introduces a new approach for solving ET problems with flag conditions. The problematic constructs are explained with the help of code examples originally found in large real world applications.

Improving evolutionary real-time testing

Embedded systems are often used in a safety-critical context, e.g. in airborne or vehicle systems. Typically, timing constraints must be satisfied so that real-time embedded systems work properly and safely. Execution time testing involves finding the best and worst case execution times to determine if timing constraints are respected. Evolutionary real-time testing (ERTT) is used to dynamically search for the extreme execution times. It can be shown that ERTT outperforms the traditional methods based on static analysis. However, during the evolutionary search, some parts of the source code are never accessed. Moreover, it turns out that ERTT delivers different extreme execution times in a high number of generations for the same test object, the results are neither reliable nor efficient. We propose a new approach to ERTT which makes use of seeding the evolutionary algorithm with test data achieving a high structural coverage. Using such test data ensures a comprehensive exploration of the search space and leads to rise the confidence in the results. We present also another improvement method based on restricting the range of the input variables in the initial population in order to reduce the search space. Experiments with these approaches demonstrate an increase of reliability in terms of constant extreme execution times and a gain in efficiency in terms of number of generations needed.

Testing the temporal behavior of real-time tasks using extended evolutionary algorithms

For real-time systems, correct system functionality depends on logical as well as on temporal correctness. Static analysis alone is not sufficient to verify the temporal behavior of real-time systems. Since existing test methods are not specialized for the verification of temporal correctness, we have developed a new testing method, namely evolutionary testing. This paper illustrates results of the first industrial application of the evolutionary test.

Evolutionary testing supported by slicing and transformation

Evolutionary testing is a search based approach to the automated generation of systematic test data, in which the search is guided by the test data adequacy criterion. Two problems for evolutionary testing are the large size of the search space and structural impediments in the implementation of the program which inhibit the formulation of a suitable fitness function to guide the search. In this paper we claim that slicing can be used to narrow the search space and transformation can be applied to the problem of structural impediments. The paper presents examples of how these two techniques have been successfully employed to make evolutionary testing both more efficient and more effective.

Improving Evolutionary Real-Time Testing by Seeding Structural Test Data

Timing constraints in embedded systems must be satisfied so that real-time embedded systems work properly and safely. Execution time testing involves finding the best and worst case execution times. Evolutionary testing is used to dynamically search for the extreme execution times. During the evolutionary search, some parts of the source code are never accessed. Moreover, it turns out that the search delivers different extreme execution times in a high number of generations. We propose a new approach which makes use of seeding the evolutionary algorithm with test data achieved a high structural coverage. This new method leads to raise the confidence in the results and to gain in efficiency in terms of number of generations needed.

Applying Evolutionary Testing to Search for Critical Defects

Software systems are used regularly in safety-relevant applications. Therefore, the occurrence of critical defects may not only cause costly recalls but may also endanger human lives. Accordingly, the development of software systems in industrial practice must comply with the highest quality requirements and standards. In practice, the most important analytical quality assurance method is dynamic testing and the most important activity to ensure this quality is test case determination. The effectiveness and efficiency of the test process can be clearly improved by Evolutionary Testing. Evolutionary Testing is a metaheuristic search technique for the generation of test cases.