Nigel Tracey

Automated program flaw finding using simulated annealing

One of the major costs in a software project is the construction of test-data. This paper outlines a generalised test-case data generation framework based on optimisation techniques. The framework can incorporate a number of testing criteria, for both functional and non-functional properties. Application of the optimisation framework to testing specification failures and exception conditions is illustrated. The results of a number of small case studies are presented and show the efficiency and effectiveness of this dynamic optimisation-base approach to generating test-data.

An automated framework for structural test-data generation

Structural testing criteria are mandated in many software development standards and guidelines. The process of generating test data to achieve 100% coverage of a given structural coverage metric is labour-intensive and expensive. This paper presents an approach to automate the generation of such test data. The test-data generation is based on the application of a dynamic optimisation-based search for the required test data. The same approach can be generalised to solve other test-data generation problems. Three such applications are discussed-boundary value analysis, assertion/run-time exception testing, and component re-use testing. A prototype tool-set has been developed to facilitate the automatic generation of test data for these structural testing problems. The results of preliminary experiments using this technique and the prototype tool-set are presented and show the efficiency and effectiveness of this approach.

A search-based automated test-data generation framework for safety-critical systems

This paper presents the results of a three year research program to develop an automated test-data generation framework to support the testing of safety-critical software systems. The generality of the framework comes from the exploitation of domain independent search techniques, allowing new test criteria to be addressed by constructing functions that quantify the suitability of test-data against the test-criteria. The paper presents four applications of the framework - specification falsification testing, structural testing, exception condition testing and worst-case execution time testing. The results of three industrial scale case-studies are also presented to show that the framework offers useful support in the development safety-critical software systems.

The second international workshop on automated program analysis, testing and verification

Program analysis, testing and verification are key techniques for building confidence in and increasing the quality of software systems. Such activities typically cost upwards of 50% of total development costs. Automation aims to allow both reduced costs and more thorough analysis, testing and verification and is vital to keep pace with increasing software complexity.

Engineering real-time behavior

This article presents a process that evaluates an application for real-time correctness throughout development and maintenance. It allows temporal correctness to be designed-in during development, rather than the more typical effort to test-in timing performance at the end of development. It avoids the costly problems that can arise when timing faults are found late in testing or, worse still, after deployment.

Automated analysis, verification and testing

In Software Testing, Verification and Reliability, NIGEL TRACEY, JOHN PENIX and WILLEM VISSER present a special issue for researchers and developers interested in automated verification, analysis and testing of software, exploring new collaborations and aiming to increase the level of co-operation between communities. Copyright

Testing and testing techniques for real-time embedded software systems

Real-time embedded systems and their associated software are becoming ever more intertwined with the running of our modern society. Many areas of endeavour including aviation, commerce, medical and office administration depend upon, and require the correct functioning of, software in order to perform their designated tasks. The failure of real-time embedded software can lead to undesirable consequences. In some cases, these consequences are safety related (threatening life or the environment), but in many cases the consequences are simply economic. Real-time embedded systems are increasingly finding their way into many mass produced products — automotive, mobile phones, consumer electronics, etc. For systems with millions of units in the field even very low software failure rates can result in massive economic consequences. Product recalls and software updates in the field are extremely expensive. For these systems software quality is of vital importance. Given this increasing reliance on software, it is vital to obtain confidence that it will perform its intended function correctly. Testing is one of the most important techniques used in industry to assess a software product and reduce the risk of failure. Testing is the exercising of the software in a controlled and monitored environment. While there are other methods available to gain confidence in software (for example static analysis, formal proofs or correctness and formal refinement) these are not as widely used as testing. Even where such techniques are used they still do not obviate the need for high quality software testing. Only dynamic testing can be used to gain confidence in the correct functioning of the software in its intended environment, testing the object code generated by the compiler and the execution of the object code by the hardware platform. Real-time embedded software presents special problems when testing. The real-time nature of the systems requires that both the temporal and functional properties of the system be tested. The embedded nature of the systems makes controllability and observability more difficult. Environmental interactions and multitasking environments increase the problems of non-determinism and reduce the repeatability of testing. This special issue presents three papers that address different aspects of real-time embedded software testing. The first, by Rob Hierons, presents an approach to generate tests directly from communicating finite state machine specifications to check implementations of those state machines. The second paper by Gautam Singh and Scott Gobrogge describes an approach to develop, prioritise and ultimately reuse test-plans for embedded software. The final paper by Henrik Thane and Hans Hansson presents a deterministic approach to testing multitasking real-time systems and analysing all possible run-time execution interleavings of the executing tasks. We would like to conclude by thanking all of the authors who submitted material for this special issue and also the reviewers for their valuable input.

The first international workshop on automated program analysis, testing and verification

This paper reports on the First International Workshop on Automated Program Analysis, Testing and Verification (WAPATV) held in Limerick on the 4th-5th June 2000, as part of the International Conference on Software Engineering 2000. We begin by presenting an overview of the workshop aims and then focus on the workshops technical program.