Martin R. Woodward

A Measure of Control Flow Complexity in Program Text

This paper discusses the need for measures of complexity and unstructuredness of programs. A simple language independent concept is put forward as a measure of control flow complexity in program text and is then developed for use as a measure of unstructuredness. The proposed metric is compared with other metrics, the most notable of which is the cyclomatic complexity measure. Some experience with automatic tools for obtaining these metrics is reported.

Mutation testing—its origin and evolution

Abstract The aim of the paper is to provide a brief review of the program testing technique known as ‘mutation testing’ and outline current research directions in this area. Mutation testing is an example of what is sometimes called an error-based testing technique. In other words, it involves the construction of test data designed to uncover specific errors or classes of errors. A large number of simple changes (mutations) are made to a program, one at a time. Test data then has to be found which distinguishes the mutated versions from the original version. Although the idea was proposed more than a decade ago, it is in some ways still a ‘new’ technique. Originally it was seen by many as costly and somewhat bizarre. However, several variants of the basic method have evolved and these, possibly in conjunction with more efficient techniques for applying the method, can help reduce the cost. Also, by guaranteeing the absence of particular errors, it may be one way to achieve the high reliability necessary in critical software. A further advantage of mutation testing is its universal applicability to all programming languages.

On program analysis

Interpretation and Static Analysis David Schmidt Kansas State University Four parts:

On the relationship between two control-flow coverage criteria: all JJ-paths and MCDC

Abstract Coverage criteria may be used to assess the adequacy of software test data. Improved test data, that takes account of any inadequacies identified by lack of coverage, may then be developed. It is natural to seek ways of comparing different criteria and the ‘subsumes’ relationship is one such way: one criterion subsumes another, if satisfying the first always implies satisfaction of the second. This paper considers two criteria: ‘all jump-to-jump paths’ (all JJ-paths) and ‘modified condition/decision coverage’ (MCDC). It might be anticipated that there would be a relationship between these criteria since both are based on advanced control-flow concepts. MCDC has particular importance since it is involved in the DO-178B standard for avionics software. However, it is shown that ‘all JJ-paths’ and MCDC are, in general, incomparable, but for programs written under certain specific constraints ‘all JJ-paths’ subsumes MCDC.

Difficulties using cohesion and coupling as quality indicators

The benefits of decomposing a software system at the design stage into a collection of modules have long been recognized, especially if the activities within each module contribute to performing one single goal, and also if the modules have, as far as possible, minimal interaction with each other. In the so-called structured design approach, qualitative scales, or spectra, for cohesion and coupling capture these notions a little more rigorously and provide a means of assessing the quality of a design.In practice the ideas of cohesion and coupling are difficult to apply. A case study with a medium-sized Fortran program is described, in which 163 students were asked to categorize the cohesion and coupling of its modules. The subjective nature of the cohesion and coupling scales is exemplified by the diversity of opinions that resulted. The evidence of other empirical studies also illustrates the problem, since researchers have attempted to assess cohesion and coupling in a variety of different ways, often with quite primitive levels of discrimination. The fundamental difficulty is that there are no agreed objective measures for the quantities, nor the means to obtain any measures. Some proposals are reviewed, as well as recent developments of the ideas to cope with abstract data types.

An Experiment to Compare the Comprehensibility of Textual and Visual Forms of Algebraic Specifications

A vital step in the creation of a visual language is adequate empirical evidence to support any claimed advantages. To this end, the effects of visualisation when applied to the domain of algebraic specifications are investigated through an experiment designed to compare the comprehensibility of one textual and two visual approaches. A statistical analysis of the results is presented and conclusions are drawn regarding the relative worth of the differing approaches.

Testability, fault size and the domain-to-range ratio: An eternal triangle

A number of different concepts have been proposed that, loosely speaking, revolve around the notion of software testability. Indeed, the concept of testability itself has been interpreted in a variety of ways by the software community. One interpretation is concerned with the extent of the modifications a program component requires, in terms of its input and output variables, so that the entire behaviour of the component is observable and controllable. Another interpretation is the ease with which faults, if present in a program, can be revealed by the testing process and the propagation, infection and execution (PIE) model has been proposed as a method of estimating this. It has been suggested that this particular interpretation of testability might be linked with the metric domain-to-range ratio (DRR), i.e. the ratio of the cardinality of the set of all inputs (the domain) to the cardinality of the set of all outputs (the range). This paper reports work in progress exploring some of the connections between the concepts mentioned. In particular, a simple mathematical link is established between domain-to-range ratio and the observability and controllability aspects of testability. In addition, the PIE model is re-considered and a relationship with fault size is observed. This leads to the suggestion that it might be more straightforward to estimate PIE testability by an adaptation of traditional mutation analysis. The latter suggestion exemplifies the main goals of the work described here, namely to seek greater understanding of testability in general and, ultimately, to find easier ways of determining it.

Insights into software testing

MARTIN WOODWARD aims to provide a brief insight into some of the many aspects of software testing. The various stages required when testing large software systems are outlined and then a selection of possible testing techniques are described and illustrated. Given the increased use of formal specifications in software development, one could exploit such a specification during testing and some of the ways this might be done are indicated. Similarly, the nature of the design approach used in the development of the software or even the nature of the particular application being developed can lead to particular test strategies. Finally, testing is not purely a technical matter; people issues and management of the process have a significant impact on the success or otherwise of this activity. A brief discussion of these topics rounds off this review. Copyright

Critical Analysis of the PIE Testability Technique

The work of Voas and colleagues has introduced, refined and applied the propagation, infection and execution (PIE) analysis technique for measuring testability of programs. The purpose of this paper is twofold: (1) to summarize and review the work done by Voas and others, and (2) to lay down a framework for automating the measurement of testability. In doing so, we introduce a prototype system, which uses the mutant schemata approach to calculate the infection estimate. The attempt, reported in this paper, to reproduce the results reported by Voas for one example, have shown that intermediate calculations are quite sensitive to minor variations in the parameters to the process, although the final testability results are in agreement.

NSEDIT: a syntax-directed editor and testing tool based on Nassi-Schneiderman charts

This paper describes a programming environment for Pascal, called NSEDIT, which runs under VAX/VMS VAX/VMS is a trademark of the Digital Equipment Corporation. . NSEDIT is based upon the notion of Nassi‐Shneiderman charts (abbreviated to N‐S charts in this paper) and is designed for ease of use by novice programmers in a teaching situation. A key component of the system is the editor, which enables structural features of Pascal to be manipulated in terms of their N‐S chart box representations. Simultaneous editing of more than one program or module is permitted. Another key feature of the system is a semiautomatic interface to the VAX/VMS Pascal compiler, with subsequent display of execution counts as part of the N‐S charts. In this way users can obtain immediate feedback on the coverage of program elements by test runs, while still inside an editing session.