Elaine J. Weyuker

Selecting Software Test Data Using Data Flow Information

This paper defines a family of program test data selection criteria derived from data flow analysis techniques similar to those used in compiler optimization. It is argued that currently used path selection criteria, which examine only the control flow of a program, are inadequate quate. Our procedure associates with each point in a program at which a variable is defined, those points at which the value is used. Several test data selection criteria, differing in the type and number of these associations, are defined and compared.

Evaluating software complexity measures

A set of properties of syntactic software complexity measures is proposed to serve as a basis for the evaluation of such measures. Four known complexity measures are evaluated and compared using these criteria. This formalized evaluation clarifies the strengths and weaknesses of the examined complexity measures, which include the statement count, cyclomatic number, effort measure, and data flow complexity measures. None of these measures possesses all nine properties, and several are found to fail to possess particularly fundamental properties; this failure calls into question their usefulness in measuring synthetic complexity. >

Predicting the location and number of faults in large software systems

Advance knowledge of which files in the next release of a large software system are most likely to contain the largest numbers of faults can be a very valuable asset. To accomplish this, a negative binomial regression model has been developed and used to predict the expected number of faults in each file of the next release of a system. The predictions are based on the code of the file in the current release, and fault and modification history of the file from previous releases. The model has been applied to two large industrial systems, one with a history of 17 consecutive quarterly releases over 4 years, and the other with nine releases over 2 years. The predictions were quite accurate: for each release of the two systems, the 20 percent of the files with the highest predicted number of faults contained between 71 percent and 92 percent of the faults that were actually detected, with the overall average being 83 percent. The same model was also used to predict which files of the first system were likely to have the highest fault densities (faults per KLOC). In this case, the 20 percent of the files with the highest predicted fault densities contained an average of 62 percent of the systems detected faults. However, the identified files contained a much smaller percentage of the code mass than the files selected to maximize the numbers of faults. The model was also used to make predictions from a much smaller input set that only contained fault data from integration testing and later. The prediction was again very accurate, identifying files that contained from 71 percent to 93 percent of the faults, with the average being 84 percent. Finally, a highly simplified version of the predictor selected files containing, on average, 73 percent and 74 percent of the faults for the two systems.

An applicable family of data flow testing criteria

The authors extend the definitions of the previously introduced family of data flow testing criteria to apply to programs written in a large subset of Pascal. They then define a family of adequacy criteria called feasible data flow testing criteria, which are derived from the data-flow testing criteria. The feasible data flow testing criteria circumvent the problem of nonapplicability of the data flow testing criteria by requiring the test data to exercise only those definition-use associations which are executable. It is shown that there are significant differences between the relationships among the data flow testing criteria and the relationships among the feasible data flow testing criteria. The authors discuss a generalized notion of the executability of a path through a program unit. A script of a testing session using their data flow testing tool, ASSET, is included. >

Analyzing partition testing strategies

Partition testing strategies, which divide a programs input domain into subsets with the tester selecting one or more elements from each subdomain, are analyzed. The conditions that affect the efficiency of partition testing are investigated, and comparisons of the fault detection capabilities of partition testing and random testing are made. The effects of subdomain modifications on partition testings ability to detect faults are studied. >

Testing component-based software: a cautionary tale

Components designed for reuse are expected to lower costs and shorten the development life cycle, but this may not prove so simple. The author emphasizes the need to closely examine a problematic aspect of component reuse: the necessity and potential expense of validating components in their new environments.

Automatically generating test data from a Boolean specification

This paper presents a family of strategies for automatically generating test data for any implementation intended to satisfy a given specification that is a Boolean formula. The fault detection effectiveness of these strategies is investigated both analytically and empirically, and the costs, assessed in terms of test set size, are compared. >

Theories of Program Testing and the Application of Revealing Subdomains

The theory of test data selection proposed by Goodenough and Gerhart is examined. In order to extend and refine this theory, the concepts of a revealing test criterion and a revealing subdomain are proposed. These notions are then used to provide a basis for constructing program tests.

Experience with performance testing of software systems: issues, an approach, and case study

An approach to software performance testing is discussed. A case study describing the experience of using this approach for testing the performance of a system used as a gateway in a large industrial client/server transaction processing application is presented.

A formal analysis of the fault-detecting ability of testing methods

Several relationships between software testing criteria, each induced by a relation between the corresponding multisets of subdomains, are examined. The authors discuss whether for each relation R and each pair of criteria, C/sub 1/ and C/sub 2/, R(C/sub 1/, C/sub 2/) guarantees that C/sub 1/ is better at detecting faults than C/sub 2/ according to various probabilistic measures of fault-detecting ability. It is shown that the fact that C/sub 1/ subsumes C/sub 2/ does not guarantee that C/sub 1/ is better at detecting faults. Relations that strengthen the subsumption relation and that have more bearing on fault-detecting ability are introduced. >