Charles E. Metz

Some practical issues of experimental design and data analysis in radiological ROC studies.

Receiver operating characteristic (ROC) analysis has been used in a broad variety of medical imaging studies during the past 15 years, and its advantages over more traditional measures of diagnostic performance are now clearly established. But despite the essential simplicity of the approach, workers in the field often find--sometimes only after an ROC study is under way--that a number of subtle issues related to experimental design and data analysis must be confronted in practice. Many of these issues have not been discussed in the literature in detail, and most are not well known. The purposes of this paper are to make users of ROC methodology in medical imaging aware of potential problems that should be confronted before an ROC study is begun and to indicate, at least broadly, how those problems may be dealt with, given the present state of the art. Some of the issues raised here can be addressed adequately by easily prescribed techniques, whereas others remain difficult and will be resolved fully only by new methodologic developments.

Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data

We show that truth-state runs in rank-ordered data constitute a natural categorization of continuously-distributed test results for maximum likelihood (ML) estimation of ROC curves. On this basis, we develop two new algorithms for fitting binormal ROC curves to continuously-distributed data: a true ML algorithm (LABROC4) and a quasi-ML algorithm (LABROC5) that requires substantially less computation with large data sets. Simulation studies indicate that both algorithms produce reliable estimates of the binormal ROC curve parameters a and b, the ROC-area index Az, and the standard errors of those estimates.

A New Approach for Testing the Significance of Differences Between ROC Curves Measured from Correlated Data

Receiver Operating Characteristic analysis is now generally recognized as the most appropriate methodology for evaluating the diagnostic performance of medical imaging procedures (1–7). ROC analysis has been used in the field of psychophysics for three decades, and its theory and experimental methodology have been developed in considerable detail (8–13). Perhaps surprisingly, the statistical properties of ROC measures had received relatively little attention until several years ago, when the limited size of practical data sets in medical applications indicated the need for careful study of this issue. Recent progress in the statistical analysis of ROC data includes the work of Metz and Kronman (14,15), who developed a bivariate test for the statistical significance of differences between ROC curves measured from independent data sets; the work of Hanley and McNeil, who studied the statistical properties of the area under an ROC curve and developed techniques to predict the number of cases required tc demonstrate the significance of differences between ROC “Area Indexes” measured from either independent (16) or correlated (17) data sets; and the work of Swets and Pickett (7), who identified three components of variation in ROC measures and outlined a general statistical protocol for testing the significance of differences in the Area Index.

Statistical Comparison of Two ROC-curve Estimates Obtained from Partially-paired Datasets

The authors propose a new generalized method for ROC-curve fitting and statistical testing that allows researchers to utilize all of the data collected in an experimental comparison of two diagnostic modalities, even if some patients have not been studied with both modalities. Their new algorithm, ROCKIT, subsumes previous algorithms as special cases. It conducts all analyses available from previous ROC software and provides 95% confidence intervals for all estimates. ROCKIT was tested on more than half a million computer-simulated datasets of various sizes and configurations repre senting a range of population ROC curves. The algorithm successfully converged for more than 99.8% of all datasets studied. The type I error rates of the new algorithms statistical test for differences in Az estimates were excellent for datasets typically en countered in practice, but diverged from alpha for datasets arising from some extreme situations. Key words. receiver operating characteristic (ROC) analysis, maximum-like lihood estimation; partially-paired data; missing data. (Med Decis Making 1998;18: 110-121)

The Exponential Radon Transform

The exponential Radon transform, a generalization of the Radon transform, is defined and is studied as a mapping of function spaces. An inversion formula is derived. The exponential Radon transform is represented in terms of Fourier transforms of its domain and range, and this leads to a characterization of the range of the transform.The exponential Radon transform is a model for some reconstruction problems in radionuclide emission computer tomography.

Computerized detection of masses in digital mammograms: Analysis of bilateral subtraction images

A computerized scheme is being developed for the detection of masses in digital mammograms. Based on the deviation from the normal architectural symmetry of the right and left breasts, a bilateral subtraction technique is used to enhance the conspicuity of possible masses. The scheme employs two pairs of conventional screen-film mammograms (the right and left mediolateral oblique views and craniocaudal views), which are digitized by a TV camera/Gould digitizer. The right and left breast images in each pair are aligned manually during digitization. A nonlinear bilateral subtraction technique that involves linking multiple subtracted images has been investigated and compared to a simple linear subtraction method. Various feature-extraction techniques are used to reduce false-positive detections resulting from the bilateral subtraction. The scheme has been evaluated using 46 pairs of clinical mammograms and was found to yield a 95% true-positive rate at an average of three false-positive detections per image. This preliminary study indicates that the scheme is potentially useful as an aid to radiologists in the interpretation of screening mammograms.

A unified analysis of exact methods of inverting the 2-D exponential radon transform, with implications for noise control in SPECT

Exact methods of inverting the two-dimensional (2-D) exponential Radon transform have been proposed by Bellini et al. (1979) and by Inouye et al. (1989), both of whom worked in the spatial-frequency domain to estimate the 2-D Fourier transform of the unattenuated sinogram; by Hawkins et al. (1988), who worked with circularly harmonic Bessel transforms; and by Tretiak and Metz (1980), who followed filtering of appropriately-modified projections by exponentially-weighted backprojection. With perfect sampling, all four of these methods are exact in the absence of projection-data noise, but empirical studies have shown that they propagate noise differently, and no underlying theoretical relationship among the methods has been evident. Here, an analysis of the 2-D Fourier transform of the modified sinogram reveals that all previously-proposed linear methods can be interpreted as special cases of a broad class of methods, and that each method in the class can be implemented, in principle, by any one of four distinct techniques. Moreover, the analysis suggests a new member of the class that Is predicted to have noise properties better than those of previously-proposed members.

Statistical significance tests for binormal ROC curves

Abstract Statistical significance tests are derived and evaluated for measuring apparent differences between an obtained and an expected binormal ROC curve, between two independent binormal ROC curves, and among groups of independent binormal ROC curves. A binormal ROC curve is described by two parameters which represent the spread of the means and the ratio of the standard deviations of the two underlying Gaussian decision variable distributions. To test the significance of apparent differences between or among ROC curves, approximate χ 2 statistics for each of the three tests were constructed from maximum likelihood estimates of the two parameters defining the binormal ROC curve. The performance of each test statistic was evaluated by simulating five-category rating scale data with equal numbers of noise and signal-plus-noise trials (set at 50, 250, and 500) for each of three typical ROC curves. For the significance test involving only one ROC curve, rating scale data were generated from the chance diagonal of the ROC space also. Although test performance was found to be somewhat dependent on the number of trials and on the location of the ROC curve in the ROC space, comparisons of the obtained and expected fractions of (falsely) significant results at various α levels showed the proposed statistical significance tests to be reliable under practical experimental conditions.

The geometric transfer function component for scintillation camera collimators with straight parallel holes.

A theoretical approach has been developed that allows the geometric transfer function component for conventional scintillation camera collimators to be predicted in closed form. If transfer function analysis is to be useful in describing imaging system performance, the image of a point source must not depend on source position in a plane parallel to the detection plane. This shift invariance can be achieved by analysis of system response in terms of an effective point spread function, defined as the normalised image of a point source that would be obtained if the camera collimator were uniformly translated (but not rotated) during image formation. The geometric component of the corresponding effective transfer function is shown to be expressed simply by the absolute square of the two-dimensional Fourier transform of a collimator hole aperture, with the spatial frequency plane scaled by a factor which depends on collimator length, source-to-collimator distance, and collimator-to-detection plane distance. Closed form algebraic expressions of the geometric transfer function have been obtained for all four common hold shapes (circular, hexagonal, square and triangular). Monte Carlo simulations and experimental measurements have shown these theoretical expressions to be highly accurate.

Effect of case selection on the performance of computer‐aided detection schemes

The choice of clinical cases used to train and test a computer-aided diagnosis (CAD) scheme can affect the test results (i.e., error rate). In this study, we deliberately modified the components of our testing database to study the effects of this modification on measured performance. Using a computerized scheme for the automated detection of breast masses from mammograms, it was found that the sensitivity of the scheme ranged between 26% and 100% (at a false positive rate of 1.0 per image) depending on the cases used to test the scheme. Even a 20% change in the cases comprising the database can reduce the measured sensitivity by 15%-25%. Because of the strong dependence of measured performance on the testing database, it is difficult to estimate reliably the accuracy of a CAD scheme. Furthermore, it is questionable to compare different CAD schemes when different cases are used for testing. Sharing databases, creating a common database, or using a quantitative measure to characterize databases are possible solutions to this problem. However, none of these solutions exists or is practiced at present. Therefore, as a short-term solution, it is recommended that the method used for selecting cases, and histograms or mean and standard deviations of relevant image features be reported whenever performance data are presented.