Sergey V. Beiden

Components-of-variance models and multiple-bootstrap experiments: An alternative method for random-effects, receiver operating characteristic analysis*

RATIONALE AND OBJECTIVES The purpose of this study was to develop an alternative approach to random-effects, receiver operating characteristic analysis inspired by a general formulation of components-of-variance models. The alternative approach is a higher-order generalization of the Dorfman, Berbaum, and Metz (DBM) approach that yields additional information on the variance structure of the problem. MATERIALS AND METHODS Six population experiments were designed to determine the six variance components in the DBM model. For practical problems, in which only a finite set of readers and patients are available, six analogous bootstrap experiments may be substituted for the population experiments to estimate the variance components. Monte Carlo simulations were performed on the population experiments, and those results were compared with the corresponding multiple-bootstrap estimates and those obtained with the DBM approach. Confidence intervals on the difference of ROC parameters for competing diagnostic modalities were estimated, and corresponding comparisons were made. RESULTS For mean values, the agreement of present estimates of variance structures with population results was excellent and, when suitably weighted and mixed, similar to or closer than that with the DBM method. For many variance structures, the confidence intervals in this study for the difference in ROC area between modalities were comparable to those with the DBM method. When reader variability was large, however, mean confidence intervals from this study were tighter than those with the DBM method and closer to population results. CONCLUSION The jackknife approach of DBM provides a linear approximation to receiver-operating-characteristic statistics that are intrinsically nonlinear. The multiple-bootstrap technique of this study, however, provides a more general, nonparametric, maximum-likelihood approach. It also yields estimates of the variance structure previously unavailable.

Assessment of medical imaging and computer-assist systems: lessons from recent experience.

In the last 2 decades major advances have been made in the field of assessment methods for medical imaging and computer-assist systems through the use of the paradigm of the receiver operating characteristic (ROC) curve. In the most recent decade this methodology was extended to embrace the complication of reader variability through advances in the multiple-reader, multiple-case (MRMC) ROC measurement and analysis paradigm. Although this approach has been widely adopted by the imaging research community, some investigators appear averse to it, possibly from concern that it could place a greater burden on the scarce resources of patient cases and readers compared to the requirements of alternative methods. The present communication argues, however, that the MRMC ROC approach to assessment in the context of reader variability may be the most resource-efficient approach available. Moreover, alternative approaches may also be statistically uninterpretable with regard to estimated summary measures of performance and their uncertainties. The authors propose that the MRMC ROC approach be considered even more widely by the larger community with responsibilities for the introduction and dissemination of medical imaging technologies to society. General principles of study design are reviewed, and important contemporary clinical trials are used as examples.

Continuous versus Categorical Data for ROC Analysis

RATIONALE AND OBJECTIVES Several authors have encouraged the use of a quasi-continuous rating scale for data collection in receiver operating characteristic (ROC) curve analysis of diagnostic modalities, rather than rating scales based on five to seven ordinal categories or levels of suspicion. Although many investigators have gone over to this method, a discussion of the issues continues. The present work provides a quantitative analysis from the viewpoint of measurement science. MATERIALS AND METHODS A simple model of the effect of data discretization or quantization on the measurement of the variance of noisy data was developed. Then Monte Carlo simulations of multiple-reader, multiple-case ROC experiments were performed and analyzed in terms of components-of-variance models to investigate the effect of data quantization in that more complex setting. RESULTS For single-reader studies, discretization into five categories can reduce the precision of ROC measurements by a large amount. The effect may be attenuated in multireader studies. CONCLUSION More precise measurements of diagnostic detection performance and thus more efficient use of resources are served by good measurement methods. These are promoted by the use of a quasi-continuous rating scale in ROC studies.

Components-of-Variance Models for Random-Effects ROC Analysis: The Case of Unequal Variance Structures Across Modalities

RATIONALE AND OBJECTIVES Several of the authors have previously published an analysis of multiple sources of uncertainty in the receiver operating characteristic (ROC) assessment and comparison of diagnostic modalities. The analysis assumed that the components of variance were the same for the modalities under comparison. The purpose of the present work is to obtain a generalization that does not require that assumption. MATERIALS AND METHODS The generalization is achieved by splitting three of the six components of variance in the previous model into modality-dependent contributions. Two distinct formulations of this approach can be obtained from alternative choices of the three components to be split; however, a one-to-one relationship exists between the magnitudes of the components estimated from these two formulations. RESULTS The method is applied to a study of multiple readers, with and without the aid of a computer-assist modality. performing the task of discriminating between benign and malignant clusters of microcalcifications. Analysis according to the first method of splitting shows large decreases in the reader and reader-by-case components of variance when the computer assist is used by the readers. Analysis in terms of the alternative splitting shows large decreases in the corresponding modality-interaction components. CONCLUSION A solution to the problem of multivariate ROC analysis without the assumption of equal variance structure across modalities has been provided. Alternative formulations lead to consistent results related by a one-to-one mapping. A surprising result is that estimates of confidence intervals and numbers of cases and readers required for a specified confidence interval remain the same in the more general model as in the restricted model.

Reader Variability in Mammography and Its Implications for Expected Utility over the Population of Readers and Cases

The multiple-reader, multiple-case (MRMC) approach to receiver operating characteristic (ROC) analysis is becoming the dominant assessment paradigm in medical imaging. Its most common version involves having many readers read every patient case in the study, a critical feature since differences among competing imaging modalities are often dominated by differences in reader performance. The present authors have carried out MRMC ROC analysis on a uniquely large data set for mammography. The analysis quantifies the great range of observed reader skill in that data set. It also demonstrates that the sample sizes are sufficiently large that the conclusions generalize to the populations sampled here with little uncertainty from the finite sample size. A schematic approach to bracketing the utility matrix is then used to study trends in the resulting expected utility functions that correspond to the range of observed ROC curves. This is done for both the screening and the diagnostic context. The results raise 2 hypotheses for further investigation. First, it is possible that the present ambiguity surrounding the effectiveness of mammography is due in part to the observed range of reader skills and corresponding expected utility functions. Second, it is possible that computer-assisted modalities for mammography may lead to improvements in the expected utility function not only for screening but also in the diagnostic context, especially for the lower performing readers.

A general model for finite-sample effects in training and testing of competing classifiers

The conventional wisdom in the field of statistical pattern recognition (SPR) is that the size of the finite test sample dominates the variance in the assessment of the performance of a classical or neural classifier. The present work shows that this result has only narrow applicability. In particular, when competing algorithms are compared, the finite training sample more commonly dominates this uncertainty. This general problem in SPR is analyzed using a formal structure recently developed for multivariate random-effects receiver operating characteristic (ROC) analysis. Monte Carlo trials within the general model are used to explore the detailed statistical structure of several representative problems in the subfield of computer-aided diagnosis in medicine. The scaling laws between variance of accuracy measures and number of training samples and number of test samples are investigated and found to be comparable to those discussed in the classic text of Fukunaga, but important interaction terms have been neglected by previous authors. Finally, the importance of the contribution of finite trainers to the uncertainties argues for some form of bootstrap analysis to sample that uncertainty. The leading contemporary candidate is an extension of the 0.632 bootstrap and associated error analysis, as opposed to the more commonly used cross-validation.

The problem of ROC analysis without truth: the EM algorithm and the information matrix

Henkelman, Kay, and Bronskill (HKB) showed that although the problem of ROC analysis without truth is underconstrained and thus not uniquely solvable in one dimension (one diagnostic test), it is in principle solvable in two or more dimensions. However, they gave no analysis of the resulting uncertainties. The present work provides a maximum-likelihood solution using the EM (expectation-maximization) algorithm for the two- dimensional case. We also provide an analysis of uncertainties in terms of Monte Carlo simulations as well as estimates based on Fisher Information Matrices for the complete- and the missing-data problem. We find that the number of patients required for a given precision of estimate for the truth- unknown problem is a very large multiple of that required for the corresponding truth-known case.

Multiple-reader studies, digital mammography, computer-aided diagnosis-and the Holy Grail of imaging physics (II)

The metaphor of the Holy Grail is used here to refer to the classic and elusive problem in medical imaging of predicting the ranking of the clinical performance of competing imaging modalities from the ranking obtained from physical laboratory measurements and signal-detection analysis, or from simple phantom studies. We show how the use of the multiple-reader, multiple-case (MRMC) ROC paradigm and new analytical techniques allows this masking effect to be quantified in terms of components-of-variance models. Moreover, we demonstrate how the components of variance associated with reader variability may be reduced when readers have the benefit of computer-assist reading aids. The remaining variability will be due to the case components, and these reflect the contribution of the technology without the masking effect of the reader. This suggests that prediction of clinical ranking of imaging systems in terms of physical measurements may become a much more tractable task in a world that includes MRMC ROC analysis of performance of radiologists with the advantage of computer-assisted reading.

Contemporary issues for experimental design in assessment of medical imaging and computer-assist systems

The dialog among investigators in academia, industry, NIH, and the FDA has grown in recent years on topics of historic interest to attendees of these SPIE sub-conferences on Image Perception, Observer Performance, and Technology Assessment. Several of the most visible issues in this regard have been the emergence of digital mammography and modalities for computer-assisted detection and diagnosis in breast and lung imaging. These issues appear to be only the “tip of the iceberg” foreshadowing a number of emerging advances in imaging technology. So it is timely to make some general remarks looking back and looking ahead at the landscape (or seascape). The advances have been facilitated and documented in several forums. The major role of the SPIE Medical Imaging Conferences i well-known to all of us. Many of us were also present at the Medical Image Perception Society and co-sponsored by CDRH and NCI in September of 2001 at Airlie House, VA. The workshops and discussions held at that conference addressed some critical contemporary issues related to how society - and in particular industry and FDA - approach the general assessment problem. A great deal of inspiration for these discussions was also drawn from several workshops in recent years sponsored by the Biomedical Imaging Program of the National Cancer Institute on these issues, in particular the problem of “The Moving Target” of imaging technology. Another critical phenomenon deserving our attention is the fact that the Fourth National Forum on Biomedical Imaging in Oncology was recently held in Bethesda, MD., February 6-7, 2003. These forums are presented by the National Cancer Institute (NCI), the Food and Drug Administration (FDA), the Centers for Medicare and Medicaid Services (CMS), and the National Electrical Manufacturers Association (NEMA). They are sponsored by the National Institutes of Health/Foundation for Advanced Education in the Sciences (NIH/FAES). These forums led to the development of the NCI’s Interagency Council on Biomedical Imaging in Oncology (ICBIO) about two and a half years ago. The purpose of the ICBIO is to assist developers of new imaging technologies for cancer screening and diagnosis to find a coherent way to interface with government agencies with responsibilities in these areas. A recent product of these activities was an overview paper written by the present authors and published this year in the Journal Academic Radiology (2). The paper includes a summary of some of the major developments in assessment methodology in recent years and includes several case studies from the public forum of the FDA’s Center for Devices & Radiological Health (CDRH). We will include a brief sketch of some of the key issues of that paper in this review.

Independent versus sequential reading in ROC studies of computer-assist modalities

This paper provides results of a statistical analysis of two methods for arranging the temporal sequencing of the unaided vs computer-assisted reading in multiple-reader, multiple- case (MRMC) receiver operating characteristic (ROC) studies of computer-aided detection of solitary pulmonary nodules (SPNs) on chest radiographs. The modes are the Independent mode, in which the readings are separated by a time on the order of one month, and the Sequential mode, in which the CAD-assisted reading immediately follows the unassisted reading. The method of Beiden, Wagner, Campbell (BWC) was used to decompose the variance of the ROC area summary accuracy measure into the components that are correlated across unaided and aided reading conditions and the components that are uncorrelated across these reading conditions. The latter are the only components of variability that contribute to the uncertainty in a measurement of the difference in reader performance between reading conditions. These uncorrelated components were dramatically reduced in the Sequential reading mode compared to the Independent reading mode-while the total reader variability remained almost constant. The results were remarkably similar across two independent studies analyzed. This may have important practical consequences because the Sequential mode is the least demanding on reader logistics and time.