Jill L. King

Empiric assessment of parameters that affect the design of multireader receiver operating characteristic studies

RATIONALE AND OBJECTIVES The authors attempted to assess experimentally the magnitude of reader variability and the correlations and interactions among cases, readers, and modalities during observer performance studies and their possible effects on study design and sample size. MATERIALS AND METHODS Published data from 32 selected receiver operating characteristic (ROC) studies were reviewed to compare the magnitude of the variance component from readers with the variance component from modality. Estimates of correlation and interactions among cases, readers, and modalities were also computed directly from ROC data ascertained during two large studies performed in our laboratory. Each of these two studies included 529 cases and six readers, but one study used eight modalities and the other nine. RESULTS Published results indicate that reader variability is task dependent and larger (P < .05) than modality variability in detection of interstitial disease. Measured correlations between modalities for the same reader were task dependent and ranged from 0.35 to 0.59. Modality-by-reader and modality-by-case interactions often are not important factors. The random error term was greater than the modality-by-reader interaction in 11 of 20 comparisons and greater than the modality-by-case interaction in eight of 20 comparisons. CONCLUSION Use of the same cases interpreted with different modes is justifiable in many situations because of the high variability from readers. This comprehensive review of existing ROC studies resulted in parameter assessments that can be used to better estimate sample-size requirements in multireader ROC studies.

Practical Issues of Experimental ROC Analysis Selection of Controls

Receiver operating characteristics (ROC) analysis has been used in many medical imaging applications during the past decade. With the recent expansion of the ROC methodology to multi-disease studies, several reports have begun to address both the theoretical and experimental design issues associated with such studies. While the appropriate selection, classification, and verification of actually positive cases is carefully addressed in the literature, similar considerations are rarely given to the selection of actually negative controls for these studies. In this paper, theoretical considerations and experimental data are provided to demonstrate the significance of this very issue.

Forced choice and ordinal discrete rating assessment of image quality: A comparison

This study compared a five-category ordinal scale and a two-alternative forced-choice subjective rating of image quality preferences in a multiabnormality environment. 140 pairs of laser-printed posteroanterior (PA) chest images were evaluated twice by three radiologists who were asked to select during a side-by-side review which image in each pair was the “better” one for the determination of the presence or absence of specific abnormalities. Each pair included one image (the digitized film at 100 μm pixel resolution and laser printed onto film) and a highly compressed (∼60∶1) and decompressed version of the digitized film that was laser printed onto film. Ratings were performed once with a five-category ordinal scale and once with a two-alternative forced-choice scale. The selection process was significantly affected by the rating scale used. The “comparable” or “equivalent for diagnosis” category was used in 88.5% of the ratings with the ordinal scale. When using the two-alternative forced-choice approach, noncompressed images were selected 66.8% of the time as being the “better” images. This resulted in a significantly lower ability to detect small differences in perceived image quality between the noncompressed and compressed images when the ordinal rating scale is used. Observer behavior can be affected by the type of question asked and the rating scale used. Observers are highly sensitive to small differences in image presentation during a side-by-side review.

Computed radiography versus screen-film mammography in detection of simulated microcalcifications: a receiver operating characteristic study based on phantom images.

RATIONALE AND OBJECTIVES The authors compare a 43-micron computed radiographic system with a mammographic screen-film system for detection of simulated microcalcifications in an observer-performance study. MATERIALS AND METHODS The task of detecting microcalcifications was simulated by imaging aluminum wire segments (200-500 microns in length; 100, 125, or 150 microns in diameter) that overlapped with tissue background structures produced by beef brisket. A total of 288 such simulations were generated and examined with both computed radiography and conventional screen-film mammography techniques. Computed radiography was performed with high-resolution plates, a 43-micron image reader, and a 43-micron laser film printer. Computed radiographic images were printed with simple contrast enhancement and compared with screen-film images in a receiver operating characteristic study in which experienced readers detected and scored the simulated microcalcifications. Observer performance was quantitated and compared by computing the area under the receiver operating characteristic curve. RESULTS Although the resolution of the computed radiography system was better than that of commercial systems, it fell short of that of screen-film systems. For the 100-micron microcalcifications, the difference in the average area under the curve was not statistically significant, but it was significant for the larger simulated microcalcifications: the average area under the curve was 0.58 for computed radiography versus 0.76 for screen-film imaging for the 125-micron microcalcifications and 0.83 versus 1.00, respectively, for the 150-micron microcalcifications. CONCLUSION Observer performance in the detection of small simulated microcalcifications (100-150 microns in diameter) is better with screen-film images than with high-resolution computed radiographic images.

Observer variation and the performance accuracy gained by averaging ratings of abnormality.

Six radiologists used continuous scales to rate 529 chest-film cases for likelihood of five different types of abnormalities (interstitial disease, nodule, pneumothorax, alveolar infiltrate, and rib fracture) in each of six replicated readings, yielding 36 separate ratings of each case for the five abnormalities. Separate data analyses of all cases and subsets of the difficult/subtle cases for each abnormality estimated the relative gains in accuracy (linear-scaled area below the ROC curve) obtained by averaging the case-ratings across (a) six independent replications by each reader (25% gain), (b) six different readers within each replication (34% gain), or (c) all 36 readings (48% gain). Although accuracy differed among both readers and abnormalities, ROC curves for the median ratings showed similar relative gains in accuracy, somewhat greater than those predicted from the measured rating correlations. A model for variance components in the observers latent decision variable could predict these gains from measured correlations in the single ratings of cases. Depending on whether the models estimates were based on realized accuracy gains or on rating correlations, about 48% or 39% of each readers total decision variance (summed variance for positive and negative cases) consisted of random (within-reader) error that was uncorrelated between replications, another 10% or 14% came from idiosyncratic responses to individual cases, and about 43% or 47% was systematic variation that all readers found in the sampled cases.

On the generalization of the receiver operating characteristic analysis to the population of readers and cases with the jackknife method: An assessment

RATIONALE AND OBJECTIVES A new methodology that analyzes receiver operating characteristic (ROC) data sets based on jackknifing and that considers both case and reader variability has been proposed. The purpose of this investigation was to compare results using this method to those using commonly reported methodology. METHODS ROC data sets using discrete and continuous rating scales were analyzed using the proposed jackknifing method, and results were compared to analysis of the same data sets using the paired t test. RESULTS The two methodologies did not result in the same significance levels, and in some cases, the difference was sufficient to affect conclusions regarding comparisons of diagnostic modalities. The probability value for the jackknifing procedure is based on large sample distribution theory, and its appropriateness is unknown for sample sizes used in practice. Also, the jackknifing technique was found to be sensitive to outliers resulting when data from the computer programs used to estimate area under the ROC curve failed to converge. CONCLUSION Although the proposed methodology yields reasonable results, several fundamental and practical issues must be addressed before it can be used widely as the analytic method of choice in ROC studies comparing different imaging techniques or reading environments.

Simultaneous and sequential display of ICU AP-chest images

The display of a large number of projection radiographs (e.g., AP chest images) for comparison purposes poses potential problems for any electronic environment. In an attempt to assess the concept of rapid sequential viewing, 10 series of AP chest images were each reviewed on a high-resolution workstation under two conditions: (1) simultaneous display of each series in a mosaic configuration; and (2) separate image display in which each image was viewed individually in a rapid sequential mode. In our study, the sequential display was believed subjectively to be of comparable or higher quality by four of six readers. Diagnostic performance (patient improved; no change; patient condition worsened) was comparable for both display modes. Readers were somewhat more comfortable with the simultaneous (mosaic) configuration. Our preliminary results indicate that after minimal training, rapid sequential viewing of AP-chest images may be a reasonable alternative for the display of a series of AP chest images in the ICU.

Correlations between time required for radiological diagnoses, readers' performance, display environments, and difficulty of cases

In a series of large ROC studies, the authors analyzed the time radiologists took to diagnose PA chest images as a function of observer performance indices (Az), display environments, and difficulty of cases. Board-certified radiologists interpreted at least 600 images each for the presence or absence of one or more of the following abnormalities: interstitial disease, nodule, and pneumothorax. Results indicated that there exists a large inter- reader variability in the time required to diagnose PA chest images. There is no correlation between a readers specific median reading time and his/her performance. Time generally increases as the number of abnormalities on a single image increases and for cases with subtle abnormalities. Results also indicated that, in general, the longer the time for interpretation of a specific case (within reader), the further the observers confidence ratings were from the truth. These findings were found to hold true regardless of the display mode. These results may have implications with regards to the appropriate methodology that should be used for imaging systems evaluations and for measurements of productivity for radiologists.

Training observers for receiver operating characteristic studies

In an ongoing, multi-reader, multi-project program dealing with the interpretation of radiological images, we have examined several issues of methodology which have not as yet been addressed that may impact on the determination of reader performance as measured by receiver operating characteristic (ROC) analysis. Among these are issues associated with the training of observers prior to such studies. We employed three types of observer training that we found to be necessary for the successful performance of such studies: 1) a general instructional session for observers on the study protocol and system operation; 2) practice with an interactive computerized feedback teaching file that demonstrates the imaging systems and familiarized readers with an idea of the types of cases that were used in the study along with their correct diagnoses; and 3) training sessions in which readers were taught the manner in which to distribute answer ratings over an ordinal confidence scale. The possible effect of such types of training on the performance and results of ROC studies should be carefully considered prior to their commencement.

Variability in reader selection of subtle cases for inclusion in a reduced size receiver operating characteristic (ROC) study

Sample size in ROC studies may be significantly reduced by including only difficult cases in the image set, but variability across studies may be a possible obstacle in the development of such a methodology due to case selection. To assess this situation, 300 cases used in a previous large ROC study, which included nine observers, were independently classified as subtle or typical by two experienced readers. Data from the previous study were reanalyzed using data sets consisting only subtle or typical images as designated by each classifier. Results showed a marked decrease in observer performance from the original studys results when only the subtle cases of either classifier were included in the analyses. For 12 of 15 possible comparisons (3 imaging modes and 3 diseases for subtle cases of either classifier were included in the analyses. For 12 of 15 possible comparisons (3 imaging modes and 3 diseases for subtle cases, 3 modes and 2 diseases for typical cases), the Spearman rho correlation coefficient between the performance indices computed for each reader for the subsets classified as subtle and typical was high and significant (P < 0.05). The results obtained in this preliminary study are encouraging and point out areas that warrant further investigation.