Dev P. Chakraborty

Observer studies involving detection and localization: Modeling, analysis, and validation

Although the receiver operating characteristic (ROC) paradigm is the accepted method for evaluation of diagnostic imaging systems, it has some serious shortcomings inasmuch as it is restricted to one observer report per image. By contrast the free-response ROC (FROC) paradigm and associated analysis method allows the observer to report multiple abnormalities within each imaging study, and uses the location of reported abnormalities to improve the measurement. Because the ROC method cannot accommodate multiple responses or use location information, its statistical power will suffer. The FROC paradigm/analysis has not enjoyed widespread acceptance because of concern about whether responses made to the same diagnostic study can be treated as independent. We propose a new jackknife FROC analysis method (JAFROC) that does not make the independence assumption. The new analysis method combines elements of FROC and the Dorfman-Berbaum-Metz (DBM) methods. To compare JAFROC to an earlier free-response analysis method (specifically the alternative free-response, or AFROC method), and to the DBM method, which uses conventional ROC scoring, we developed a model for generating simulated FROC data. The simulation model is based on an eye-movement model of how experts evaluate images. It allowed us to examine null hypothesis (NH) behavior and statistical power of the different methods. We found that AFROC analysis did not pass the NH test, being unduly conservative. Both the JAFROC method and the DBM method passed the NH test, but JAFROC had more statistical power than the DBM method. The results of this comparison suggest that future studies of diagnostic performance may enjoy improved statistical power or reduced sample size requirements through the use of the JAFROC method.

Breast tomosynthesis and digital mammography: a comparison of diagnostic accuracy

OBJECTIVE Our aim was to compare the ability of radiologists to detect breast cancers using one-view breast tomosynthesis (BT) and two-view digital mammography (DM) in an enriched population of diseased patients and benign and/or healthy patients. METHODS All participants gave informed consent. The BT and DM examinations were performed with about the same average glandular dose to the breast. The study population comprised patients with subtle signs of malignancy seen on DM and/or ultrasonography. Ground truth was established by pathology, needle biopsy and/or by 1-year follow-up by mammography, which retrospectively resulted in 89 diseased breasts (1 breast per patient) with 95 malignant lesions and 96 healthy or benign breasts. Two experienced radiologists, who were not participants in the study, determined the locations of the malignant lesions. Five radiologists, experienced in mammography, interpreted the cases independently in a free-response study. The data were analysed by the receiver operating characteristic (ROC) and jackknife alternative free-response ROC (JAFROC) methods, regarding both readers and cases as random effects. RESULTS The diagnostic accuracy of BT was significantly better than that of DM (JAFROC: p=0.0031, ROC: p=0.0415). The average sensitivity of BT was higher than that of DM (∼90% vs ∼79%; 95% confidence interval of difference: 0.036, 0.108) while the average false-positive fraction was not significantly different (95% confidence interval of difference: -0.117, 0.010). CONCLUSION The diagnostic accuracy of BT was superior to DM in an enriched population.

Breast tissue density quantification via digitized mammograms

Studies reported in the literature indicate that breast cancer risk is associated with mammographic densities. An objective, repeatable, and a quantitative measure of risk derived from mammographic densities will be of considerable use in recommending alternative screening paradigms and/or preventive measures. However, image processing efforts toward this goal seem to be sparse in the literature, and automatic and efficient methods do not seem to exist. Here, the authors describe and validate an automatic and reproducible method to segment dense tissue regions from fat within breasts from digitized mammograms using scale-based fuzzy connectivity methods. Different measures for characterizing mammographic density are computed from the segmented regions and their robustness in terms of their linear correlation across two different projections-cranio-caudal and medio-lateral-oblique-are studied. The accuracy of the method is studied by computing the area of mismatch of segmented dense regions using the proposed method and using manual outlining. A comparison between the mammographic density parameter taking into account the original intensities and that just considering the segmented area indicates that the former may have some advantages over the latter.

Validation and Statistical Power Comparison of Methods for Analyzing Free-response Observer Performance Studies

RATIONALE AND OBJECTIVES The aim of this work was to validate and compare the statistical powers of proposed methods for analyzing free-response data using a search-model-based simulator. MATERIALS AND METHODS A free-response data simulator is described that can model a single reader interpreting the same cases in two modalities, or two computer-aided detection (CAD) algorithms, or two human observers, interpreting the same cases in one modality. A variance components model, analogous to the Roe and Metz receiver-operating characteristic (ROC) data simulator, is described; it models intracase and intermodality correlations in free-response studies. Two generic observers were simulated: a quasi-human observer and a quasi-CAD algorithm. Null hypothesis (NH) validity and statistical powers of ROC, jackknife alternative free-response operating characteristic (JAFROC), a variant of JAFROC termed JAFROC-1, initial detection and candidate analysis (IDCA), and a nonparametric (NP) approach were investigated. RESULTS All methods had valid NH behavior over a wide range of simulator parameters. For equal numbers of normal and abnormal cases, for the human observer, the statistical power ranking of the methods was JAFROC-1 > JAFROC > (IDCA approximately NP) > ROC. For the CAD algorithm, the ranking was (NP approximately IDCA) > (JAFROC-1 approximately JAFROC) > ROC. In either case, the statistical power of the highest ranked method exceeded that of the lowest ranked method by about a factor of two. Dependence of statistical power on simulator parameters followed expected trends. For data sets with more abnormal cases than normal cases, JAFROC-1 power significantly exceeded JAFROC power. CONCLUSION Based on this work, the recommendation is to use JAFROC-1 for human observers (including human observers with CAD assist) and the NP method for evaluating CAD algorithms.

Dose dependence of mass and microcalcification detection in digital mammography: free response human observer studies.

The purpose of this study was to evaluate the effect of dose reduction in digital mammography on the detection of two lesion types-malignant masses and clusters of microcalcifications. Two free-response observer studies were performed-one for each lesion type. Ninety screening images were retrospectively selected; each image was originally acquired under automatic exposure conditions, corresponding to an average glandular dose of 1.3 mGy for a standard breast (50 mm compressed breast thickness with 50% glandularity). For each study, one to three simulated lesions were added to each of 40 images (abnormals) while 50 were kept without lesions (normals). Two levels of simulated system noise were added to the images yielding two new image sets, corresponding to simulated dose levels of 50% and 30% of the original images (100%). The manufacturers standard display processing was subsequently applied to all images. Four radiologists experienced in mammography evaluated the images by searching for lesions and marking and assigning confidence levels to suspicious regions. The search data were analyzed using jackknife free-response (JA-FROC) methodology. For the detection of masses, the mean figure-of-merit (FOM) averaged over all readers was 0.74, 0.71, and 0.68 corresponding to dose levels of 100%, 50%, and 30%, respectively. These values were not statistically different from each other (F= 1.67, p=0.19) but showed a decreasing trend. In contrast, in the microcalcification study the mean FOM was 0.93, 0.67, and 0.38 for the same dose levels and these values were all significantly different from each other (F = 109.84, p < 0.0001). The results indicate that lowering the present dose level by a factor of two compromised the detection of microcalcifications but had a weaker effect on mass detection.

THE DIAGNOSTIC ACCURACY OF DUAL-VIEW DIGITAL MAMMOGRAPHY, SINGLE-VIEW BREAST TOMOSYNTHESIS AND A DUAL-VIEW COMBINATION OF BREAST TOMOSYNTHESIS AND DIGITAL MAMMOGRAPHY IN A FREE-RESPONSE OBSERVER PERFORMANCE STUDY

The purpose of the present study was to compare the diagnostic accuracy of dual-view digital mammography (DM), single-view breast tomosynthesis (BT) and BT combined with the opposite DM view. Patients with subtle lesions were selected to undergo BT examinations. Two radiologists who are non-participants in the study and have experience in using DM and BT determined the locations and extents of lesions in the images. Five expert mammographers interpreted the cases using the free-response paradigm. The task was to mark and rate clinically reportable findings suspicious for malignancy and clinically relevant benign findings. The marks were scored with reference to the outlined regions into lesion localization or non-lesion localization, and analysed by the jackknife alternative free-response receiver operating characteristic method. The analysis yielded statistically significant differences between the combined modality and dual-view DM (p < 0.05). No differences were found between single-view BT and dual-view DM or between single-view BT and the combined modality.

Diagnostic Efficacy of Handheld Devices for Emergency Radiologic Consultation

OBJECTIVE Orthopedic injury and intracranial hemorrhage are commonly encountered in emergency radiology, and accurate and timely diagnosis is important. The purpose of this study was to determine whether the diagnostic accuracy of handheld computing devices is comparable to that of monitors that might be used in emergency teleconsultation. SUBJECTS AND METHODS Two handheld devices, a Dell Axim personal digital assistant (PDA) and an Apple iPod Touch device, were studied. The diagnostic efficacy of each device was tested against that of secondary-class monitors (primary class being clinical workstation display) for each of two image types-posteroanterior wrist radiographs and slices from CT of the brain-yielding four separate observer performance studies. Participants read a bank of 30 wrist or brain images searching for a specific abnormality (distal radial fracture, fresh intracranial bleed) and rated their confidence in their decisions. A total of 168 readings by examining radiologists of the American Board of Radiology were gathered, and the results were subjected to receiver operating characteristics analysis. RESULTS In the PDA brain CT study, the scores of PDA readings were significantly higher than those of monitor readings for all observers (p < or = 0.01) and for radiologists who were not neuroradiology specialists (p < or = 0.05). No statistically significant differences between handheld device and monitor findings were found for the PDA wrist images or in the iPod Touch device studies, although some comparisons approached significance. CONCLUSION Handheld devices show promise in the field of emergency teleconsultation for detection of basic orthopedic injuries and intracranial hemorrhage. Further investigation is warranted.

Statistical Power in Observer-Performance Studies: Comparison of the Receiver Operating Characteristic and Free-Response Methods in Tasks Involving Localization

RATIONALE AND OBJECTIVES Statistical power, defined as the probability of detecting real differences between imaging modalities, determines the cost in terms of readers and cases of conducting receiver operating characteristic (ROC) studies. Neglect of location information in lesion-detection studies analyzed with the ROC method can compromise power. Use of the alternative free-response ROC (AFROC) method, which considers location information, has been discouraged, because it neglects intraimage correlations. The relative statistical powers of the two methods, however, have not been tested. The purpose of this study was to compare the statistical power of ROC and AFROC methods using simulations. MATERIALS AND METHODS A new model including intraimage correlations was developed to describe the decision variable sampling and to simulate data for ROC and AFROC analyses. Five readers and 200 cases (half of which contained one signal) were simulated for each trial. Two hundred trials, equally split between the null hypothesis and alternative hypothesis, were run. Ratings were analyzed with the Dorfman-Berbaum-Metz method, and separation of the null hypothesis and alternative hypothesis distributions was calculated. RESULTS The AFROC method yielded higher power than the ROC method. Separation of the null hypothesis and alternative hypothesis distributions was larger by a factor of 1.6 regardless of the presence or absence of intraimage correlations. The effect of the incorrect localizations during ROC analysis of localization data is believed to be the major reason for the enhanced power of the AFROC method. CONCLUSION The AFROC method can yield higher power than the ROC method for studies involving lesion localization. Greater consideration of this methodology is warranted.

A search model and figure of merit for observer data acquired according to the free-response paradigm

Search is a basic activity that is performed routinely in many different tasks. In the context of medical imaging it involves locating lesions in images under conditions of uncertainty regarding the number and locations of lesions that may be present. A search model is presented that applies to situations, as in the free-response paradigm, where on each image the number of normal regions that could be mistaken for lesions is unknown, and the number of observer generated localizations of suspicious regions (marks) is unpredictable. The search model is based on a two-stage model that has been proposed in the literature, according to which, at the first stage (the preattentive stage) the observer uses mainly peripheral vision to identify likely lesion candidates, and at the second stage the observer decides (i.e., cognitively evaluates) whether or not to report the candidates. The search model regards the unpredictable numbers of lesion and non-lesion localizations as random variables and models them via appropriate statistical distributions. The model has three parameters quantifying the lesion signal-to-noise ratio, the observers expertise at rejecting non-lesion locations, and the observers expertise at finding lesions. A figure-of-merit quantifying the observers search performance is described. The search model bears a close resemblance to the initial detection and candidate analysis (IDCA) model that has been recently proposed for analysing computer aided detection (CAD) algorithms. The ability to analytically model and quantify the search process would enable more powerful assessment and optimization of performance in these activities, which could be highly significant.

A comparison of two data analyses from two observer performance studies using Jackknife ROC and JAFROC

The authors compared two methodological approaches, Jackknife ROC and JAFROC, in analyzing data ascertained during FROC (free-response receiver operating characteristics) type studies. Observer rating data obtained from two observer performance studies were analyzed. During the first study, seven radiologists interpreted 120 mammography examinations depicting 57 masses under five different conditions with and without the results of computer-aided detection (CAD). In the second study, eight radiologists interpreted 110 examinations depicting 51 masses under six different display conditions with and without CAD results. Readers rated the detection task in a FROC type response. Jackknife ROC (using the software of LABMRMC with the highest rating per case) and JAFROC were used to compute differences, if any, in summary performance levels among all reading modes in each study as well as for all paired data sets. The results of the different analytical approaches are compared. The overall results for all modes were significantly different for the first study (p < 0.05) and not significant (p > 0.05) for the second study using either analytical approach. In the first study, the performance levels represented by three paired data sets were significantly different (p < 0.05) when computed using LABMRMC and four pairs were significantly different (p < 0.05) using JAFROC. In eight of ten pairs, JAFROC produced lower p values than LABMRMC. In the second study, LABMRMC showed no significant differences for any paired data sets and JAFROC showed a significant difference for one pair. In 15 of 16 pairs, p values computed by JAFROC were lower than those computed by LABMRMC.