Warwick Lee

Malignancy Detection in Digital Mammograms: Important Reader Characteristics and Required Case Numbers

RATIONALE AND OBJECTIVES To determine the relationship between heightened levels of reader performance and reader practice in terms of number of cases read and previous experience. MATERIALS AND METHODS A test set of mammograms was developed comprising 50 cases. These cases consisted of 15 abnormals (biopsy proven) and 35 normals (confirmed at subsequent rescreen). Sixty-nine breast image readers reviewed these cases independently and their performance was measured by recording their individual receiver operating characteristic score (area under the curve), sensitivity, and specificity. These measures of performance were then compared to a range of factors relating to the reader such as years of certification and reporting, number of cases read per year, previous experiences, and satisfaction levels. Correlation analyses using Spearman methods were performed along with the Mann-Whitney test to detect differences in performance between specific reader groups. RESULTS Improved reader performance was found for years certified (P = .004), years of experience (P = .0001), and hours reading per week (P = .003) shown by positive statistical significant relationships with Az values (area under receiver operating characteristic curve). Statistical comparisons of Az values scored for individuals who read varying number of cases per year showed that those individuals whose annual mammographic case load was 5000 or more (P = .03) or between 2000 and 4999 (P = .05), had statistically significantly higher scores than those who read less than 1000 cases per year. CONCLUSION The results of this study have shown variations in reader performance relating to parameters of reader practice and experience. Levels of variance are shown and potential acceptance levels for diagnostic efficacy are proposed which may inform policy makers, judicial systems and public debate.

Screening mammography: test set data can reasonably describe actual clinical reporting.

PURPOSE To establish the extent to which test set reading can represent actual clinical reporting in screening mammography. MATERIALS AND METHODS Institutional ethics approval was granted, and informed consent was obtained from each participating screen reader. The need for informed consent with respect to the use of patient materials was waived. Two hundred mammographic examinations were selected from examinations reported by 10 individual expert screen readers, resulting in 10 reader-specific test sets. Data generated from actual clinical reports were compared with three test set conditions: clinical test set reading with prior images, laboratory test set reading with prior images, and laboratory test set reading without prior images. A further set of five expert screen readers was asked to interpret a common set of images in two identical test set conditions to establish a baseline for intraobserver variability. Confidence scores (from 1 to 4) were assigned to the respective decisions made by readers. Region-of-interest (ROI) figures of merit (FOMs) and side-specific sensitivity and specificity were described for the actual clinical reporting of each reader-specific test set and were compared with those for the three test set conditions. Agreement between pairs of readings was performed by using the Kendall coefficient of concordance. RESULTS Moderate or acceptable levels of agreement were evident (W = 0.69-0.73, P < .01) when describing group performance between actual clinical reporting and test set conditions that were reasonably close to the established baseline (W = 0.77, P < .01) and were lowest when prior images were excluded. Higher median values for ROI FOMs were demonstrated for the test set conditions than for the actual clinical reporting values; this was possibly linked to changes in sensitivity. CONCLUSION Reasonable levels of agreement between actual clinical reporting and test set conditions can be achieved, although inflated sensitivity may be evident with test set conditions.

Assessing reader performance in radiology, an imperfect science: lessons from breast screening.

The purpose of this article is to review the limitations associated with current methods of assessing reader accuracy in mammography screening programmes. Clinical audit is commonly used as a quality-assurance tool to monitor the performance of screen readers; however, a number of the metrics employed, such as recall rate as a surrogate for specificity, do not always accurately measure the intended clinical feature. Alternatively, standardized screening test sets, which benefit from ease of application, immediacy of results, and quicker assessment of quality improvement plans, suffer from experimental confounders, thus questioning the relevance of these laboratory-type screening test sets to clinical performance. Four key factors that impact on the external validity of screening test sets were identified: the nature and extent of scrutiny of ones action, the artificiality of the environment, the over-simplification of responses, and prevalence of abnormality. The impact of these factors on radiological and other contexts is discussed, and although it is important to acknowledge the benefit of standardized screening test sets, issues relating to the relevance of test sets to clinical activities remain. The degree of correlation between performance based on real-life clinical audit and performances at screen read test sets must be better understood and specific causal agents for any lack of correlation identified.

Quantitative measures confirm the inverse relationship between lesion spiculation and detection of breast masses.

OBJECTIVE To identify specific mammographic appearances that reduce the mammographic detection of breast cancer. MATERIALS AND METHODS This study received institutional board review approval and all readers gave informed consent. A set of 60 mammograms each consisting of craniocaudal and mediolateral oblique projections were presented to 129 mammogram Breastscreen readers. The images consisted of 20 positive cases with single and multicentric masses in 16 and 4 cases, respectively (resulting in a total of 24 cancers), and readers were asked to identify and locate the lesions. Each lesion was then ranked according to a detectability rating (ie, the number of observers who correctly located the lesion divided by the total number of observers), and this was correlated with breast density, lesion size, and various descriptors of lesion shape and texture. RESULTS Negative and positive correlations between lesion detection and density (r = -0.64, P = .007) and size (r = 0.65, P = .005), respectively, were demonstrated. In terms of lesion size and shape, there were significant correlations between the probability of detection and area (r = 0.43, P = .04), perimeter (r = 0.66, P = .0004), lesion elongation (r = 0.49, P = .02), and lesion nonspiculation (r = 0.78, P < .0001). CONCLUSIONS The results of this study have identified specific lesion characteristics associated with shape that may contribute to reduced cancer detection. Mammographic sensitivity may be adversely affected without appropriate attention to spiculation.

The impact of mammographic density and lesion location on detection

The aim of this study is to examine the impact of breast density and lesion location on detection. A set of 55 mammographic images (23 abnormal images with 26 lesions and 32 normal images) were examined by 22 expert radiologists. The images were classified by an expert radiologist according to the Synoptic Breast Imaging Report of the National Breast Cancer Centre (NBCC) as having low mammographic density (D1<25% glandular and D2> 25-50% glandular) or high density (D3 51-75% glandular and D4> 75-glandular). The observers freely examined the images and located any malignancy using a 5-point confidence. Performance was defined using the following metrics: sensitivity, location sensitivity, specificity, receiver operating characteristic (ROC Az) curves and jackknife free-response receiver operator characteristics (JAFROC) figures of merit. Significant increases in sensitivity (p= 0.0174) and ROC (p=0.0001) values were noted for the higher density compared with lower density images according to NBCC classification. No differences were seen in radiologists’ performance between lesions within or outside the fibroglandular region. In conclusion, analysis of our data suggests that radiologists scored higher using traditional metrics in higher mammographic density images without any improvement in lesion localisation. Lesion location whether within or outside the fibroglandular region appeared to have no impact on detection abilities suggesting that if a masking effect is present the impact is minimal. Eye-tracking analyses are ongoing.

Mammographic screening for breast cancer: A review

In 2011, BreastScreen Australia celebrated 20 years of mammographic screening for breast cancer in Australia. There has been a reduction in mortality from breast cancer over the last two decades, coincident with mammographic screening. However, there are concerns that mammographic screening may result in overdiagnosis of breast cancer and that the reduction in mortality from breast cancer is the result of better treatment rather than screening. This article reviews the evidence on which mammographic screening for breast cancer is based, considers the issue of overdiagnosis of breast cancer by screening mammography, and assesses the role of screening mammography in the reduction in breast cancer mortality seen over the last two decades.

Audit feedback on reading performance of screening mammograms: An international comparison.

Objective Providing feedback to mammography radiologists and facilities may improve interpretive performance. We conducted a web-based survey to investigate how and why such feedback is undertaken and used in mammographic screening programmes. Methods The survey was sent to representatives in 30 International Cancer Screening Network member countries where mammographic screening is offered. Results Seventeen programmes in 14 countries responded to the survey. Audit feedback was aimed at readers in 14 programmes, and facilities in 12 programmes. Monitoring quality assurance was the most common purpose of audit feedback. Screening volume, recall rate, and rate of screen-detected cancers were typically reported performance measures. Audit reports were commonly provided annually, but more frequently when target guidelines were not reached. Conclusion The purpose, target audience, performance measures included, form and frequency of the audit feedback varied amongst mammographic screening programmes. These variations may provide a basis for those developing and improving such programmes.

Certain performance values arising from mammographic test set readings correlate well with clinical audit

Test sets have been increasingly utilised to augment clinical audit in breast screening programmes; however, their relationship has never been satisfactorily understood. This study examined the relationship between mammographic test set performance and clinical audit data.

Mammography test sets: Reading location and prior images do not affect group performance

AIM To examine how the location where reading takes place and the availability of prior images can affect performance in breast test-set reading. MATERIALS AND METHODS Under optimized viewing conditions, 10 expert screen readers each interpreted a reader-specific set of images containing 200 mammographic cases. Readers, randomly divided into two groups read images under one of two pairs of conditions: clinical read with prior images and laboratory read with prior images; laboratory read with prior images and laboratory read without prior images. Region-of-interest (ROI) figure-of-merit (FOM) was analysed using JAFROC software. Breast side-specific sensitivity and specificity were tested using Wilcoxon matched-pairs signed rank tests. Agreement between pairs of readings was measured using Kendalls coefficient of concordance. RESULTS Group performances between test-set readings demonstrated similar ROI FOMs, sensitivity and specificity median values, and acceptable levels of agreement between pairs of readings were shown (W = 0.75-0.79, p < 0.001) for both pairs of reading conditions. On an individual reader level, two readers demonstrated significant decreases (p < 0.05) in ROI FOMs when prior images were unavailable. Reading location had an inconsistent impact on individual performance. CONCLUSION Reading location and availability of prior images did not significantly alter group performance.

Assessing the effect of a true-positive recall case in screening mammography: does perceptual priming alter radiologists' performance?

OBJECTIVE To measure the effect of the insertion of less-difficult malignant cases on subsequent breast cancer detection by breast imaging radiologists. METHODS The research comprises two studies. Study 1: 8 radiologists read 2 sets of images each consisting of 40 mammographic cases. Set A contained four abnormal cases, and Set B contained six abnormal cases, including two priming cases (less difficult malignancies) placed at intervals of three and five subsequent cases before a subtle cancer. Study 2: 16 radiologists read a third condition of the same cases, known as Set C, containing six abnormal cases and two priming cases immediately preceding the subtle cancer cases. The readers were asked to localize malignancies and give confidence ratings on decisions. RESULTS Although not significant, a decrease in performance was observed in Set B compared with in Set A. There was a significant increase in the receiver operating characteristic (ROC) area under the curve (z = -2.532; p = 0.0114) and location sensitivity (z = -2.128; p = 0.0333) between the first and second halves of Set A and a marginal improvement in jackknife free-response ROC figure of merit (z = -1.89; p = 0.0587) between the first and second halves of Set B. In Study 2, Set C yielded no significant differences between the two halves of the study. CONCLUSION Overall findings show no evidence that priming with lower difficulty malignant cases affects the detection of higher difficulty cancers; however, performance may decrease with priming. ADVANCES IN KNOWLEDGE This research suggests that inserting additional malignant cases in screening mammography sets as an audit tool may potentially lead to a decrease in performance of experienced breast radiologists.