David J. Manning

Viewing another person's eye movements improves identification of pulmonary nodules in chest x-ray inspection

Double reading of chest x-rays is often used to ensure that fewer abnormalities are missed, but very little is known about how the search behavior of others affects observer performance. A series of experiments investigated whether radiographers benefit from knowing where another person looked for pulmonary nodules, and whether the expertise of the model providing the search behavior was a contributing factor. Experiment 1 compared the diagnostic performance of novice and experienced radiographers examining chest x-rays and found that both groups performed better when shown the search behavior of either a novice radiographer or an expert radiologist. Experiment 2 established that benefits in performance only arose when the eye movements shown were related to the search for nodules; however, only the novices diagnostic performance consistently improved when shown the experts search behavior. Experiment 3 reexamined the contribution of task, image, and the expertise of the model underlying this benefit. Consistent with Experiment 1, novice radiographers were better at identifying nodules when shown either a naïves search behavior or an expert radiologists search behavior, but they demonstrated no improvement when shown a naïve model not searching for nodules. Our results suggest that although the benefits of this form of attentional guidance may be short-lived, novices can scaffold their decisions based on the search behavior of others.

A free-response evaluation determining value in the computed tomography attenuation correction image for revealing pulmonary incidental findings: a phantom study.

RATIONALE AND OBJECTIVESnThe purpose of this study was to compare lesion-detection performance when interpreting computed tomography (CT) images that are acquired for attenuation correction when performing single photon emission computed tomography/computed tomography (SPECT/CT) myocardial perfusion studies. In the United Kingdom, there is a requirement that these images be interpreted; thus, it is necessary to understand observer performance on these images.nnnMATERIALS AND METHODSnAn anthropomorphic chest phantom with inserted spherical lesions of different sizes and contrasts was scanned on five different SPECT/CT systems using site-specific CT protocols for SPECT/CT myocardial perfusion imaging. Twenty-one observers (0-4 years of CT experience) searched 26 image slices (17 abnormal, containing 1-3 lesions, and 9 normal, containing no lesions) for each CT acquisition. The observers marked and rated perceived lesions under the free-response paradigm. Four analyses were conducted using jackknife alternative free-response receiver operating characteristic (JAFROC) analysis: (1) 20-pixel acceptance radius (AR) with all 21 readers, abbreviated to 20/ALL analysis, (2) 40-pixel AR with 21 readers (40/ALL), (3) 20-pixel AR with 14 readers experienced in CT (20/EXP), and (4) 20-pixel AR with 7 readers with no CT experience (20/NOT). The significance level of the test was set so as to conservatively control the overall probability of a type I error to <0.05.nnnRESULTSnThe mean JAFROC figure of merit (FOM) for the five CT acquisitions for the 20/ALL study were 0.602, 0.639, 0.372, 0.475, and 0.719 with a significant difference in lesion-detection performance evident between all individual treatment pairs (P < .0001) with the exception of the 1-2 pairing, which was not significant (these differed only in milliamp seconds). System 5, which had the highest performance, had the smallest slice thickness and the largest matrix size. For the other analyses, the system orderings remained unchanged, and the significance of FOM difference findings remained identical to those for 20/ALL, with one exception: for 20/EXP analysis the 1-2 difference became significant with the higher milliamp seconds superior. Improved detection performance was associated with a smaller slice thickness, increased matrix size, and, to a lesser extent, increased tube charge.nnnCONCLUSIONSnProtocol variations for CT-based attenuation correction (AC) in SPECT/CT imaging have a measurable impact on lesion-detection performance. The results imply that z-axis resolution and matrix size had the greatest impact on lesion detection, with a weaker but detectable dependence on the product of milliamp and seconds.

The effect of computer-aided detection markers on visual search and reader performance during concurrent reading of CT colonography

AbstractObjectiveWe aimed to identify the effect of computer-aided detection (CAD) on visual search and performance in CT Colonography (CTC) of inexperienced and experienced readers.MethodsFifteen endoluminal CTC examinations were recorded, each with one polyp, and two videos were generated, one with and one without a CAD mark. Forty-two readers (17 experienced, 25 inexperienced) interpreted the videos during infrared visual search recording. CAD markers and polyps were treated as regions of interest in data processing. This multi-reader, multi-case study was analysed using multilevel modelling.ResultsCAD drew readers’ attention to polyps faster, accelerating identification times: median ‘time to first pursuit’ was 0.48xa0s (IQR 0.27 to 0.87xa0s) with CAD, versus 0.58xa0s (IQR 0.35 to 1.06xa0s) without. For inexperienced readers, CAD also held visual attention for longer. All visual search metrics used to assess visual gaze behaviour demonstrated statistically significant differences when “with” and “without” CAD were compared. A significant increase in the number of correct polyp identifications across all readers was seen with CAD (74xa0% without CAD, 87xa0% with CAD; pu2009<u20090.001).ConclusionsCAD significantly alters visual search and polyp identification in readers viewing three-dimensional endoluminal CTC. For polyp and CAD marker pursuit times, CAD generally exerted a larger effect on inexperienced readers.Key Points• Visual gaze is attracted by computer-assisted detection (CAD) marks on polypsn • Inexperienced readers’ gaze is affected more by CAD than experienced readers.• CAD marks could mean that the unannotated endoluminal surface is relatively neglected.• Correct polyp identification is increased significantly by CAD.

Accurate localization of incidental findings on the computed tomography attenuation correction image: the influence of tube current variation.

This observer performance study assessed lesion detection in the computed tomography attenuation correction image, as would be produced for myocardial perfusion imaging over a tube current (mA) range. A static anthropomorphic chest phantom containing simulated pulmonary lesions was scanned using the four available mA values (1, 1.5, 2 and 2.5) on a GE Infinia Hawkeye 4. All other computed tomography acquisition parameters remained constant throughout. Twenty-seven cases showing zero to four lesions were produced for a free-response receiver-operating characteristic method. Image observations were completed using our novel web-based ROCView software under controlled conditions. The Jackknife alternative free-response receiver-operating characteristic (JAFROC) figure of merit was used for significance testing, wherein a difference in lesion detection performance was considered significant at P values less than 0.05. Twenty readers with varying computed tomography experience (0–24 years) evaluated 108 images using an ordinal scale to score confidence. The JAFROC analysis showed that there was no statistically significant difference in performance between mA values (P=0.439) for this sample of observers. In conclusion, no significant difference in lesion detection performance was seen between the mA values. This suggests that there is no value in using anything other than the lowest mA value for the investigation of incidental extracardiac findings.

The value of observer performance studies in dose optimization: a focus on free-response receiver operating characteristic methods.

Receiver operating characteristic (ROC) analysis has been successfully used in radiology to help determine the combined success of system and observer. There is great value in these methods for assessing new and existing techniques to see if diagnostic accuracy can be improved. Within all aspects of radiology there should be compliance with the as-low-as-reasonably-achievable principle, which requires optimization of the diagnostic suitability of the image. Physical measures of image quality have long been used in the assessment of system performance, but these alone are not sufficient to assess diagnostic capability. It is imperative that the observer be included in any assessment of diagnostic performance. The free-response ROC paradigm has been developed as a statistically powerful advancement of traditional ROC analysis that allows a precise interpretation of complex images by adding location information to the level of observer confidence. The following review of free-response ROC methodology will explain how observer performance methods can be valuable in image optimization, including examples of how these have already been successful in hybrid imaging.

A software system for the simulation of chest lesions

We report on the development of a novel software tool for the simulation of chest lesions. This software tool was developed for use in our study to attain optimal ambient lighting conditions for chest radiology. This study involved 61 consultant radiologists from the American Board of Radiology. Because of its success, we intend to use the same tool for future studies. The software has two main functions: the simulation of lesions and retrieval of information for ROC (Receiver Operating Characteristic) and JAFROC (Jack-Knife Free Response ROC) analysis. The simulation layer operates by randomly selecting an image from a bank of reportedly normal chest x-rays. A random location is then generated for each lesion, which is checked against a reference lung-map. If the location is within the lung fields, as derived from the lung-map, a lesion is superimposed. Lesions are also randomly selected from a bank of manually created chest lesion images. A blending algorithm determines which are the best intensity levels for the lesion to sit naturally within the chest x-ray. The same software was used to run a study for all 61 radiologists. A sequence of images is displayed in random order. Half of these images had simulated lesions, ranging from subtle to obvious, and half of the images were normal. The operator then selects locations where he/she thinks lesions exist and grades the lesion accordingly. We have found that this software was very effective in this study and intend to use the same principles for future studies.

Classification of radiological errors in chest radiographs, using support vector machine on the spatial frequency features of false- negative and false-positive regions

Aim: To optimize automated classification of radiological errors during lung nodule detection from chest radiographs (CxR) using a support vector machine (SVM) run on the spatial frequency features extracted from the local background of selected regions. Background: The majority of the unreported pulmonary nodules are visually detected but not recognized; shown by the prolonged dwell time values at false-negative regions. Similarly, overestimated nodule locations are capturing substantial amounts of foveal attention. Spatial frequency properties of selected local backgrounds are correlated with human observer responses either in terms of accuracy in indicating abnormality position or in the precision of visual sampling the medical images. Methods: Seven radiologists participated in the eye tracking experiments conducted under conditions of pulmonary nodule detection from a set of 20 postero-anterior CxR. The most dwelled locations have been identified and subjected to spatial frequency (SF) analysis. The image-based features of selected ROI were extracted with un-decimated Wavelet Packet Transform. An analysis of variance was run to select SF features and a SVM schema was implemented to classify False-Negative and False-Positive from all ROI. Results: A relative high overall accuracy was obtained for each individually developed Wavelet-SVM algorithm, with over 90% average correct ratio for errors recognition from all prolonged dwell locations. Conclusion: The preliminary results show that combined eye-tracking and image-based features can be used for automated detection of radiological error with SVM. The work is still in progress and not all analytical procedures have been completed, which might have an effect on the specificity of the algorithm.

Searching in axial and 3D CT visualisations

Traditional diagnostic modalities have been, for the most part, static two-dimensional images displayed on film or computer screen. More recent diagnostic modalities are solely computer-based and consist of large data-sets of multiple images. Image perception and visual search using these new modalities are complicated by the need to interact with the computer in order to navigate through the data. This paper reports the late-breaking results from two small studies into visual search within two types of CT Colonography (CTC) visualisations. The twelve novice observers in the study were taking part in a week-long course in CTC and were tested at the beginning and end of the course. A number of expert observers were also recorded. The two visualisations used in the study were 2D axial view and 3D colon fly-through. In both cases, searching was performed by inspecting the colon wall, but by two distinct mechanisms. The first study recorded observer eye-gaze and image navigation in a CTC axial view. The search strategy was to follow the lumen of the colon and detect abnormalities in the colon wall. The observer used the physical computer interface to navigate through the set of axial images to perform this task. The 3D fly-through study recorded observer eye-gaze whilst watching a recording of a computed flight through the colon lumen. Unlike the axial view there was no computer control, so inspection of the colon surface was dictated by the speed of flight through the colon.

Effect of reconstruction methods and x-ray tube current-time product on nodule detection in an anthropomorphic thorax phantom: A crossed-modality JAFROC observer study.

Purpose: To evaluate nodule detection in an anthropomorphic chest phantom in computed tomography (CT) images reconstructed with adaptive iterative dose reduction 3D (AIDR3D) and filtered back projection (FBP) over a range of tube current–time product (mAs). Methods: Two phantoms were used in this study: (i) an anthropomorphic chest phantom was loaded with spherical simulated nodules of 5, 8, 10, and 12 mm in diameter and +100, −630, and −800 Hounsfield units electron density; this would generate CT images for the observer study; (ii) a whole-body dosimetry verification phantom was used to ultimately estimate effective dose and risk according to the model of the BEIR VII committee. Both phantoms were scanned over a mAs range (10, 20, 30, and 40), while all other acquisition parameters remained constant. Images were reconstructed with both AIDR3D and FBP. For the observer study, 34 normal cases (no nodules) and 34 abnormal cases (containing 1–3 nodules, mean 1.35 ± 0.54) were chosen. Eleven observers evaluated images from all mAs and reconstruction methods under the free-response paradigm. A crossed-modality jackknife alternative free-response operating characteristic (JAFROC) analysis method was developed for data analysis, averaging data over the two factors influencing nodule detection in this study: mAs and image reconstruction (AIDR3D or FBP). A Bonferroni correction was applied and the threshold for declaring significance was set at 0.025 to maintain the overall probability of Type I error at α = 0.05. Contrast-to-noise (CNR) was also measured for all nodules and evaluated by a linear least squares analysis. Results: For random-reader fixed-case crossed-modality JAFROC analysis, there was no significant difference in nodule detection between AIDR3D and FBP when data were averaged over mAs [F(1, 10) = 0.08, p = 0.789]. However, when data were averaged over reconstruction methods, a significant difference was seen between multiple pairs of mAs settings [F(3, 30) = 15.96, p < 0.001]. Measurements of effective dose and effective risk showed the expected linear dependence on mAs. Nodule CNR was statistically higher for simulated nodules on images reconstructed with AIDR3D (p < 0.001). Conclusions: No significant difference in nodule detection performance was demonstrated between images reconstructed with FBP and AIDR3D. mAs was found to influence nodule detection, though further work is required for dose optimization.

A phantom-based JAFROC observer study of two CT reconstruction methods: the search for optimisation of lesion detection and effective dose

Purpose: To investigate the dose saving potential of iterative reconstruction (IR) in a computed tomography (CT) examination of the thorax. Materials and Methods: An anthropomorphic chest phantom containing various configurations of simulated lesions (5, 8, 10 and 12mm; +100, -630 and -800 Hounsfield Units, HU) was imaged on a modern CT system over a tube current range (20, 40, 60 and 80mA). Images were reconstructed with (IR) and filtered back projection (FBP). An ATOM 701D (CIRS, Norfolk, VA) dosimetry phantom was used to measure organ dose. Effective dose was calculated. Eleven observers (15.11±8.75 years of experience) completed a free response study, localizing lesions in 544 single CT image slices. A modified jackknife alternative free-response receiver operating characteristic (JAFROC) analysis was completed to look for a significant effect of two factors: reconstruction method and tube current. Alpha was set at 0.05 to control the Type I error in this study. Results: For modified JAFROC analysis of reconstruction method there was no statistically significant difference in lesion detection performance between FBP and IR when figures-of-merit were averaged over tube current (F(1,10)=0.08, p = 0.789). For tube current analysis, significant differences were revealed between multiple pairs of tube current settings (F(3,10) = 16.96, p<0.001) when averaged over image reconstruction method. Conclusion: The free-response study suggests that lesion detection can be optimized at 40mA in this phantom model, a measured effective dose of 0.97mSv. In high-contrast regions the diagnostic value of IR, compared to FBP, is less clear.