Leng Dong

Measuring performance in the interpretation of chest radiographs: a pilot study

AIM To develop a system to assess the image interpretation performance of radiologists in identifying signs of malignancy on chest radiographs. MATERIALS AND METHODS A test set of 30 chest radiographs was chosen by an experienced radiologist consisting of 11 normal and 19 abnormal cases. The malignant cases all had biopsy-proven pathology; the normal and benign cases all had at least 2 years of imaging follow-up. Fourteen radiologists with a range of experiences were recruited. Participants individually read the test set displayed on a standard reporting workstation, with their findings entered directly into a laptop running specially designed reporting software. For each case, relevant clinical information was given and the reader was asked to mark any perceived abnormality and rate their level of suspicion on a five-point scale (normal, benign, indeterminate, suspicious, or malignant). On completion, participants were given instant feedback with performance parameters including sensitivity and specificity automatically calculated. An opportunity was then given to review the cases together with an expert opinion and pathology. The time each participant took to complete the test was recorded. RESULTS Six consultant radiologists who took part showed significantly better performance as determined by receiver operating characteristic (ROC) analysis compared to eight specialist registrars (area under the ROC curve [AUC]=0.9297 and 0.7648 respectively, p=0.003). There was a significant correlation with years of experience in the interpretation of chest radiographs and performance on the test set (r=0.573, p=0.032). Consultant radiologists completed the test significantly more quickly that the specialist registrars: mean time 19.65 minutes compared to 26.51 minutes (p=0.033). CONCLUSION It is possible to use a test set to measure individual differences in the interpretation of chest radiographs. This has the potential to be a useful tool in performance testing.

Laparoscopic surgical skills training: an investigation of the potential of using surgeons' visual search behaviour as a performance indicator

Laparoscopic surgery is a difficult perceptual-motor task and effective and efficient training in the technique is important. Viewing previously recorded laparoscopic operations is a possible available training technique for surgeons to increase their knowledge of such minimal access surgery (MAS). It is not well known whether this is a useful technique, how effective it is or what effect it has on the surgeon watching the recorded video. As part of an on-going series of studies into laparoscopic surgery, an experiment was conducted to examine whether surgical skill level has an effect on the visual search behaviour of individuals of different surgical experience when they examine such imagery. Medically naive observers, medical students, junior surgeons and experienced surgeons viewed a laparoscopic recording of a recent operation. Initial examination of the recorded eye movement data indicated commonalities between all observers, largely irrespective of surgical experience. This, it is argued, is due to visual search in this situation largely being driven by the dynamic nature of the images. The data were then examined in terms of surgical steps and also in terms of interventions when differences were found related to surgical experience. Consequently, it is argued that monitoring the eye movements of trainee surgeons whilst they watch pre-recorded operations is a potential useful adjunct to existing training regimes.

Breast screening: understanding case difficulty and the nature of errors

In the UK all screeners undertake the PERFORMS scheme where they read annual sets of challenging cases. During this assessment, they give each case a confidence rating on whether it should be recalled. If they decide to recall a case, they also indicate the center of any key mammographic features on a display of the relevant mammographic case view. Expert radiological opinion defines what the key abnormalities (targets) are in any case. Data can then be analyzed using ROC and JAFROC approaches, and particularly for the latter, assessing whether a user has correctly located a feature or not is important. Using image pixel information alone it is possible to delineate correct localization of an abnormality from an incorrect location by defining an area of interest. To explore such location information in more detail, data from the last year of the PERFORMS scheme were reanalyzed and the location responses for each of the 675 participants on 120 screening cases examined. Additionally, expert radiological opinions had been garnered for various reasons, including accurately delineating any abnormalities. An algorithmic approach is developed which assesses whether users’ indications should be included as correct abnormality identification or not, based on the feedback location information of all participants’ indicated locations and the relative position of an indicated location to the abnormality. This approach is proposed to be superior to simple pixel distance approaches which measure a fixed distance from the centre of a target to the user’s indicated location. The approach adds to the experimenter’s repertoire of tools when examining user errors and case difficulty in medical imaging research.

A potential method to identify poor breast screening performance

In the UK all breast screeners undertake the PERFORMS scheme where they annually read case sets of challenging cases. From the subsequent data it is possible to identify any individual who is performing significantly lower than their peers. This can then facilitate them being offered further targeted training to improve performance. However, currently this under-performance can only be calculated once all screeners have taken part, which means the feedback can potentially take several months. To determine whether such performance outliers could usefully be identified approximately much earlier the data from the last round of the scheme were re-analysed. From the information of 283 participants, 1,000 groups of them were selected randomly for fixed group sizes varying from four to 50 individuals. After applying bootstrapping on 1,000 groups, a distribution of low performance threshold values was constructed. Then the accuracy of estimation was determined by calculating the median value and standard error of this distribution as compared with the known actual results. Data indicate that increasing sample sizes improved the estimation of the median and decreased the standard error. Using information from as few as 25 individuals allowed an approximation of the known outlier cut off value and this improved with larger sample sizes. This approach is now implemented in the PERFORMS scheme to enable individuals who have difficulties, as compared to their peers, to be identified very early after taking part which can then help them to improve their performance.

Early identification of substandard breast screening performers

All UK breast screeners can voluntarily undertake the PERFORMS scheme, where they examine recent screening cases receiving immediate feedback. Once all individuals have participated, data are calculated on how they performed compared with peers allowing poor performers to be identified. A way of potentially identifying such poor performers much earlier is proposed.

Analysis of visual search behaviour from experienced radiologists interpreting digital breast tomosynthesis (DBT) images: a pilot study

Digital Breast Tomosynthesis has several advantages over traditional 2D mammography. However, the cost-effectiveness to implement DBT modality into breast screening programmes is still under investigation. The DBT modality has been integrated into a regional breast screening program in Italy for several years. The purpose of this study is to examine the experienced Italian DBT readers’ visual search behaviour and summarise their visual inspection patterns. Seven experienced radiologists took part in the study, reading a set of DBT cases with a mixture of both normal and abnormal cases whilst their eye movements data were recorded. They read the cases through a fixed procedure starting with a 2D overview and then went through the DBT view of each side of the breasts. It was found that the experienced readers tended to perform a global-focal scan over the 2D view to detect the abnormality and then ‘drilled’ through the DBT slices, interpreting the details of the feature. The reading speed was also investigated to see if there was any difference in length of time when expert radiologists examine both normal and abnormal cases. The results showed that there was no significant difference in time between normal and abnormal cases. The eye movement patterns revealed that experienced DBT readers covered more areas on the 2D view and fixated longer and with more dwells inside the AOI in the 3D view. Based on these findings it is hoped that by understanding the visual search patterns of the experienced DBT radiologists, it could potentially help DBT trainees to develop more efficient interpretation approaches.

Exploring the potential of analysing visual search behaviour data using FROC (free-response receiver operating characteristic) method: an initial study

Visual search techniques and FROC analysis have been widely used in radiology to understand medical image perceptual behaviour and diagnostic performance. The potential of exploiting the advantages of both methodologies is of great interest to medical researchers. In this study, eye tracking data of eight dental practitioners was investigated. The visual search measures and their analyses are considered here. Each participant interpreted 20 dental radiographs which were chosen by an expert dental radiologist. Various eye movement measurements were obtained based on image area of interest (AOI) information. FROC analysis was then carried out by using these eye movement measurements as a direct input source. The performance of FROC methods using different input parameters was tested. The results showed that there were significant differences in FROC measures, based on eye movement data, between groups with different experience levels. Namely, the area under the curve (AUC) score evidenced higher values for experienced group for the measurements of fixation and dwell time. Also, positive correlations were found for AUC scores between the eye movement data conducted FROC and rating based FROC. FROC analysis using eye movement measurements as input variables can act as a potential performance indicator to deliver assessment in medical imaging interpretation and assess training procedures. Visual search data analyses lead to new ways of combining eye movement data and FROC methods to provide an alternative dimension to assess performance and visual search behaviour in the area of medical imaging perceptual tasks.

How quickly do breast screeners learn their skills

The UK’s Breast Screening Programme is 27 years old and many experienced breast radiologists are now retiring, coupled with an influx of new screening personnel. It is important to the ongoing Programme that new mammography readers are quickly up to the skill level of experienced readers. This raises the question of how quickly the necessary cancer detection skills are learnt. All breast screening radiologists in the UK read educational training sets of challenging FFDM images (the PERFORMS® scheme) yearly to maintain and improve their performance in real life screening. Data were examined from the PERFORMS® annual scheme for 54 new screeners, 55 screeners who have been screening for one year and also for more experienced screeners (597 screeners). Not surprisingly, significant differences in cancer detection rate were found between new readers and both of the other groups. Additionally, the performance of 48 new readers who have now been screening for about a year and have taken part twice in the PERFORMS® scheme were further examined where again a significant difference in cancer detection was found. These data imply that cancer detection skills are learnt quickly in the first year of screening. Information was also examined concerning the volume of cases participants read and other factors.

Is there a difference in reading time when normal and abnormal DBT cases are examined by DBT experienced radiologists? [Poster]

Poor resources in breast imaging across the country have prompted us to think about how we investigate younger patients. The incidence of breast cancer in women and men under the age of 24 is very low (1.5 per 100000 in the UK 2012-4), yet assessing this cohort can prove a significant imaging burden and contribute to patient anxiety. We have therefore reviewed the practice regarding 16-24 years at two large centres on the south coast over the last four years. Our hope is to be able to create better guidance on how best to investigate these patients in the future. The review included clinical, imaging and histopathology findings. Our review showed approximately 4000 Breast Ultrasounds were carried out in over 3000 patients, with 251 breast biopsies being performed. Of these biopsies, only 10 had histology of B3 or above, with just two proving malignant. Our data suggest clinical suspicion, known high family risk, and behavior of the breast mass are better predictors of malignancy than imaging. Learning objectives: Is tissue sampling valuable in this age group? Are features in the presentation better predictors of diagnosis than imaging? What are the key imaging features to look for? When is pre-operative tissue sampling valuable? How can we best use our limited workforce and other resources? PB.58 Phyllodes tumours: review of imaging characteristics of benign, borderline and malignant tumours over a 10-year period Nuala Healy, Gormlaith Hargaden, Fidelma Flanagan, Clare Smith, Angela O’Brien Eccles Unit BreastCheck (The Irish National Breast Screening Program), Dublin, Ireland; Department of Radiology, Mater Misericordiae University Hospital, Dublin, Ireland Correspondence: Nuala Healy Breast Cancer Research 2017, 19(Suppl 1):PB.58 Introduction Phyllodes tumours are rare fibroepithelial lesions, accounting for 0.3 – 0.5% of breast tumours. They are subdivided into benign, borderline and malignant lesions based on specific pathological features. As malignant lesions have the propensity to rapidly grow and metastasize it is important to identify these lesions and treat them appropriately. The aim of this study was to review all Phyllodes tumours identified at our institution over a 10-year period, to determine the imaging features of benign, borderline and malignant Phyllodes tumours. Methods A prospectively maintained database was interrogated to identify cases of Phyllodes tumours from January 2008 to June 2017. Basic demographic features were recorded and cases were reviewed to determine the imaging features, including size, US, mammographic and MRI features. Histology and individual follow-up of cases were also reviewed to determine outcomes. Results A total of 50 patients were identified with Phyllodes tumours over the 10-year period, 10 malignant, 5 borderline and 35 benign. The average size of the malignant lesions was 8.2cm, 4.2 cm for borderline and 2.8 cm for benign lesions. 70% of the malignant lesions were markedly hypoechoic and 60% had a heterogeneous echogenicity on US, 17% of the benign and borderline tumours were markedly hypoechoic and 30% had a heterogeneous echogenicity. One malignant and one benign Phyllodes developed recurrence. Conclusion In this series, 20% of Phyllodes tumours were malignant, being larger than the benign and borderline lesions and more likely to be markedly hypoechoic and have a heterogeneous echotexture on US.This is a poster presented at the British Society of Breast Radiology Annual Scientific Meeting 2017 in Dublin on 5-7th November.

Eye tracking method compatible with dual-screen mammography workstation

In this paper a new approach is proposed to track the perceptual behaviour of radiologists when they examine mammographic images displayed on large dual clinical monitors. Zooming and panning are inevitably performed by the radiologist to examine such large images by using the DICOM viewing software. Such image manipulating movements on the target displays makes eye tracking techniques difficult to perform and also the size of the dual clinical monitors makes existing eye tracking techniques generally inadequate. Hence a method using the Smart Eye Pro eye tracker and optical character recognition techniques was designed to relate the recorded radiologists’ eye gaze behaviour on the monitors to the actual zoomed and panned medical image areas. This then allows clinical studies involving radiologists interacting with these mammographic images to be successfully carried out.