Validation of Natural Language Processing to Determine the Presence and Size of Abdominal Aortic Aneurysms in a Large Integrated Health System.

Abstract

OBJECTIVE\nPrior studies of the natural history of abdominal aortic aneurysms (AAA) have been limited by small cohort sizes or heterogeneous analyses of pooled data. By quickly and efficiently extracting imaging data from the health record, natural language processing (NLP) has potential to substantially improve the way we study and care for patients with AAA. The aim of this study was to test the ability of an NLP tool to accurately identify the presence or absence of AAA and detect the maximal abdominal aortic diameter in a large dataset of imaging-study reports.\n\n\nMETHODS\nRelevant imaging study reports (n=230,660) from 2003-2017 were obtained for 32,778 patients followed in a prospective aneurysm surveillance registry within a large, diverse integrated healthcare system. A commercially available NLP algorithm was used to assess the presence of AAA, confirm the absence of AAA, and extract the maximal diameter of the abdominal aorta, if stated. Blinded expert manual review of 18,000 randomly selected imaging reports was used as the gold standard. Positive predictive value (PPV or precision) and sensitivity (recall) were calculated along with Kappa statistics.\n\n\nRESULTS\nOf the randomly selected 18,000 studies that underwent expert review, 48.7% were positive for AAA. In confirming the presence of a AAA, the inter-rater reliability of the NLP compared to expert review shows a kappa value of 0.84 (95% CI 0.83, 0.85) with a PPV of 95% and a sensitivity of 88.5%. The NLP algorithm shows similar results for confirming the absence of AAA with a kappa of 0.79 (95% CI 0.799, 0.80), PPV of 77.7%, and sensitivity of 91.9%. The kappa, PPV, and sensitivity of the NLP for correctly identifying the maximal aortic diameter are 0.88 (95% CI 0.87, 0.89), 88.8%, and 88.2% respectively.\n\n\nCONCLUSION\nUse of NLP software can accurately analyze large volumes of radiology report data to detect AAA disease and assemble a contemporary aortic diameter-based cohort of patients for longitudinal analysis to guide surveillance, medical management, and operative decision making. It can also potentially be used to identify pre- and post-operative AAA patients lost to follow-up in the EMR, leverage human resources engaged in the ongoing surveillance of AAA patients and facilitate the construction and implementation of AAA screening programs.