Jeffrey P. Ferraro

Improving performance of natural language processing part-of-speech tagging on clinical narratives through domain adaptation

OBJECTIVE Natural language processing (NLP) tasks are commonly decomposed into subtasks, chained together to form processing pipelines. The residual error produced in these subtasks propagates, adversely affecting the end objectives. Limited availability of annotated clinical data remains a barrier to reaching state-of-the-art operating characteristics using statistically based NLP tools in the clinical domain. Here we explore the unique linguistic constructions of clinical texts and demonstrate the loss in operating characteristics when out-of-the-box part-of-speech (POS) tagging tools are applied to the clinical domain. We test a domain adaptation approach integrating a novel lexical-generation probability rule used in a transformation-based learner to boost POS performance on clinical narratives. METHODS Two target corpora from independent healthcare institutions were constructed from high frequency clinical narratives. Four leading POS taggers with their out-of-the-box models trained from general English and biomedical abstracts were evaluated against these clinical corpora. A high performing domain adaptation method, Easy Adapt, was compared to our newly proposed method ClinAdapt. RESULTS The evaluated POS taggers drop in accuracy by 8.5-15% when tested on clinical narratives. The highest performing tagger reports an accuracy of 88.6%. Domain adaptation with Easy Adapt reports accuracies of 88.3-91.0% on clinical texts. ClinAdapt reports 93.2-93.9%. CONCLUSIONS ClinAdapt successfully boosts POS tagging performance through domain adaptation requiring a modest amount of annotated clinical data. Improving the performance of critical NLP subtasks is expected to reduce pipeline error propagation leading to better overall results on complex processing tasks.

Natural language processing to identify pneumonia from radiology reports

This study aimed to develop Natural Language Processing (NLP) approaches to supplement manual outcome validation, specifically to validate pneumonia cases from chest radiograph reports.

Performance and Utilization of an Emergency Department Electronic Screening Tool for Pneumonia

A ppropriate treatment of pneumonia begins with accurate diagnosis. However, clinicians have difficulty integrating data for clinical decision making. Significant variability in pneumonia management exists in the emergency department (ED). Decision support might decrease variability and improve care, but physician utilization is historically low. An alerting tool is needed for physicians to utilize computer-based pneumonia decision support. We developed a real-time electronic screening tool that identifies patients with pneumonia by applying Bayesian probabilistic logic to the electronic medical record, and we implemented the tool in 4 EDs. A “P” appears on the ED electronic tracker board when pneumonia likelihood reaches 40%. Clicking on the icon displays pneumonia likelihood and the relevant data (Figure). After confirming the diagnosis, the ED physician proceeds with a linked decision support tool that provides management recommendations.

A study of the transferability of influenza case detection systems between two large healthcare systems

Objectives This study evaluates the accuracy and transferability of Bayesian case detection systems (BCD) that use clinical notes from emergency department (ED) to detect influenza cases. Methods A BCD uses natural language processing (NLP) to infer the presence or absence of clinical findings from ED notes, which are fed into a Bayesain network classifier (BN) to infer patients’ diagnoses. We developed BCDs at the University of Pittsburgh Medical Center (BCDUPMC) and Intermountain Healthcare in Utah (BCDIH). At each site, we manually built a rule-based NLP and trained a Bayesain network classifier from over 40,000 ED encounters between Jan. 2008 and May. 2010 using feature selection, machine learning, and expert debiasing approach. Transferability of a BCD in this study may be impacted by seven factors: development (source) institution, development parser, application (target) institution, application parser, NLP transfer, BN transfer, and classification task. We employed an ANOVA analysis to study their impacts on BCD performance. Results Both BCDs discriminated well between influenza and non-influenza on local test cases (AUCs > 0.92). When tested for transferability using the other institution’s cases, BCDUPMC discriminations declined minimally (AUC decreased from 0.95 to 0.94, p<0.01), and BCDIH discriminations declined more (from 0.93 to 0.87, p<0.0001). We attributed the BCDIH decline to the lower recall of the IH parser on UPMC notes. The ANOVA analysis showed five significant factors: development parser, application institution, application parser, BN transfer, and classification task. Conclusion We demonstrated high influenza case detection performance in two large healthcare systems in two geographically separated regions, providing evidentiary support for the use of automated case detection from routinely collected electronic clinical notes in national influenza surveillance. The transferability could be improved by training Bayesian network classifier locally and increasing the accuracy of the NLP parser.

The design and evaluation of a Bayesian system for detecting and characterizing outbreaks of influenza

The prediction and characterization of outbreaks of infectious diseases such as influenza remains an open and important problem. This paper describes a framework for detecting and characterizing outbreaks of influenza and the results of testing it on data from ten outbreaks collected from two locations over five years. We model outbreaks with compartment models and explicitly model non-influenza influenza-like illnesses.

Comparison of 2 Natural Language Processing Methods for Identification of Bleeding Among Critically Ill Patients

Key Points Question Can a natural language processing approach that uses text from clinical notes identify bleeding events among critically ill patients? Findings In this diagnostic study of a rules-based natural language processing model to identify bleeding events using clinical notes, the model was superior to a machine learning approach, with high sensitivity and negative predictive value. The extra trees machine learning model had high sensitivity but poor positive predictive value. Meaning Bleeding complications can be detected with a high-throughput natural language processing algorithm, an approach that can be used for quality improvement and prevention programs.

Epidemiology and Clinical Features of Invasive Fungal Infection in a US Health Care Network

Abstract Background A better understanding of the epidemiology and clinical features of invasive fungal infection (IFI) is integral to improving outcomes. We describe a novel case-finding methodology, reporting incidence, clinical features, and outcomes of IFI in a large US health care network. Methods All available records in the Intermountain Healthcare Enterprise Data Warehouse from 2006 to 2015 were queried for clinical data associated with IFI. The resulting data were overlaid in 124 different combinations to identify high-probability IFI cases. The cohort was manually reviewed, and exclusions were applied. European Organization for Research and Treatment of Cancer/Invasive Fungal Infections Cooperative Group and the National Institute of Allergy and Infectious Diseases Mycoses Study Group Consensus Group definitions were adapted to categorize IFI in a broad patient population. Linear regression was used to model variation in incidence over time. Results A total of 3374 IFI episodes occurred in 3154 patients. The mean incidence was 27.2 cases/100 000 patients per year, and there was a mean annual increase of 0.24 cases/100 000 patients (P = .21). Candidiasis was the most common (55%). Dimorphic fungi, primarily Coccidioides spp., comprised 25.1% of cases, followed by Aspergillus spp. (8.9%). The median age was 55 years, and pediatric cases accounted for 13%; 26.1% of patients were on immunosuppression, 14.9% had autoimmunity or immunodeficiency, 13.3% had active malignancy, and 5.9% were transplant recipients. Lymphopenia preceded IFI in 22.1% of patients. Hospital admission occurred in 76.2%. The median length of stay was 16 days. All-cause mortality was 17.0% at 42 days and 28.8% at 1 year. Forty-two-day mortality was highest in Aspergillus spp. (27.5%), 20.5% for Candida, and lowest for dimorphic fungi (7.5%). Conclusions In this population, IFI was not uncommon, affected a broad spectrum of patients, and was associated with high crude mortality.

A Bayesian system to detect and characterize overlapping outbreaks

Outbreaks of infectious diseases such as influenza are a significant threat to human health. Because there are different strains of influenza which can cause independent outbreaks, and influenza can affect demographic groups at different rates and times, there is a need to recognize and characterize multiple outbreaks of influenza. This paper describes a Bayesian system that uses data from emergency department patient care reports to create epidemiological models of overlapping outbreaks of influenza. Clinical findings are extracted from patient care reports using natural language processing. These findings are analyzed by a case detection system to create disease likelihoods that are passed to a multiple outbreak detection system. We evaluated the system using real and simulated outbreaks. The results show that this approach can recognize and characterize overlapping outbreaks of influenza. We describe several extensions that appear promising.

The Role of Coreference Resolution in Outbreak Reporting and Detection

A coreference resolution system was developed using publicly available text documents unrelated to public health. The purpose of this study was to evaluate the accuracy of the same system on disease outbreak reports. The unmodified system was able to correctly identify 73.91% of outbreak report references, showing that coreference resolution can perform well in health-related text even when trained on unrelated text. We discuss challenges of CR specific to the public health domain and how using CR is an important step in automatically identifying and extracting data otherwise hidden by referring expressions.