Steven Horng

Electronic medical record phenotyping using the anchor and learn framework

BACKGROUND Electronic medical records (EMRs) hold a tremendous amount of information about patients that is relevant to determining the optimal approach to patient care. As medicine becomes increasingly precise, a patients electronic medical record phenotype will play an important role in triggering clinical decision support systems that can deliver personalized recommendations in real time. Learning with anchors presents a method of efficiently learning statistically driven phenotypes with minimal manual intervention. MATERIALS AND METHODS We developed a phenotype library that uses both structured and unstructured data from the EMR to represent patients for real-time clinical decision support. Eight of the phenotypes were evaluated using retrospective EMR data on emergency department patients using a set of prospectively gathered gold standard labels. RESULTS We built a phenotype library with 42 publicly available phenotype definitions. Using information from triage time, the phenotype classifiers have an area under the ROC curve (AUC) of infection 0.89, cancer 0.88, immunosuppressed 0.85, septic shock 0.93, nursing home 0.87, anticoagulated 0.83, cardiac etiology 0.89, and pneumonia 0.90. Using information available at the time of disposition from the emergency department, the AUC values are infection 0.91, cancer 0.95, immunosuppressed 0.90, septic shock 0.97, nursing home 0.91, anticoagulated 0.94, cardiac etiology 0.92, and pneumonia 0.97. DISCUSSION The resulting phenotypes are interpretable and fast to build, and perform comparably to statistically learned phenotypes developed with 5000 manually labeled patients. CONCLUSION Learning with anchors is an attractive option for building a large public repository of phenotype definitions that can be used for a range of health IT applications, including real-time decision support.

Prospective pilot study of a tablet computer in an Emergency Department.

BACKGROUND The recent availability of low-cost tablet computers can facilitate bedside information retrieval by clinicians. OBJECTIVE To evaluate the effect of physician tablet use in the Emergency Department. DESIGN Prospective cohort study comparing physician workstation usage with and without a tablet. SETTING 55,000 visits/year Level 1 Emergency Department at a tertiary academic teaching hospital. PARTICIPANTS 13 emergency physicians (7 Attendings, 4 EM3s, and 2 EM1s) worked a total of 168 scheduled shifts (130 without and 38 with tablets) during the study period. INTERVENTION Physician use of a tablet computer while delivering direct patient care in the Emergency Department. MAIN OUTCOME MEASURES The primary outcome measure was the time spent using the Emergency Department Information System (EDIS) at a computer workstation per shift. The secondary outcome measure was the number of EDIS logins at a computer workstation per shift. RESULTS Clinician use of a tablet was associated with a 38min (17-59) decrease in time spent per shift using the EDIS at a computer workstation (p<0.001) after adjusting for clinical role, location, and shift length. The number of logins was also associated with a 5-login (2.2-7.9) decrease per shift (p<0.001) after adjusting for other covariates. CONCLUSION Clinical use of a tablet computer was associated with a reduction in the number of times physicians logged into a computer workstation and a reduction in the amount of time they spent there using the EDIS. The presumed benefit is that decreasing time at a computer workstation increases physician availability at the bedside. However, this association will require further investigation.

Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning

Objective To demonstrate the incremental benefit of using free text data in addition to vital sign and demographic data to identify patients with suspected infection in the emergency department. Methods This was a retrospective, observational cohort study performed at a tertiary academic teaching hospital. All consecutive ED patient visits between 12/17/08 and 2/17/13 were included. No patients were excluded. The primary outcome measure was infection diagnosed in the emergency department defined as a patient having an infection related ED ICD-9-CM discharge diagnosis. Patients were randomly allocated to train (64%), validate (20%), and test (16%) data sets. After preprocessing the free text using bigram and negation detection, we built four models to predict infection, incrementally adding vital signs, chief complaint, and free text nursing assessment. We used two different methods to represent free text: a bag of words model and a topic model. We then used a support vector machine to build the prediction model. We calculated the area under the receiver operating characteristic curve to compare the discriminatory power of each model. Results A total of 230,936 patient visits were included in the study. Approximately 14% of patients had the primary outcome of diagnosed infection. The area under the ROC curve (AUC) for the vitals model, which used only vital signs and demographic data, was 0.67 for the training data set, 0.67 for the validation data set, and 0.67 (95% CI 0.65–0.69) for the test data set. The AUC for the chief complaint model which also included demographic and vital sign data was 0.84 for the training data set, 0.83 for the validation data set, and 0.83 (95% CI 0.81–0.84) for the test data set. The best performing methods made use of all of the free text. In particular, the AUC for the bag-of-words model was 0.89 for training data set, 0.86 for the validation data set, and 0.86 (95% CI 0.85–0.87) for the test data set. The AUC for the topic model was 0.86 for the training data set, 0.86 for the validation data set, and 0.85 (95% CI 0.84–0.86) for the test data set. Conclusion Compared to previous work that only used structured data such as vital signs and demographic information, utilizing free text drastically improves the discriminatory ability (increase in AUC from 0.67 to 0.86) of identifying infection.

Learning a Health Knowledge Graph from Electronic Medical Records

Demand for clinical decision support systems in medicine and self-diagnostic symptom checkers has substantially increased in recent years. Existing platforms rely on knowledge bases manually compiled through a labor-intensive process or automatically derived using simple pairwise statistics. This study explored an automated process to learn high quality knowledge bases linking diseases and symptoms directly from electronic medical records. Medical concepts were extracted from 273,174 de-identified patient records and maximum likelihood estimation of three probabilistic models was used to automatically construct knowledge graphs: logistic regression, naive Bayes classifier and a Bayesian network using noisy OR gates. A graph of disease-symptom relationships was elicited from the learned parameters and the constructed knowledge graphs were evaluated and validated, with permission, against Google’s manually-constructed knowledge graph and against expert physician opinions. Our study shows that direct and automated construction of high quality health knowledge graphs from medical records using rudimentary concept extraction is feasible. The noisy OR model produces a high quality knowledge graph reaching precision of 0.85 for a recall of 0.6 in the clinical evaluation. Noisy OR significantly outperforms all tested models across evaluation frameworks (p < 0.01).

A rules based algorithm to generate problem lists using emergency department medication reconciliation

OBJECTIVES To evaluate the sensitivity and specificity of a problem list automatically generated from the emergency department (ED) medication reconciliation. METHODS We performed a retrospective cohort study of patients admitted via the ED who also had a prior inpatient admission within the past year of an academic tertiary hospital. Our algorithm used the First Databank ontology to group medications into therapeutic classes, and applied a set of clinically derived rules to them to predict obstructive lung disease, hypertension, diabetes, congestive heart failure (CHF), and thromboembolism (TE) risk. This prediction was compared to problem lists in the last discharge summary in the electronic health record (EHR) as well as the emergency attending note. RESULTS A total of 603 patients were enrolled from 03/29/2013-04/30/2013. The algorithm had superior sensitivity for all five conditions versus the attending problem list at the 99% confidence level (Obstructive Lung Disease 0.93 vs 0.47, Hypertension 0.93 vs 0.56, Diabetes 0.97 vs 0.73, TE Risk 0.82 vs 0.36, CHF 0.85 vs 0.38), while the attending problem list had superior specificity for both hypertension (0.76 vs 0.94) and CHF (0.87 vs 0.98). The algorithm had superior sensitivity for all conditions versus the EHR problem list (Obstructive Lung Disease 0.93 vs 0.34, Hypertension 0.93 vs 0.30, Diabetes 0.97 vs 0.67, TE Risk 0.82 vs 0.23, CHF 0.85 vs 0.32), while the EHR problem list also had superior specificity for detecting hypertension (0.76 vs 0.95) and CHF (0.87 vs 0.99). CONCLUSION The algorithm was more sensitive than clinicians for all conditions, but less specific for conditions that are not treated with a specific class of medications. This suggests similar algorithms may help identify critical conditions, and facilitate thorough documentation, but further investigation, potentially adding alternate sources of information, may be needed to reliably detect more complex conditions.

Evaluation of the Angus ICD9-CM Sepsis Abstraction Criteria

Objective Validate the infection component of the Angus International Classification of Diseases, Ninth Revision, Clinical Modification (ICD9-CM) sepsis abstraction criteria Design Observational cohort study Setting 55,000 visits/year Adult Emergency Department (ED) Patients All consecutive ED patient visits between 12/16/2011 and 08/13/2012 were included in the study. Patients were excluded if there was a missing outcome measure. Interventions None. Measurements and Main Results The primary outcome measure was suspected infection at conclusion of the ED work-up as judged by the physician. There were 34,796 patients who presented to the ED between 12/16/11 and 8/13/12, of which 31,755 (91%) patients were included and analyzed. The original Angus sepsis abstraction criteria had a sensitivity of 55%, specificity of 97%, PPV of 82%, NPV of 88%, accuracy of 87%, and a F1 score of 0.66. The modified Angus sepsis abstraction criteria which includes codes added after the original publication had a sensitivity of 65%, specificity of 96%, PPV of 81%, NPV of 91%, accuracy of 89%, and F1 score of 0.72. Conclusions In our study, the Angus abstraction criteria have high specificity (97%), but moderate sensitivity (55%) in identifying patients with suspected infection as defined by physician at the time of disposition from the emergency department. Given these findings, it is likely that we are underestimating the true incidence of sepsis in the United States and worldwide.

Contextual Autocomplete: A Novel User Interface Using Machine Learning to Improve Ontology Usage and Structured Data Capture for Presenting Problems in the Emergency Department

Objective To determine the effect of contextual autocomplete, a user interface that uses machine learning, on the efficiency and quality of documentation of presenting problems (chief complaints) in the emergency department (ED). Materials and Methods We used contextual autocomplete, a user interface that ranks concepts by their predicted probability, to help nurses enter data about a patient’s reason for visiting the ED. Predicted probabilities were calculated using a previously derived model based on triage vital signs and a brief free text note. We evaluated the percentage and quality of structured data captured using a prospective before-and-after study design. Results A total of 279,231 patient encounters were analyzed. Structured data capture improved from 26.2% to 97.2% (p<0.0001). During the post-implementation period, presenting problems were more complete (3.35 vs 3.66; p=0.0004), as precise (3.59 vs. 3.74; p=0.1), and higher in overall quality (3.38 vs. 3.72; p=0.0002). Our system reduced the mean number of keystrokes required to document a presenting problem from 11.6 to 0.6 (p<0.0001), a 95% improvement. Discussion We have demonstrated a technique that captures structured data on nearly all patients. We estimate that our system reduces the number of man-hours required annually to type presenting problems at our institution from 92.5 hours to 4.8 hours. Conclusion Implementation of a contextual autocomplete system resulted in improved structured data capture, ontology usage compliance, and data quality.

Consensus Development Of A Modern Ontology Of Emergency Department Presenting Problems: The HierArchical Presenting Problem OntologY (HaPPy)

Objective Numerous attempts have been made to create a standardized ‘presenting problem’ or ‘chief complaint’ list to characterize the nature of an Emergency Department visit. Previous attempts have failed to gain widespread adoption as none were freely sharable and contained the right level of specificity, structure, and clinical relevance to gain acceptance by the larger emergency medicine community. Using real-world data, we constructed a presenting problem list that addresses these challenges. Materials and Methods We prospectively captured the presenting problems for 112,612 consecutive emergency department patient encounters at an urban, academic, Level I trauma center. No patients were excluded. We used a modified Delphi consensus process to iteratively derive our system using real-world data. We used the first 95% of encounters to derive our ontology; the remaining 5% for validation. All concepts were mapped to SNOMED-CT. Results Our system consists of a polyhierarchical ontology containing 690 unique concepts, 2,113 synonyms, and 30,605 non-visible descriptions to correct misspellings and non-standard terminology. Our ontology successfully captured structured data for 95.8% of visits in our validation dataset. Discussion and Conclusion We present the HierArchical Presenting Problem ontologY (HaPPy). This ontology was empirically derived then iteratively validated by an expert consensus panel. HaPPy contains 690 presenting problem concepts, each concept being mapped to SNOMED-CT. This freely sharable ontology should help to facilitate presenting problem based quality metrics, research, and patient care.

Derivation and Validation of a Record Linkage Algorithm between EMS and the Emergency Department

Background Linking EMS electronic patient care reports (ePCRs) to ED records can provide clinicians access to vital information that can alter management. It can also create rich databases for research and quality improvement. Unfortunately, previous attempts at ePCR - ED record linkage have had limited success. Objective To derive and validate an automated record linkage algorithm between EMS ePCR’s and ED records using supervised machine learning. Methods All consecutive ePCR’s from a single EMS provider between June 2013 and June 2015 were included. A primary reviewer matched ePCR’s to a list of ED patients to create a gold standard. Age, gender, last name, first name, social security number (SSN), and date of birth (DOB) were extracted. Data was randomly split into 80%/20% training and test data sets. We derived missing indicators, identical indicators, edit distances, and percent differences. A multivariate logistic regression model was trained using 5k fold cross-validation, using label k-fold, L2 regularization, and class re-weighting. Results A total of 14,032 ePCRs were included in the study. Inter-rater reliability between the primary and secondary reviewer had a Kappa of 0.9. The algorithm had a sensitivity of 99.4%, a PPV of 99.9% and AUC of 0.99 in both the training and test sets. DOB match had the highest odd ratio of 16.9, followed by last name match (10.6). SSN match had an odds ratio of 3.8. Conclusions We were able to successfully derive and validate a probabilistic record linkage algorithm from a single EMS ePCR provider to our hospital EMR.