Dustin McEvoy

Analysis of clinical decision support system malfunctions: a case series and survey

Objective To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. Materials and Methods We identified and investigated several CDSS malfunctions at Brigham and Women’s Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. Results We identified four CDSS malfunctions at Brigham and Women’s Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually. Discussion CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate. Conclusion CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

Variation in high-priority drug-drug interaction alerts across institutions and electronic health records.

Objective: The United States Office of the National Coordinator for Health Information Technology sponsored the development of a “high-priority” list of drug-drug interactions (DDIs) to be used for clinical decision support. We assessed current adoption of this list and current alerting practice for these DDIs with regard to alert implementation (presence or absence of an alert) and display (alert appearance as interruptive or passive). Materials and methods: We conducted evaluations of electronic health records (EHRs) at a convenience sample of health care organizations across the United States using a standardized testing protocol with simulated orders. Results: Evaluations of 19 systems were conducted at 13 sites using 14 different EHRs. Across systems, 69% of the high-priority DDI pairs produced alerts. Implementation and display of the DDI alerts tested varied between systems, even when the same EHR vendor was used. Across the drug pairs evaluated, implementation and display of DDI alerts differed, ranging from 27% (4/15) to 93% (14/15) implementation. Discussion: Currently, there is no standard of care covering which DDI alerts to implement or how to display them to providers. Opportunities to improve DDI alerting include using differential displays based on DDI severity, establishing improved lists of clinically significant DDIs, and thoroughly reviewing organizational implementation decisions regarding DDIs. Conclusion: DDI alerting is clinically important but not standardized. There is significant room for improvement and standardization around evidence-based DDIs.

Efficiency and Usability of a Near Field Communication-Enabled Tablet for Medication Administration

Background Barcode-based technology coupled with the electronic medication administration record (e-MAR) reduces medication errors and potential adverse drug events (ADEs). However, many current barcode-enabled medication administration (BCMA) systems are difficult to maneuver and often require multiple barcode scans. We developed a prototype, next generation near field communication-enabled medication administration (NFCMA) system using a tablet. Objective We compared the efficiency and usability of the prototype NFCMA system with the traditional BCMA system. Methods We used a mixed-methods design using a randomized observational cross-over study, a survey, and one-on-one interviews to compare the prototype NFCMA system with a traditional BCMA system. The study took place at an academic medical simulation center. Twenty nurses with BCMA experience participated in two simulated patient medication administration scenarios: one using the BCMA system, and the other using the prototype NFCMA system. We collected overall scenario completion time and number of medication scanning attempts per scenario, and compared those using paired t tests. We also collected participant feedback on the prototype NFCMA system using the modified International Business Machines (IBM) Post-Study System Usability Questionnaire (PSSUQ) and a semistructured interview. We performed descriptive statistics on participant characteristics and responses to the IBM PSSUQ. Interview data was analyzed using content analysis with a qualitative description approach to review and categorize feedback from participants. Results Mean total time to complete the scenarios using the NFCMA and the BCMA systems was 202 seconds and 182 seconds, respectively (P=.09). Mean scan attempts with the NFCMA was 7.6 attempts compared with 6.5 attempts with the BCMA system (P=.12). In the usability survey, 95% (19/20) of participants agreed that the prototype NFCMA system was easy to use and easy to learn, with a pleasant interface. Participants expressed interest in using the NFCMA tablet in the hospital; suggestions focused on implementation issues, such as storage of the mobile devices and infection control methods. Conclusions The NFCMA system had similar efficiency to the BCMA system in a simulated scenario. The prototype NFCMA system was well received by nurses and offers promise to improve nurse medication administration efficiency.

Clinical decision support alert malfunctions: analysis and empirically derived taxonomy

Abstract Objective To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

Computerized prescriber order entry–related patient safety reports: analysis of 2522 medication errors

Objective: To examine medication errors potentially related to computerized prescriber order entry (CPOE) and refine a previously published taxonomy to classify them. Materials and Methods: We reviewed all patient safety medication reports that occurred in the medication ordering phase from 6 sites participating in a United States Food and Drug Administration–sponsored project examining CPOE safety. Two pharmacists independently reviewed each report to confirm whether the error occurred in the ordering/prescribing phase and was related to CPOE. For those related to CPOE, we assessed whether CPOE facilitated (actively contributed to) the error or failed to prevent the error (did not directly cause it, but optimal systems could have potentially prevented it). A previously developed taxonomy was iteratively refined to classify the reports. Results: Of 2522 medication error reports, 1308 (51.9%) were related to CPOE. Of these, CPOE facilitated the error in 171 (13.1%) and potentially could have prevented the error in 1137 (86.9%). The most frequent categories of “what happened to the patient” were delays in medication reaching the patient, potentially receiving duplicate drugs, or receiving a higher dose than indicated. The most frequent categories for “what happened in CPOE” included orders not routed to or received at the intended location, wrong dose ordered, and duplicate orders. Variations were seen in the format, categorization, and quality of reports, resulting in error causation being assignable in only 403 instances (31%). Discussion and Conclusion: Errors related to CPOE commonly involved transmission errors, erroneous dosing, and duplicate orders. More standardized safety reporting using a common taxonomy could help health care systems and vendors learn and implement prevention strategies.

Using statistical anomaly detection models to find clinical decision support malfunctions

Objective Malfunctions in Clinical Decision Support (CDS) systems occur due to a multitude of reasons, and often go unnoticed, leading to potentially poor outcomes. Our goal was to identify malfunctions within CDS systems. Methods We evaluated 6 anomaly detection models: (1) Poisson Changepoint Model, (2) Autoregressive Integrated Moving Average (ARIMA) Model, (3) Hierarchical Divisive Changepoint (HDC) Model, (4) Bayesian Changepoint Model, (5) Seasonal Hybrid Extreme Studentized Deviate (SHESD) Model, and (6) E-Divisive with Median (EDM) Model and characterized their ability to find known anomalies. We analyzed 4 CDS alerts with known malfunctions from the Longitudinal Medical Record (LMR) and Epic® (Epic Systems Corporation, Madison, WI, USA) at Brigham and Womens Hospital, Boston, MA. The 4 rules recommend lead testing in children, aspirin therapy in patients with coronary artery disease, pneumococcal vaccination in immunocompromised adults and thyroid testing in patients taking amiodarone. Results Poisson changepoint, ARIMA, HDC, Bayesian changepoint and the SHESD model were able to detect anomalies in an alert for lead screening in children and in an alert for pneumococcal conjugate vaccine in immunocompromised adults. EDM was able to detect anomalies in an alert for monitoring thyroid function in patients on amiodarone. Conclusions Malfunctions/anomalies occur frequently in CDS alert systems. It is important to be able to detect such anomalies promptly. Anomaly detection models are useful tools to aid such detections.

Enhancing problem list documentation in electronic health records using two methods: the example of prior splenectomy

Background Quality improvement professionals often choose between patient-specific interventions, like clinical decision support (CDS), and population-based interventions, like registries or care management. In this paper, we explore the synergy of these two strategies, targeting the problem of procedure documentation for patients with a history of splenectomy. Methods We developed a population health documentation (PHD) intervention and a CDS intervention to improve splenectomy documentation within our electronic health record. Rates of splenectomy documentation were collected before and after the implementation of both interventions to assess their impact on the rate of procedure documentation. Results Both the PHD and CDS interventions led to statistically significant (p<0.001) increases in the baseline rate of splenectomy documentation of 27.4 documentations per month. During the PHD intervention, 444.7 splenectomies were documented per month, while 40.8 splenectomies per month were documented during the CDS intervention. Discussion Both approaches were successful, with the PHD intervention leading to a larger number of incremental procedure documentations, in batches, and the CDS intervention augmenting procedure documentation on an ongoing basis. Our results suggest that population health and CDS strategies complement each other and, where possible, should be used in conjunction. Conclusions PHD and CDS strategies may best be used in conjunction to create a symbiotic relationship in which current problem and procedure documentation gaps are closed using PHD strategies, while new gaps are prevented through ongoing CDS interventions

Changes in hospital bond ratings after the transition to a new electronic health record

Objective To assess the impact of electronic health record (EHR) implementation on hospital finances. Materials and Methods We analyzed the impact of EHR implementation on bond ratings and net income from service to patients (NISP) at 32 hospitals that recently implemented a new EHR and a set of controls. Results After implementing an EHR, 7 hospitals had a bond downgrade, 7 had a bond upgrade, and 18 had no changes. There was no difference in the likelihood of bond rating changes or in changes to NISP following EHR go-live when compared to control hospitals. Discussion Most hospitals in our analysis saw no change in bond ratings following EHR go-live, with no significant differences observed between EHR implementation and control hospitals. There was also no apparent difference in NISP. Conclusions Implementation of an EHR did not appear to have an impact on bond ratings at the hospitals in our analysis.

Methods for detecting malfunctions in clinical decision support systems

Clinical decision support systems, when used effectively, can improve the quality of care. However, such systems can malfunction, and these malfunctions can be difficult to detect. In this poster, we describe four methods of detecting and resolving issues with clinical decision support: 1) statistical anomaly detection, 2) visual analytics and dashboards, 3) user feedback analysis, 4) taxonomization of failure modes/effects.