Thu-Trang T. Hickman

Analysis of clinical decision support system malfunctions: a case series and survey

Objective To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. Materials and Methods We identified and investigated several CDSS malfunctions at Brigham and Women’s Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. Results We identified four CDSS malfunctions at Brigham and Women’s Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually. Discussion CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate. Conclusion CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

Computerised prescribing for safer medication ordering: still a work in progress

Fuelled by compelling evidence that computerised provider order entry (CPOE) improves medication safety and the infusion of tens of billions of federal electronic medical record (EMR) stimulus dollars, electronic medication prescribing in the USA has gone from 70% of prescriptions being written electronically in just the past six years.1–4 Most medications are now ordered electronically both inside and outside the hospital, and they are being sent in electronically to pharmacies. Although many of the initial obstacles to the widespread adoption of CPOE such as physician resistance, lack of standards for electronically transmitting prescriptions to pharmacies and lack of standards for drug databases (leading some organisations to resort to ‘home-grown’ solutions) have been largely overcome, CPOE remains a work in progress. A series of studies by the U.S. Institute of Medicine and Office of the National Coordinator for Health Information Technology (HIT) have recently spotlighted a number of potential safety risks.5–7 To better understand these risks and the opportunities for improvement, particularly as they relate to drug names and drug ordering, the U.S. Food and Drug Administration Center for Drug Evaluation and Researchs Division of Medication Error Prevention and Analysis contracted the Brigham and Womens Hospital (BWH) Center for Patient Safety Research and Practice to study CPOE and risks that could potentially lead to medication errors. The findings from this 2-year investigation have recently been compiled into a White Paper entitled Computerized Prescriber Order Entry Medication Safety (CPOEMS): Uncovering and Learning from Issues and Errors .8 It documents issues about which policy and patient safety leaders, along with clinicians, pharmacists and patients, need to be aware to minimise the risk of CPOE-related errors. Here, we share the main findings and our own perspectives based on this review of 10 CPOE systems (four inpatient, six outpatient) at six …

Variation in high-priority drug-drug interaction alerts across institutions and electronic health records.

Objective: The United States Office of the National Coordinator for Health Information Technology sponsored the development of a “high-priority” list of drug-drug interactions (DDIs) to be used for clinical decision support. We assessed current adoption of this list and current alerting practice for these DDIs with regard to alert implementation (presence or absence of an alert) and display (alert appearance as interruptive or passive). Materials and methods: We conducted evaluations of electronic health records (EHRs) at a convenience sample of health care organizations across the United States using a standardized testing protocol with simulated orders. Results: Evaluations of 19 systems were conducted at 13 sites using 14 different EHRs. Across systems, 69% of the high-priority DDI pairs produced alerts. Implementation and display of the DDI alerts tested varied between systems, even when the same EHR vendor was used. Across the drug pairs evaluated, implementation and display of DDI alerts differed, ranging from 27% (4/15) to 93% (14/15) implementation. Discussion: Currently, there is no standard of care covering which DDI alerts to implement or how to display them to providers. Opportunities to improve DDI alerting include using differential displays based on DDI severity, establishing improved lists of clinically significant DDIs, and thoroughly reviewing organizational implementation decisions regarding DDIs. Conclusion: DDI alerting is clinically important but not standardized. There is significant room for improvement and standardization around evidence-based DDIs.

Clinical decision support alert malfunctions: analysis and empirically derived taxonomy

Abstract Objective To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

Computerized prescriber order entry–related patient safety reports: analysis of 2522 medication errors

Objective: To examine medication errors potentially related to computerized prescriber order entry (CPOE) and refine a previously published taxonomy to classify them. Materials and Methods: We reviewed all patient safety medication reports that occurred in the medication ordering phase from 6 sites participating in a United States Food and Drug Administration–sponsored project examining CPOE safety. Two pharmacists independently reviewed each report to confirm whether the error occurred in the ordering/prescribing phase and was related to CPOE. For those related to CPOE, we assessed whether CPOE facilitated (actively contributed to) the error or failed to prevent the error (did not directly cause it, but optimal systems could have potentially prevented it). A previously developed taxonomy was iteratively refined to classify the reports. Results: Of 2522 medication error reports, 1308 (51.9%) were related to CPOE. Of these, CPOE facilitated the error in 171 (13.1%) and potentially could have prevented the error in 1137 (86.9%). The most frequent categories of “what happened to the patient” were delays in medication reaching the patient, potentially receiving duplicate drugs, or receiving a higher dose than indicated. The most frequent categories for “what happened in CPOE” included orders not routed to or received at the intended location, wrong dose ordered, and duplicate orders. Variations were seen in the format, categorization, and quality of reports, resulting in error causation being assignable in only 403 instances (31%). Discussion and Conclusion: Errors related to CPOE commonly involved transmission errors, erroneous dosing, and duplicate orders. More standardized safety reporting using a common taxonomy could help health care systems and vendors learn and implement prevention strategies.

Quantitative evaluation of changes in gait after extended cerebrospinal fluid drainage for normal pressure hydrocephalus

Idiopathic normal pressure hydrocephalus (iNPH) is characterized by gait instability, urinary incontinence and cognitive dysfunction. These symptoms can be relieved by cerebrospinal fluid (CSF) drainage, but the time course and nature of the improvements are poorly characterized. Attempts to prospectively identify iNPH patients responsive to CSF drainage by evaluating presenting gait quality or via extended lumbar cerebrospinal fluid drainage (eLCD) trials are common, but the reliability of such approaches is unclear. Here we combine eLCD trials with computerized quantitative gait measurements to predict shunt responsiveness in patients undergoing evaluation for possible iNPH. In this prospective cohort study, 50 patients presenting with enlarged cerebral ventricles and gait, urinary, and/or cognitive difficulties were evaluated for iNPH using a computerized gait analysis system during a 3day trial of eLCD. Gait speed, stride length, cadence, and the Timed Up and Go test were quantified before and during eLCD. Qualitative assessments of incontinence and cognition were obtained throughout the eLCD trial. Patients who improved after eLCD underwent ventriculoperitoneal shunt placement, and symptoms were reassessed serially over the next 3 to 15months. There was no significant difference in presenting gait characteristics between patients who improved after drainage and those who did not. Gait improvement was not observed until 2 or more days of continuous drainage in most cases. Symptoms improved after eLCD in 60% of patients, and all patients who improved after eLCD also improved after shunt placement. The degree of improvement after eLCD correlated closely with that observed after shunt placement.

Using statistical anomaly detection models to find clinical decision support malfunctions

Objective Malfunctions in Clinical Decision Support (CDS) systems occur due to a multitude of reasons, and often go unnoticed, leading to potentially poor outcomes. Our goal was to identify malfunctions within CDS systems. Methods We evaluated 6 anomaly detection models: (1) Poisson Changepoint Model, (2) Autoregressive Integrated Moving Average (ARIMA) Model, (3) Hierarchical Divisive Changepoint (HDC) Model, (4) Bayesian Changepoint Model, (5) Seasonal Hybrid Extreme Studentized Deviate (SHESD) Model, and (6) E-Divisive with Median (EDM) Model and characterized their ability to find known anomalies. We analyzed 4 CDS alerts with known malfunctions from the Longitudinal Medical Record (LMR) and Epic® (Epic Systems Corporation, Madison, WI, USA) at Brigham and Womens Hospital, Boston, MA. The 4 rules recommend lead testing in children, aspirin therapy in patients with coronary artery disease, pneumococcal vaccination in immunocompromised adults and thyroid testing in patients taking amiodarone. Results Poisson changepoint, ARIMA, HDC, Bayesian changepoint and the SHESD model were able to detect anomalies in an alert for lead screening in children and in an alert for pneumococcal conjugate vaccine in immunocompromised adults. EDM was able to detect anomalies in an alert for monitoring thyroid function in patients on amiodarone. Conclusions Malfunctions/anomalies occur frequently in CDS alert systems. It is important to be able to detect such anomalies promptly. Anomaly detection models are useful tools to aid such detections.

Outpatient CPOE orders discontinued due to ‘erroneous entry’: prospective survey of prescribers’ explanations for errors

Background Computerised prescriber order entry (CPOE) systems users often discontinue medications because the initial order was erroneous. Objective To elucidate error types by querying prescribers about their reasons for discontinuing outpatient medication orders that they had self-identified as erroneous. Methods During a nearly 3 year retrospective data collection period, we identified 57 972 drugs discontinued with the reason ‘Error (erroneous entry).” Because chart reviews revealed limited information about these errors, we prospectively studied consecutive, discontinued erroneous orders by querying prescribers in near-real-time to learn more about the erroneous orders. Results From January 2014 to April 2014, we prospectively emailed prescribers about outpatient drug orders that they had discontinued due to erroneous initial order entry. Of 2 50 806 medication orders in these 4 months, 1133 (0.45%) of these were discontinued due to error. From these 1133, we emailed 542 unique prescribers to ask about their reason(s) for discontinuing these mediation orders in error. We received 312 responses (58% response rate). We categorised these responses using a previously published taxonomy. The top reasons for these discontinued erroneous orders included: medication ordered for wrong patient (27.8%, n=60); wrong drug ordered (18.5%, n=40); and duplicate order placed (14.4%, n=31). Other common discontinued erroneous orders related to drug dosage and formulation (eg, extended release versus not). Oxycodone (3%) was the most frequent drug discontinued error. Conclusion Drugs are not infrequently discontinued ‘in error.’ Wrong patient and wrong drug errors constitute the leading types of erroneous prescriptions recognised and discontinued by prescribers. Data regarding erroneous medication entries represent an important source of intelligence about how CPOE systems are functioning and malfunctioning, providing important insights regarding areas for designing CPOE more safely in the future.