Angela Ai

Analysis of clinical decision support system malfunctions: a case series and survey

Objective To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. Materials and Methods We identified and investigated several CDSS malfunctions at Brigham and Women’s Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. Results We identified four CDSS malfunctions at Brigham and Women’s Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually. Discussion CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate. Conclusion CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

Variation in high-priority drug-drug interaction alerts across institutions and electronic health records.

Objective: The United States Office of the National Coordinator for Health Information Technology sponsored the development of a “high-priority” list of drug-drug interactions (DDIs) to be used for clinical decision support. We assessed current adoption of this list and current alerting practice for these DDIs with regard to alert implementation (presence or absence of an alert) and display (alert appearance as interruptive or passive). Materials and methods: We conducted evaluations of electronic health records (EHRs) at a convenience sample of health care organizations across the United States using a standardized testing protocol with simulated orders. Results: Evaluations of 19 systems were conducted at 13 sites using 14 different EHRs. Across systems, 69% of the high-priority DDI pairs produced alerts. Implementation and display of the DDI alerts tested varied between systems, even when the same EHR vendor was used. Across the drug pairs evaluated, implementation and display of DDI alerts differed, ranging from 27% (4/15) to 93% (14/15) implementation. Discussion: Currently, there is no standard of care covering which DDI alerts to implement or how to display them to providers. Opportunities to improve DDI alerting include using differential displays based on DDI severity, establishing improved lists of clinically significant DDIs, and thoroughly reviewing organizational implementation decisions regarding DDIs. Conclusion: DDI alerting is clinically important but not standardized. There is significant room for improvement and standardization around evidence-based DDIs.

Clinical decision support alert malfunctions: analysis and empirically derived taxonomy

Abstract Objective To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

Communication failure: analysis of prescribers’ use of an internal free-text field on electronic prescriptions

Importance Electronic prescribing promises to improve the safety and clarity of prescriptions. However, it also can introduce miscommunication between prescribers and pharmacists. There are situations where information that is meant to be sent to pharmacists is not sent to them, which has the potential for dangerous errors. Objective To examine how frequently prescribers or administrative personnel put information intended for pharmacists in a field not sent to pharmacists, classify the type of information included, and assess the potential harm associated with these missed messages. Design, Setting, Participants Medication record data from our legacy electronic health record were requested for ambulatory care patients seen at an academic medical center from January 1, 2000, to May 31, 2015 (20 123 881 records). From this database, 6 060 272 medication orders met our inclusion criteria. We analyzed a random sample of 10 000 medication orders with internal comments. Main Outcomes and Measures Reviewers classified internal comments for intent. Comments intended for pharmacists were also sorted into descriptive categories and analyzed for the potential for patient harm. Results We found that 11.7% of the prescriptions in our sample contained comments that were intended to be sent to pharmacists. Many comments contained information about the dose, route, or duration of the prescription (38.0%). Approximately a third of the comments intended for pharmacists contained information that had the potential for significant or severe harm if not communicated. Conclusion We found undelivered comments that were clearly intended for pharmacists and contained important information for either pharmacists or patients. This poses a legitimate safety concern, as a portion of comments contained information that could have prevented severe or significant harm.

A Picture is Worth 1,000 Words

OBJECTIVE To understand how clinicians utilize image uploading tools in a home grown electronic health records (EHR) system. METHODS A content analysis of patient notes containing non-radiological images from the EHR was conducted. Images from 4,000 random notes from July 1, 2009 -June 30, 2010 were reviewed and manually coded. Codes were assigned to four properties of the image: (1) image type, (2) role of image uploader (e.g. MD, NP, PA, RN), (3) practice type (e.g. internal medicine, dermatology, ophthalmology), and (4) image subject. RESULTS 3,815 images from image-containing notes stored in the EHR were reviewed and manually coded. Of those images, 32.8% were clinical and 66.2% were non-clinical. The most common types of the clinical images were photographs (38.0%), diagrams (19.1%), and scanned documents (14.4%). MDs uploaded 67.9% of clinical images, followed by RNs with 10.2%, and genetic counselors with 6.8%. Dermatology (34.9%), ophthalmology (16.1%), and general surgery (10.8%) uploaded the most clinical images. The content of clinical images referencing body parts varied, with 49.8% of those images focusing on the head and neck region, 15.3% focusing on the thorax, and 13.8% focusing on the lower extremities. CONCLUSION The diversity of image types, content, and uploaders within a home grown EHR system reflected the versatility and importance of the image uploading tool. Understanding how users utilize image uploading tools in a clinical setting highlights important considerations for designing better EHR tools and the importance of interoperability between EHR systems and other health technology. CITATION AC Ai, FL Maloney, T-T Hickman, AR Wilcox, H Ramelson, A Wright. A picture is worth 1,000 words: The use of clinical images in electronic medical records. Appl Clin Inform 2017; 8: 710-718 https://doi.org/10.4338/ACI-2016-10-RA-0180.

Methods for detecting malfunctions in clinical decision support systems

Clinical decision support systems, when used effectively, can improve the quality of care. However, such systems can malfunction, and these malfunctions can be difficult to detect. In this poster, we describe four methods of detecting and resolving issues with clinical decision support: 1) statistical anomaly detection, 2) visual analytics and dashboards, 3) user feedback analysis, 4) taxonomization of failure modes/effects.