Sameer Malhotra

Accuracy of Electronically Reported “Meaningful Use” Clinical Quality Measures: A Cross-sectional Study

BACKGROUND The federal Electronic Health Record Incentive Program requires electronic reporting of quality from electronic health records, beginning in 2014. Whether electronic reports of quality are accurate is unclear. OBJECTIVE To measure the accuracy of electronic reporting compared with manual review. DESIGN Cross-sectional study. SETTING A federally qualified health center with a commercially available electronic health record. PATIENTS All adult patients eligible in 2008 for 12 quality measures (using 8 unique denominators) were identified electronically. One hundred fifty patients were randomly sampled per denominator, yielding 1154 unique patients. MEASUREMENTS Receipt of recommended care, assessed by both electronic reporting and manual review. Sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, and absolute rates of recommended care were measured. RESULTS Sensitivity of electronic reporting ranged from 46% to 98% per measure. Specificity ranged from 62% to 97%, positive predictive value from 57% to 97%, and negative predictive value from 32% to 99%. Positive likelihood ratios ranged from 2.34 to 24.25 and negative likelihood ratios from 0.02 to 0.61. Differences between electronic reporting and manual review were statistically significant for 3 measures: Electronic reporting underestimated the absolute rate of recommended care for 2 measures (appropriate asthma medication [38% vs. 77%; P < 0.001] and pneumococcal vaccination [27% vs. 48%; P < 0.001]) and overestimated care for 1 measure (cholesterol control in patients with diabetes [57% vs. 37%; P = 0.001]). LIMITATION This study addresses the accuracy of the measure numerator only. CONCLUSION Wide measure-by-measure variation in accuracy threatens the validity of electronic reporting. If variation is not addressed, financial incentives intended to reward high quality may not be given to the highest-quality providers. PRIMARY FUNDING SOURCE Agency for Healthcare Research and Quality.

Physician experiences transitioning between an older versus newer electronic health record for electronic prescribing

PURPOSE Federal incentives to adopt interoperable, certified electronic health records (EHRs) with electronic prescribing (e-prescribing) are motivating providers using older EHRs to transition to newer EHRs. The objective of this study was to describe, from the perspective of experienced EHR users, the transition from an older, locally developed EHR with minimal clinical decision support (CDS) for e-prescribing to a newer, commercial EHR with more robust CDS for e-prescribing. METHODS This qualitative, case study consisted of observations and semi-structured interviews of adult internal medicine faculty members (n=19) at an academic-affiliated ambulatory care clinic from January through November 2009. All providers transitioned from the older, locally developed EHR to the newer, commercial EHR in April 2008. We analyzed field notes of observations and transcripts of semi-structured interviews using qualitative methods guided by a grounded theory approach. RESULTS We identified key themes describing physician experiences. Despite intensive effort by the information systems team to ease the transition, even these experienced e-prescribers found transitioning extremely difficult. The commercial EHR was not perceived as improving medication safety, despite having more robust CDS. Additionally, physicians felt the commercial EHR was too complex, reducing their efficiency. CONCLUSIONS This is among the first studies examining physician experiences transitioning between an older, locally developed EHR to a newer, commercial EHR with more robust CDS for e-prescribing. Understanding physician experiences with this type of transition and their general preferences for prescribing applications may lead to less disruptive system implementations and better designed EHRs that are more readily accepted by providers. In this way, productivity and safety benefits may be maximized while mitigating potential threats associated with transitions. TRIAL REGISTRATION ClinicalTrials.gov, Identifier: NCT00603070.

A long-term follow-up evaluation of electronic health record prescribing safety

OBJECTIVE To be eligible for incentives through the Electronic Health Record (EHR) Incentive Program, many providers using older or locally developed EHRs will be transitioning to new, commercial EHRs. We previously evaluated prescribing errors made by providers in the first year following transition from a locally developed EHR with minimal prescribing clinical decision support (CDS) to a commercial EHR with robust CDS. Following system refinements, we conducted this study to assess the rates and types of errors 2 years after transition and determine the evolution of errors. MATERIALS AND METHODS We conducted a mixed methods cross-sectional case study of 16 physicians at an academic-affiliated ambulatory clinic from April to June 2010. We utilized standardized prescription and chart review to identify errors. Fourteen providers also participated in interviews. RESULTS We analyzed 1905 prescriptions. The overall prescribing error rate was 3.8 per 100 prescriptions (95% CI 2.8 to 5.1). Error rates were significantly lower 2 years after transition (p<0.001 compared to pre-implementation, 12 weeks and 1 year after transition). Rates of near misses remained unchanged. Providers positively appreciated most system refinements, particularly reduced alert firing. DISCUSSION Our study suggests that over time and with system refinements, use of a commercial EHR with advanced CDS can lead to low prescribing error rates, although more serious errors may require targeted interventions to eliminate them. Reducing alert firing frequency appears particularly important. Our results provide support for federal efforts promoting meaningful use of EHRs. CONCLUSIONS Ongoing error monitoring can allow CDS to be optimally tailored and help achieve maximal safety benefits. CLINICAL TRIALS REGISTRATION ClinicalTrials.gov, Identifier: NCT00603070.

Variation in high-priority drug-drug interaction alerts across institutions and electronic health records.

Objective: The United States Office of the National Coordinator for Health Information Technology sponsored the development of a “high-priority” list of drug-drug interactions (DDIs) to be used for clinical decision support. We assessed current adoption of this list and current alerting practice for these DDIs with regard to alert implementation (presence or absence of an alert) and display (alert appearance as interruptive or passive). Materials and methods: We conducted evaluations of electronic health records (EHRs) at a convenience sample of health care organizations across the United States using a standardized testing protocol with simulated orders. Results: Evaluations of 19 systems were conducted at 13 sites using 14 different EHRs. Across systems, 69% of the high-priority DDI pairs produced alerts. Implementation and display of the DDI alerts tested varied between systems, even when the same EHR vendor was used. Across the drug pairs evaluated, implementation and display of DDI alerts differed, ranging from 27% (4/15) to 93% (14/15) implementation. Discussion: Currently, there is no standard of care covering which DDI alerts to implement or how to display them to providers. Opportunities to improve DDI alerting include using differential displays based on DDI severity, establishing improved lists of clinically significant DDIs, and thoroughly reviewing organizational implementation decisions regarding DDIs. Conclusion: DDI alerting is clinically important but not standardized. There is significant room for improvement and standardization around evidence-based DDIs.

Multimodal Monitoring in the Pediatric Intensive Care Unit: New Modalities and Informatics Challenges

We review several newer modalities to monitor the brain in children with acute neurologic disease in the pediatric intensive care unit, such as partial brain tissue oxygen tension (PbtO2), jugular venous oxygen saturation (SjvO2), near infrared spectroscopy (NIRS), thermal diffusion measurement of cerebral blood flow, cerebral microdialysis, and EEG. We then discuss the informatics challenges to acquire, consolidate, analyze, and display the data. Acquisition includes multiple data types: discrete, waveform, and continuous. Consolidation requires device interoperability and time synchronization. Analysis could include pressure reactivity index and quantitative EEG. Displays should communicate the patients current status, longitudinal and trend information, and critical alarms.

JOURNAL CLUB: Predictors of Provider Response to Clinical Decision Support: Lessons Learned From the Medicare Imaging Demonstration

OBJECTIVE The efficacy of imaging clinical decision support (CDS) varies. Our objective was to identify CDS factors contributing to imaging order cancellation or modification. SUBJECTS AND METHODS This pre-post study was performed across four institutions participating in the Medicare Imaging Demonstration. The intervention was CDS at order entry for selected outpatient imaging procedures. On the basis of the information entered, computerized alerts indicated to providers whether orders were not covered by guidelines, appropriate, of uncertain appropriateness, or inappropriate according to professional society guidelines. Ordering providers could override or accept CDS. We considered actionable alerts to be those that could generate an immediate order behavior change in the ordering physician (i.e., cancellation of inappropriate orders or modification of orders of uncertain appropriateness that had a recommended alternative). Chi-square and logistic regression identified predictors of order cancellation or modification after an alert. RESULTS A total of 98,894 radiology orders were entered (83,114 after the intervention). Providers ignored 98.9%, modified 1.1%, and cancelled 0.03% of orders in response to alerts. Actionable alerts had a 10 fold higher rate of modification (8.1% vs 0.7%; p < 0.0001) or cancellation (0.2% vs 0.02%; p < 0.0001) orders compared with nonactionable alerts. Orders from institutions with preexisting imaging CDS had a sevenfold lower rate of cancellation or modification than was seen at sites with newly implemented CDS (1.4% vs 0.2%; p < 0.0001). In multivariate analysis, actionable alerts were 12 times more likely to result in order cancellation or modification. Orders at sites with preexisting CDS were 7.7 times less likely to be cancelled or modified (p < 0.0001). CONCLUSION Using results from the Medicare Imaging Demonstration project, we identified potential factors that were associated with CDS effect on provider imaging ordering; these findings may have implications for future design of such computerized systems.

Medicare Imaging Demonstration: Assessing Attributes of Appropriate Use Criteria and Their Influence on Ordering Behavior

OBJECTIVE Persistent concern exists about the variable and possibly inappropriate utilization of high-cost imaging tests. The purpose of this study is to assess the influence of appropriate use criteria attributes on altering ambulatory imaging orders deemed inappropriate. MATERIALS AND METHODS This secondary analysis included Medicare Imaging Demonstration data collected from three health care systems in 2011-2013 via the use of clinical decision support (CDS) during ambulatory imaging order entry. The CDS system captured whether orders were inappropriate per the appropriate use criteria of professional societies and provided advice during the intervention period. For orders deemed inappropriate, we assessed the impact of the availability of alternative test recommendations, conflicts with local best practices, and the strength of evidence for appropriate use criteria on the primary outcome of cancellation or modification of inappropriate orders. Expert review determined conflicts with local best practices for 250 recommendations for abdominal and thoracic CT orders. Strength of evidence was assessed for the 15 most commonly triggered recommendations that were deemed inappropriate. A chi-square test was used for univariate analysis. RESULTS A total of 1691 of 63,222 imaging test orders (2.7%) were deemed inappropriate during the intervention period; this amount decreased from 364 of 11,675 test orders (3.1%) in the baseline period (p < 0.00001). Of 270 inappropriate recommendations with alternative test recommendations, 28 (10.4%) were modified, compared with four of 1024 inappropriate recommendations without alternatives (0.4%) (p < 0.0001). Seventy-eight of 250 recommendations (31%) conflicted with local best practices, but only six of 69 inappropriate recommendations (9%) conflicted (p < 0.001). No inappropriate recommendations that conflicted with local best practices were modified. All 15 commonly triggered recommendations had an Oxford Centre for Evidence-Based Medicine level of evidence of 5 (i.e., expert opinion). CONCLUSION Orders for imaging tests that were deemed inappropriate were modified infrequently, more often with alternative recommendations present and only for appropriate use criteria consistent with local best practices.