Lauren McCullagh

Efficacy of an Evidence-Based Clinical Decision Support in Primary Care Practices: A Randomized Clinical Trial

IMPORTANCE There is consensus that incorporating clinical decision support into electronic health records will improve quality of care, contain costs, and reduce overtreatment, but this potential has yet to be demonstrated in clinical trials. OBJECTIVE To assess the influence of a customized evidence-based clinical decision support tool on the management of respiratory tract infections and on the effectiveness of integrating evidence at the point of care. DESIGN, SETTING, AND PARTICIPANTS In a randomized clinical trial, we implemented 2 well-validated integrated clinical prediction rules, namely, the Walsh rule for streptococcal pharyngitis and the Heckerling rule for pneumonia. INTERVENTIONS AND MAIN OUTCOMES AND MEASURES: The intervention group had access to the integrated clinical prediction rule tool and chose whether to complete risk score calculators, order medications, and generate progress notes to assist with complex decision making at the point of care. RESULTS The intervention group completed the integrated clinical prediction rule tool in 57.5% of visits. Providers in the intervention group were significantly less likely to order antibiotics than the control group (age-adjusted relative risk, 0.74; 95% CI, 0.60-0.92). The absolute risk of the intervention was 9.2%, and the number needed to treat was 10.8. The intervention group was significantly less likely to order rapid streptococcal tests compared with the control group (relative risk, 0.75; 95% CI, 0.58-0.97; P= .03). CONCLUSIONS AND RELEVANCE The integrated clinical prediction rule process for integrating complex evidence-based clinical decision report tools is of relevant importance for national initiatives, such as Meaningful Use. TRIAL REGISTRATION clinicaltrials.gov Identifier: NCT01386047.

External validation of the risk assessment model of the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) for medical patients in a tertiary health system.

Background Hospitalized medical patients are at risk for venous thromboembolism (VTE). Universal application of pharmacological thromboprophylaxis has the potential to place a large number of patients at increased bleeding risk. In this study, we aimed to externally validate the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) VTE risk assessment model in a hospitalized general medical population. Methods and Results We identified medical discharges that met the IMPROVE protocol. Cases were defined as hospital‐acquired VTE and confirmed by diagnostic study within 90 days of index hospitalization; matched controls were also identified. Risk factors for VTE were based on the IMPROVE risk assessment model (aged >60 years, prior VTE, intensive care unit or coronary care unit stay, lower limb paralysis, immobility, known thrombophilia, and cancer) and were measured and assessed. A total of 19 217 patients met the inclusion criteria. The overall VTE event rate was 0.7%. The IMPROVE risk assessment model identified 2 groups of the cohort by VTE incidence rate: The low‐risk group had a VTE event rate of 0.42 (95% CI 0.31 to 0.53), corresponding to a score of 0 to 2, and the at‐risk group had a VTE event rate of 1.29 (95% CI 1.01 to 1.57), corresponding to a score of ≥3. Low‐risk status for VTE encompassed 68% of the patient cohort. The area under the receiver operating characteristic curve was 0.702, which was in line with the derivation cohort findings. Conclusions The IMPROVE VTE risk assessment model validation cohort revealed good discrimination and calibration for both the overall VTE risk model and the identification of low‐risk and at‐risk medical patient groups, using a risk score of ≥3. More than two thirds of the entire cohort had a score ≤2.

Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary of findings tables with a new format

OBJECTIVES The current format of summary of findings (SoFs) tables for presenting effect estimates and associated quality of evidence improve understanding and assist users finding key information in systematic reviews. Users of SoF tables have demanded alternative formats to express findings from systematic reviews. STUDY DESIGN AND SETTING We conducted a randomized controlled trial among systematic review users to compare the relative merits of a new format with the current formats of SoF tables regarding understanding, accessibility of information, satisfaction, and preference. Our primary goal was to show that the new format is not inferior to the current format. RESULTS Of 390 potentially eligible subjects, 290 were randomized. Of seven items testing understanding, three showed similar results, two showed small differences favoring the new format, and two (understanding risk difference and quality of the evidence associated with a treatment effect) showed large differences favoring the new format [63% (95% confidence interval {CI}: 55, 71) and 62% (95% CI: 52, 71) more correct answers, respectively]. Respondents rated information in the alternative format as more accessible overall and preferred the new format over the current format. CONCLUSIONS While providing at least similar levels of understanding for some items and increased understanding for others, users prefer the new format of SoF tables.

Usability Testing of a Complex Clinical Decision Support Tool in the Emergency Department: Lessons Learned

Background As the electronic health record (EHR) becomes the preferred documentation tool across medical practices, health care organizations are pushing for clinical decision support systems (CDSS) to help bring clinical decision support (CDS) tools to the forefront of patient-physician interactions. A CDSS is integrated into the EHR and allows physicians to easily utilize CDS tools. However, often CDSS are integrated into the EHR without an initial phase of usability testing, resulting in poor adoption rates. Usability testing is important because it evaluates a CDSS by testing it on actual users. This paper outlines the usability phase of a study, which will test the impact of integration of the Wells CDSS for pulmonary embolism (PE) diagnosis into a large urban emergency department, where workflow is often chaotic and high stakes decisions are frequently made. We hypothesize that conducting usability testing prior to integration of the Wells score into an emergency room EHR will result in increased adoption rates by physicians. Objective The objective of the study was to conduct usability testing for the integration of the Wells clinical prediction rule into a tertiary care center’s emergency department EHR. Methods We conducted usability testing of a CDS tool in the emergency department EHR. The CDS tool consisted of the Wells rule for PE in the form of a calculator and was triggered off computed tomography (CT) orders or patients’ chief complaint. The study was conducted at a tertiary hospital in Queens, New York. There were seven residents that were recruited and participated in two phases of usability testing. The usability testing employed a “think aloud” method and “near-live” clinical simulation, where care providers interacted with standardized patients enacting a clinical scenario. Both phases were audiotaped, video-taped, and had screen-capture software activated for onscreen recordings. Results Phase I: Data from the “think-aloud” phase of the study showed an overall positive outlook on the Wells tool in assessing a patient for a PE diagnosis. Subjects described the tool as “well-organized” and “better than clinical judgment”. Changes were made to improve tool placement into the EHR to make it optimal for decision-making, auto-populating boxes, and minimizing click fatigue. Phase II: After incorporating the changes noted in Phase 1, the participants noted tool improvements. There was less toggling between screens, they had all the clinical information required to complete the tool, and were able to complete the patient visit efficiently. However, an optimal location for triggering the tool remained controversial. Conclusions This study successfully combined “think-aloud” protocol analysis with “near-live” clinical simulations in a usability evaluation of a CDS tool that will be implemented into the emergency room environment. Both methods proved useful in the assessment of the CDS tool and allowed us to refine tool usability and workflow.

External validation of the IMPROVE Bleeding Risk Assessment Model in medical patients.

The IMPROVE Bleed Risk Assessment Model (RAM) remains the only bleed RAM in hospitalised medical patients using 11 clinical and laboratory factors. The aim of our study was to externally validate the IMPROVE Bleed RAM. A retrospective chart review was conducted between October 1, 2012 and July 31, 2014. We applied the point scoring system to compute risk scores for each patient in the validation sample. We then dichotomised the patients into those with a score <7 (low risk) vs ≥ 7 (high risk), as outlined in the original study, and compared the rates of any bleed, non-major bleed, and major bleed. Among the 12,082 subjects, there was an overall 2.6 % rate of any bleed within 14 days of admission. There was a 2.12 % rate of any bleed in those patients with a score of < 7 and a 4.68 % rate in those with a score ≥ 7 [Odds Ratio (OR) 2.3 (95 % CI=1.8-2.9), p<0.0001]. MB rates were 1.5 % in the patients with a score of < 7 and 3.2 % in the patients with a score of ≥ 7, [OR 2.2 (95 % CI=1.6-2.9), p<0.0001]. The ROC curve was 0.63 for the validation sample. This study represents the largest externally validated Bleed RAM in a hospitalised medically ill patient population. A cut-off point score of 7 or above was able to identify a high-risk patient group for MB and any bleed. The IMPROVE Bleed RAM has the potential to allow for more tailored approaches to thromboprophylaxis in medically ill hospitalised patients.

A Framework for Usable and Effective Clinical Decision Support: Experience from the iCPR Randomized Clinical Trial.

Introduction: The promise of Clinical Decision Support (CDS) has always been to transform patient care and improve patient outcomes through the delivery of timely and appropriate recommendations that are patient specific and, more often than not, are appropriately actionable. However, the users of CDS—providers—are frequently bombarded with inappropriate and inapplicable CDS that often are not informational, not integrated into the workflow, not patient specific, and that may present out of date and irrelevant recommendations. Methods: The integrated clinical prediction rule (iCPR) project was a randomized clinical trial (RCT) conducted to determine if a novel form of CDS, i.e., clinical prediction rules (CPRs), could be efficiently integrated into workflow and result in changes in outcomes (e.g., antibiotic ordering) when embedded within a commercial electronic health record (EHR). We use the lessons learned from the iCPR project to illustrate a framework for constructing usable, useful, and effective actionable CDS while employing off-the-shelf functionality in a production system. Innovations that make up the framework combine the following: (1) active and actionable decision support, (2) multiple rounds of usability testing with iterative development for user acceptance, (3) numerous context sensitive triggers, (4) dedicated training and support for users of the CDS tool for user adoption, and (5) support from clinical and administrative leadership. We define “context sensitive triggers” as being workflow events (i.e., context) that result in a CDS intervention. Discussion: Success of the framework can be measured by CDS adoption (i.e., intervention is being used), acceptance (compliance with recommendations), and clinical outcomes (where appropriate). This framework may have broader implications for the deployment of Health Information Technology (HIT). Results and Conclusion: iCPR was well adopted(57.4% of users) and accepted (42.7% of users). Usability testing identified and fixed many issues before the iCPR RCT. The level of leadership support and clinical guidance for iCPR was key in establishing a culture of acceptance for both the tool and its recommendations contributing to adoption and acceptance. The dedicated training and support lead to the majority of the residents reporting a high level of comfort with both iCPR tools strep pharyngitis (64.4 percent) and pneumonia (62.7 percent) as well as a high likelihood of using the tools in the future. A surprising framework addition resulted from usability testing: context sensitive triggers.

Design and implementation of electronic health record integrated clinical prediction rules (iCPR): a randomized trial in diverse primary care settings

BackgroundClinical prediction rules (CPRs) represent a method of determining individual patient risk to help providers make more accurate decisions at the point of care. Well-validated CPRs are underutilized but may decrease antibiotic overuse for acute respiratory infections. The integrated clinical prediction rules (iCPR) study builds on a previous single clinic study to integrate two CPRs into the electronic health record and assess their impact on practice. This article discusses study design and implementation of a multicenter cluster randomized control trial of the iCPR clinical decision support system, including the tool adaptation, usability testing, staff training, and implementation study to disseminate iCPR at multiple clinical sites across two health care systems.MethodsThe iCPR tool is based on two well-validated CPRs, one for strep pharyngitis and one for pneumonia. The iCPR tool uses the reason for visit to trigger a risk calculator. Provider completion of the risk calculator provides a risk score, which is linked to an order set. Order sets guide evidence-based care and include progress note documentation, tests, prescription medications, and patient instructions. The iCPR tool was refined based on interviews with providers, medical assistants, and clinic managers, and two rounds of usability testing. “Near live” usability testing with simulated patients was used to ensure that iCPR fit into providers’ clinical workflows. Thirty-three Family Medicine and General Internal Medicine primary care clinics were recruited at two institutions. Clinics were randomized to academic detailing about strep pharyngitis and pneumonia diagnosis and treatment (control) or academic detailing plus use of the iCPR tool (intervention). The primary outcome is the difference in antibiotic prescribing rates between the intervention and control groups with secondary outcomes of difference in rapid strep and chest x-ray ordering. Use of the components of the iCPR will also be assessed.DiscussionThe iCPR study uses a strong user-centered design and builds on the previous initial study, to assess whether CPRs integrated in the electronic health record can change provider behavior and improve evidence-based care in a broad range of primary care clinics.Trial registrationClinicaltrials.gov (NCT02534987)

Struggling to bring clinical prediction rules to the point of care: missed opportunities to impact patient care

Clinical prediction rules can help clinicians make personalized and complex decisions at the point of care. They have the potential to improve patient care outcomes and reduce healthcare costs, but remain underutilized. There are a number of well-derived and validated clinical prediction rules. Few, however, have been studied by means of an impact analysis or successfully integrated into provider workflow. A heavily identified area of opportunity for integration is the electronic health record. There are, however, a number of barriers to adoption at both the infrastructure and organizational levels. Research efforts should focus on impact analysis and how to successfully implement existing, well-validated clinical prediction rules into daily practice. Recommendations include emphasis on a collaborative framework, using existing technologies, and utilization of usability and workflow integration methodology.

Formative assessment and design of a complex clinical decision support tool for pulmonary embolism

Electronic health record (EHR)-based clinical decision support (CDS) tools are rolled out with the urgency to meet federal requirements without time for usability testing and refinement of the user interface. As part of a larger project to design, develop and integrate a pulmonary embolism CDS tool for emergency physicians, we conducted a formative assessment to determine providers’ level of interest and input on designs and content. This was a study to conduct a formative assessment of emergency medicine (EM) physicians that included focus groups and key informant interviews. The focus of this study was twofold, to determine the general attitude towards CDS tool integration and the ideal integration point into the clinical workflow. To accomplish this, we first approached EM physicians in a focus group, then, during key informant interviews, we presented workflow designs and gave a scenario to help the providers visualise how the CDS tool works. Participants were asked questions regarding the trigger location, trigger words, integration into their workflow, perceived utility and heuristic of the tool. Results from the participants’ survey responses to trigger location, perceived utility and efficiency, indicated that the providers felt the tool would be more of a hindrance than an aid. However, some providers commented that they had not had exposure to CDS tools but had used online calculators, and thought the tools would be helpful at the point-of-care if integrated into the EHR. Furthermore, there was a preference for an order entry wireframe. This study highlights several factors to consider when designing CDS tools: (1) formative assessment of EHR functionality and clinical environment workflow, (2) focus groups and key informative interviews to incorporate providers’ perceptions of CDS and workflow integration and/or (3) the demonstration of proposed workflows through wireframes to help providers visualise design concepts.

Healthcare provider perceptions of clinical prediction rules.

Objectives To examine internal medicine and emergency medicine healthcare provider perceptions of usefulness of specific clinical prediction rules. Setting The study took place in two academic medical centres. A web-based survey was distributed and completed by participants between 1 January and 31 May 2013. Participants Medical doctors, doctors of osteopathy or nurse practitioners employed in the internal medicine or emergency medicine departments at either institution. Primary and secondary outcome measures The primary outcome was to identify the clinical prediction rules perceived as most useful by healthcare providers specialising in internal medicine and emergency medicine. Secondary outcomes included comparing usefulness scores of specific clinical prediction rules based on provider specialty, and evaluating associations between usefulness scores and perceived characteristics of these clinical prediction rules. Results Of the 401 healthcare providers asked to participate, a total of 263 (66%), completed the survey. The CHADS2 score was chosen by most internal medicine providers (72%), and Pulmonary Embolism Rule-Out Criteria (PERC) score by most emergency medicine providers (45%), as one of the top three most useful from a list of 24 clinical prediction rules. Emergency medicine providers rated their top three significantly more positively, compared with internal medicine providers, as having a better fit into their workflow (p=0.004), helping more with decision-making (p=0.037), better fitting into their thought process when diagnosing patients (p=0.001) and overall, on a 10-point scale, more useful (p=0.009). For all providers, the perceived qualities of useful at point of care, helps with decision making, saves time diagnosing, fits into thought process, and should be the standard of clinical care correlated highly (≥0.65) with overall 10-point usefulness scores. Conclusions Healthcare providers describe clear preferences for certain clinical prediction rules, based on medical specialty.