Thomas McGinn

A Framework for Assessing the Value of Diagnostic Imaging in the Era of Comparative Effectiveness Research

In June 2009, the Federal Coordinating Council for Comparative Effectiveness Research submitted a report to the President and Congress in which the Council described the purpose of comparative effectiveness research (CER) as developing evidence-based information for interventions and determining under what circumstances an intervention is effective (1). With the enactment of the Patient Protection and Affordable Care Act, a Patient-centered Outcomes Research Institute (PCORI) was established to assist decision makers in making evidence-based health decisions through synthesis and dissemination of clinical CER of health interventions (2). Its founding has underscored a heightened need for health policy makers to consider the impact of health care technologies on final outcomes of interest--for example, functional status, quality of life, disability, major clinical events, and mortality (3-5).

Efficacy of an Evidence-Based Clinical Decision Support in Primary Care Practices: A Randomized Clinical Trial

IMPORTANCE There is consensus that incorporating clinical decision support into electronic health records will improve quality of care, contain costs, and reduce overtreatment, but this potential has yet to be demonstrated in clinical trials. OBJECTIVE To assess the influence of a customized evidence-based clinical decision support tool on the management of respiratory tract infections and on the effectiveness of integrating evidence at the point of care. DESIGN, SETTING, AND PARTICIPANTS In a randomized clinical trial, we implemented 2 well-validated integrated clinical prediction rules, namely, the Walsh rule for streptococcal pharyngitis and the Heckerling rule for pneumonia. INTERVENTIONS AND MAIN OUTCOMES AND MEASURES: The intervention group had access to the integrated clinical prediction rule tool and chose whether to complete risk score calculators, order medications, and generate progress notes to assist with complex decision making at the point of care. RESULTS The intervention group completed the integrated clinical prediction rule tool in 57.5% of visits. Providers in the intervention group were significantly less likely to order antibiotics than the control group (age-adjusted relative risk, 0.74; 95% CI, 0.60-0.92). The absolute risk of the intervention was 9.2%, and the number needed to treat was 10.8. The intervention group was significantly less likely to order rapid streptococcal tests compared with the control group (relative risk, 0.75; 95% CI, 0.58-0.97; P= .03). CONCLUSIONS AND RELEVANCE The integrated clinical prediction rule process for integrating complex evidence-based clinical decision report tools is of relevant importance for national initiatives, such as Meaningful Use. TRIAL REGISTRATION clinicaltrials.gov Identifier: NCT01386047.

Validation of a Hepatitis C Screening Tool in Primary Care

BACKGROUND Although hepatitis C virus (HCV) has an estimated national prevalence of 1.8%, testing rates are lower than those recommended by guidelines, particularly in primary care. A critical step is the ability to identify patients at increased risk who should be screened. We sought to prospectively derive and validate a clinical predication tool to assist primary care providers in identifying patients who should be tested for HCV antibodies. METHODS A total of 1000 randomly selected patients attending an inner-city primary care clinic filled out a 27-item questionnaire assessing 5 HCV risk factor domains: work, medical, exposure, personal care, and social history. Afterward, the patients underwent HCV antibody testing. Multivariable logistic regression analysis was performed to identify risk factors associated with HCV antibodies. RESULTS There was an 8.3% (95% confidence interval, 6.7%-10.2%) prevalence of HCV antibodies. The patients who were HCV antibody positive were more likely to be male, older, and insured by Medicaid (P < or = .02). Those who had risk factors within the medical, exposure, and social history domains were more likely to be HCV antibody positive. The area under the receiver operating characteristic curve for the screening tool based on these 3 domains was 0.77. With an increasing number of positive domains, there was a higher likelihood of HCV antibody positivity. Only 2% of patients with 0 risk factors had HCV antibodies. CONCLUSIONS A prediction tool can be used to accurately identify patients at high risk of HCV who may benefit from serologic screening. Future studies should assess whether wider use of this tool may lead to improved outcomes.

Implementation of culturally targeted patient navigation system for screening colonoscopy in a direct referral system

Low-income minorities often face system-based and personal barriers to screening colonoscopy (SC). Culturally targeted patient navigation (CTPN) programs employing professional navigators (Pro-PNs) or community-based peer navigators (Peer-PNs) can help overcome barriers but are not widely implemented. In East Harlem, NY, USA, where approximately half the residents participate in SC, 315 African American patients referred for SC at a primary care clinic with a Direct Endoscopic Referral System were recruited between May 2008 and May 2010. After medical clearance, 240 were randomized to receive CTPN delivered by a Pro-PN (n = 106) or Peer-PN (n = 134). Successful navigation was measured by SC adherence rate, patient satisfaction and navigator trust. Study enrollment was 91.4% with no significant differences in SC adherence rates between Pro-PN (80.0%) and Peer-PN (71.3%) (P = 0.178). Participants in both groups reported high levels of satisfaction and trust. These findings suggest that CTPN Pro-PN and Peer-PN programs are effective in this urban primary care setting. We detail how we recruited and trained navigators, how CTPN was implemented and provide a preliminary answer to our questions of the study aims: can peer navigators be as effective as professionals and what is the potential impact of patient navigation on screening adherence?

Validation of a Model to Predict Perioperative Mortality from Lung Cancer Resection in the Elderly

RATIONALE Surgical resection is the mainstay therapy for localized non-small cell lung cancer (NSCLC), yet elderly patients are less likely to be treated due to concerns about morbidity and mortality related to surgery. OBJECTIVES To validate and refine a clinical model to predict 30-day perioperative mortality (POM) in elderly patients undergoing curative resection for lung cancer. METHODS We identified 14,297 patients aged 65 years and older with stage I, II, or IIIA NCSLC from the Surveillance, Epidemiology, and End-Results Registry linked to Medicare claims. We used logistic regression analysis to identify independent risk factors for POM and to validate and refine a previously derived prediction model. MEASUREMENTS AND MAIN RESULTS Overall, POM was 4.6% (95% confidence interval, 4.2-4.9%). Multiple regression analysis revealed that greater age, male sex, resections of multiple lobes, advanced stage, greater tumor size, and certain comorbidities were associated with increased risk for POM. These risk factors were similar to those observed in the prior model. When patients were stratified according to their predicted risk of POM, the observed mortality increased from 1.2 to more than 10%. CONCLUSIONS Among elderly patients with lung cancer, a prediction rule can identify those patients at higher risk for fatal complications from surgery. Further studies should evaluate whether use of the model can lead to improvements in treatment decision making.

External validation of the risk assessment model of the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) for medical patients in a tertiary health system.

Background Hospitalized medical patients are at risk for venous thromboembolism (VTE). Universal application of pharmacological thromboprophylaxis has the potential to place a large number of patients at increased bleeding risk. In this study, we aimed to externally validate the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) VTE risk assessment model in a hospitalized general medical population. Methods and Results We identified medical discharges that met the IMPROVE protocol. Cases were defined as hospital‐acquired VTE and confirmed by diagnostic study within 90 days of index hospitalization; matched controls were also identified. Risk factors for VTE were based on the IMPROVE risk assessment model (aged >60 years, prior VTE, intensive care unit or coronary care unit stay, lower limb paralysis, immobility, known thrombophilia, and cancer) and were measured and assessed. A total of 19 217 patients met the inclusion criteria. The overall VTE event rate was 0.7%. The IMPROVE risk assessment model identified 2 groups of the cohort by VTE incidence rate: The low‐risk group had a VTE event rate of 0.42 (95% CI 0.31 to 0.53), corresponding to a score of 0 to 2, and the at‐risk group had a VTE event rate of 1.29 (95% CI 1.01 to 1.57), corresponding to a score of ≥3. Low‐risk status for VTE encompassed 68% of the patient cohort. The area under the receiver operating characteristic curve was 0.702, which was in line with the derivation cohort findings. Conclusions The IMPROVE VTE risk assessment model validation cohort revealed good discrimination and calibration for both the overall VTE risk model and the identification of low‐risk and at‐risk medical patient groups, using a risk score of ≥3. More than two thirds of the entire cohort had a score ≤2.

Usability Testing of a Complex Clinical Decision Support Tool in the Emergency Department: Lessons Learned

Background As the electronic health record (EHR) becomes the preferred documentation tool across medical practices, health care organizations are pushing for clinical decision support systems (CDSS) to help bring clinical decision support (CDS) tools to the forefront of patient-physician interactions. A CDSS is integrated into the EHR and allows physicians to easily utilize CDS tools. However, often CDSS are integrated into the EHR without an initial phase of usability testing, resulting in poor adoption rates. Usability testing is important because it evaluates a CDSS by testing it on actual users. This paper outlines the usability phase of a study, which will test the impact of integration of the Wells CDSS for pulmonary embolism (PE) diagnosis into a large urban emergency department, where workflow is often chaotic and high stakes decisions are frequently made. We hypothesize that conducting usability testing prior to integration of the Wells score into an emergency room EHR will result in increased adoption rates by physicians. Objective The objective of the study was to conduct usability testing for the integration of the Wells clinical prediction rule into a tertiary care center’s emergency department EHR. Methods We conducted usability testing of a CDS tool in the emergency department EHR. The CDS tool consisted of the Wells rule for PE in the form of a calculator and was triggered off computed tomography (CT) orders or patients’ chief complaint. The study was conducted at a tertiary hospital in Queens, New York. There were seven residents that were recruited and participated in two phases of usability testing. The usability testing employed a “think aloud” method and “near-live” clinical simulation, where care providers interacted with standardized patients enacting a clinical scenario. Both phases were audiotaped, video-taped, and had screen-capture software activated for onscreen recordings. Results Phase I: Data from the “think-aloud” phase of the study showed an overall positive outlook on the Wells tool in assessing a patient for a PE diagnosis. Subjects described the tool as “well-organized” and “better than clinical judgment”. Changes were made to improve tool placement into the EHR to make it optimal for decision-making, auto-populating boxes, and minimizing click fatigue. Phase II: After incorporating the changes noted in Phase 1, the participants noted tool improvements. There was less toggling between screens, they had all the clinical information required to complete the tool, and were able to complete the patient visit efficiently. However, an optimal location for triggering the tool remained controversial. Conclusions This study successfully combined “think-aloud” protocol analysis with “near-live” clinical simulations in a usability evaluation of a CDS tool that will be implemented into the emergency room environment. Both methods proved useful in the assessment of the CDS tool and allowed us to refine tool usability and workflow.

External validation of the IMPROVE Bleeding Risk Assessment Model in medical patients.

The IMPROVE Bleed Risk Assessment Model (RAM) remains the only bleed RAM in hospitalised medical patients using 11 clinical and laboratory factors. The aim of our study was to externally validate the IMPROVE Bleed RAM. A retrospective chart review was conducted between October 1, 2012 and July 31, 2014. We applied the point scoring system to compute risk scores for each patient in the validation sample. We then dichotomised the patients into those with a score <7 (low risk) vs ≥ 7 (high risk), as outlined in the original study, and compared the rates of any bleed, non-major bleed, and major bleed. Among the 12,082 subjects, there was an overall 2.6 % rate of any bleed within 14 days of admission. There was a 2.12 % rate of any bleed in those patients with a score of < 7 and a 4.68 % rate in those with a score ≥ 7 [Odds Ratio (OR) 2.3 (95 % CI=1.8-2.9), p<0.0001]. MB rates were 1.5 % in the patients with a score of < 7 and 3.2 % in the patients with a score of ≥ 7, [OR 2.2 (95 % CI=1.6-2.9), p<0.0001]. The ROC curve was 0.63 for the validation sample. This study represents the largest externally validated Bleed RAM in a hospitalised medically ill patient population. A cut-off point score of 7 or above was able to identify a high-risk patient group for MB and any bleed. The IMPROVE Bleed RAM has the potential to allow for more tailored approaches to thromboprophylaxis in medically ill hospitalised patients.

A Framework for Usable and Effective Clinical Decision Support: Experience from the iCPR Randomized Clinical Trial.

Introduction: The promise of Clinical Decision Support (CDS) has always been to transform patient care and improve patient outcomes through the delivery of timely and appropriate recommendations that are patient specific and, more often than not, are appropriately actionable. However, the users of CDS—providers—are frequently bombarded with inappropriate and inapplicable CDS that often are not informational, not integrated into the workflow, not patient specific, and that may present out of date and irrelevant recommendations. Methods: The integrated clinical prediction rule (iCPR) project was a randomized clinical trial (RCT) conducted to determine if a novel form of CDS, i.e., clinical prediction rules (CPRs), could be efficiently integrated into workflow and result in changes in outcomes (e.g., antibiotic ordering) when embedded within a commercial electronic health record (EHR). We use the lessons learned from the iCPR project to illustrate a framework for constructing usable, useful, and effective actionable CDS while employing off-the-shelf functionality in a production system. Innovations that make up the framework combine the following: (1) active and actionable decision support, (2) multiple rounds of usability testing with iterative development for user acceptance, (3) numerous context sensitive triggers, (4) dedicated training and support for users of the CDS tool for user adoption, and (5) support from clinical and administrative leadership. We define “context sensitive triggers” as being workflow events (i.e., context) that result in a CDS intervention. Discussion: Success of the framework can be measured by CDS adoption (i.e., intervention is being used), acceptance (compliance with recommendations), and clinical outcomes (where appropriate). This framework may have broader implications for the deployment of Health Information Technology (HIT). Results and Conclusion: iCPR was well adopted(57.4% of users) and accepted (42.7% of users). Usability testing identified and fixed many issues before the iCPR RCT. The level of leadership support and clinical guidance for iCPR was key in establishing a culture of acceptance for both the tool and its recommendations contributing to adoption and acceptance. The dedicated training and support lead to the majority of the residents reporting a high level of comfort with both iCPR tools strep pharyngitis (64.4 percent) and pneumonia (62.7 percent) as well as a high likelihood of using the tools in the future. A surprising framework addition resulted from usability testing: context sensitive triggers.

Hepatitis C treatment completion in individuals with psychiatric comorbidity and depression.

Hepatitis C virus is a common bloodborne pathogen. Patient, provider, and health care system factors combine to constrain access to treatment and have led to low rates of treatment initiation and continuation among medically eligible individuals. Behavioral health comorbidity, which is common in the patient population, has historically been an exclusion criterion and is one such barrier to care. We implemented an interdisciplinary nurse-managed primary care–based hepatitis C evaluation and treatment program to address behavioral health needs concurrently in an effort to increase treatment initiation and continuation rates among comorbid individuals. We found no association between having a psychiatric or substance use history and treatment discontinuation in our patient cohort. Likewise, there was no association in our cohort between becoming depressed or anxious while undergoing treatment and treatment discontinuation. The results of our study concur with others that have shown that addressing behavioral health comorbidities concurrently with hepatitis C evaluation and treatment may improve treatment continuation rates among comorbid patients, thereby helping to remove barriers to treatment of chronic hepatitis C.