Rose Gunn

Learning Evaluation: blending quality improvement and implementation research methods to study healthcare innovations

BackgroundIn healthcare change interventions, on-the-ground learning about the implementation process is often lost because of a primary focus on outcome improvements. This paper describes the Learning Evaluation, a methodological approach that blends quality improvement and implementation research methods to study healthcare innovations.MethodsLearning Evaluation is an approach to multi-organization assessment. Qualitative and quantitative data are collected to conduct real-time assessment of implementation processes while also assessing changes in context, facilitating quality improvement using run charts and audit and feedback, and generating transportable lessons. Five principles are the foundation of this approach: (1) gather data to describe changes made by healthcare organizations and how changes are implemented; (2) collect process and outcome data relevant to healthcare organizations and to the research team; (3) assess multi-level contextual factors that affect implementation, process, outcome, and transportability; (4) assist healthcare organizations in using data for continuous quality improvement; and (5) operationalize common measurement strategies to generate transportable results.ResultsLearning Evaluation principles are applied across organizations by the following: (1) establishing a detailed understanding of the baseline implementation plan; (2) identifying target populations and tracking relevant process measures; (3) collecting and analyzing real-time quantitative and qualitative data on important contextual factors; (4) synthesizing data and emerging findings and sharing with stakeholders on an ongoing basis; and (5) harmonizing and fostering learning from process and outcome data. Application to a multi-site program focused on primary care and behavioral health integration shows the feasibility and utility of Learning Evaluation for generating real-time insights into evolving implementation processes.ConclusionsLearning Evaluation generates systematic and rigorous cross-organizational findings about implementing healthcare innovations while also enhancing organizational capacity and accelerating translation of findings by facilitating continuous learning within individual sites. Researchers evaluating change initiatives and healthcare organizations implementing improvement initiatives may benefit from a Learning Evaluation approach.

Start-Up and Ongoing Practice Expenses of Behavioral Health and Primary Care Integration Interventions in the Advancing Care Together (ACT) Program

Purpose: Provide credible estimates of the start-up and ongoing effort and incremental practice expenses for the Advancing Care Together (ACT) behavioral health and primary care integration interventions. Methods: Expenditure data were collected from 10 practice intervention sites using an instrument with a standardized general format that could accommodate the unique elements of each intervention. Results: Average start-up effort expenses were

Evaluation of an Electronic Health Record (EHR) Tool for Integrated Behavioral Health in Primary Care

44,076 and monthly ongoing effort expenses per patient were

Understanding Care Integration from the Ground Up: Five Organizing Constructs that Shape Integrated Practices

40.39. Incremental expenses averaged

Preparing the Workforce for Behavioral Health and Primary Care Integration

20,788 for start-up and

Electronic Health Record Challenges, Workarounds, and Solutions Observed in Practices Integrating Behavioral Health and Primary Care

4.58 per patient for monthly ongoing activities. Variations in expenditures across practices reflect the differences in intervention specifics and organizational settings. Differences in effort to incremental expenditures reflect the extensive use of existing resources in implementing the interventions. Conclusions: ACT program incremental expenses suggest that widespread adoption would likely have a relatively modest effect on overall health systems expenditures. Practice effort expenses are not trivial and may pose barriers to adoption. Payers and purchasers interested in attaining widespread adoption of integrated care must consider external support to practices that accounts for both incremental and effort expense levels. Existing knowledge transfer mechanisms should be employed to minimize developmental start-up expenses and payment reform focused toward value-based, Triple Aim–oriented reimbursement and purchasing mechanisms are likely needed.

Integrating Behavioral Health and Primary Care: Consulting, Coordinating and Collaborating Among Professionals

Background: Integrating behavioral health into primary care can improve care quality; however, most electronic health records are not designed to meet the needs of integrated teams. We worked with practices and behavioral health (BH) clinicians to design a suite of electronic health record tools to address these needs (“BH e-Suite”). The purpose of this article is to examine whether implementation of the BH e-Suite changes process of care, intermediate clinical outcomes, and patient experiences, and whether its use is acceptable to practice members and BH clinicians. Methods: We conducted a convergent mixed-methods proof-of-concept study, implementing the BH e-Suite across 6 Oregon federally qualified community health centers (“intervention clinics”). We matched intervention clinics to 6 control clinics, based on location and patient panel characteristics, to assess whether process of care (Patient Health Questionnaire-9 [PHQ-9] and Generalized Anxiety Disorder-7 screening) and intermediate outcomes (PHQ-9, Generalized Anxiety Disorder-7 scores) changed postimplementation. Prepost patient surveys were used to assess changes in patient experience. To elucidate factors influencing implementation, we merged quantitative findings with structured observations, surveys, and interviews with practice members. Results: Implementation improved process of care (PHQ-9 screening). During the course of the study, change in intermediate outcomes was not observed. Degree of BH e-Suite implementation varied: 2 clinics fully implemented, 2 partially implemented, and 2 practices did not implement at all. Initial practice conditions (eg, low resistance to change, higher capacity), process characteristics (eg, thoughtful planning), and individual characteristics (eg, high self-efficacy) were related to degree of implementation. Conclusions: Health information technology tools designed for behavioral health integration must fit the needs of clinics for the successful uptake and improvement in patient experiences. Research is needed to further assess the effectiveness of this tool in improving patient outcomes and to optimize broader dissemination of this tool among primary care clinics integrating behavioral health.