Sara Jacobs

Organizational readiness for implementing change: a psychometric assessment of a new measure

BackgroundOrganizational readiness for change in healthcare settings is an important factor in successful implementation of new policies, programs, and practices. However, research on the topic is hindered by the absence of a brief, reliable, and valid measure. Until such a measure is developed, we cannot advance scientific knowledge about readiness or provide evidence-based guidance to organizational leaders about how to increase readiness. This article presents results of a psychometric assessment of a new measure called Organizational Readiness for Implementing Change (ORIC), which we developed based on Weiner’s theory of organizational readiness for change.MethodsWe conducted four studies to assess the psychometric properties of ORIC. In study one, we assessed the content adequacy of the new measure using quantitative methods. In study two, we examined the measure’s factor structure and reliability in a laboratory simulation. In study three, we assessed the reliability and validity of an organization-level measure of readiness based on aggregated individual-level data from study two. In study four, we conducted a small field study utilizing the same analytic methods as in study three.ResultsContent adequacy assessment indicated that the items developed to measure change commitment and change efficacy reflected the theoretical content of these two facets of organizational readiness and distinguished the facets from hypothesized determinants of readiness. Exploratory and confirmatory factor analysis in the lab and field studies revealed two correlated factors, as expected, with good model fit and high item loadings. Reliability analysis in the lab and field studies showed high inter-item consistency for the resulting individual-level scales for change commitment and change efficacy. Inter-rater reliability and inter-rater agreement statistics supported the aggregation of individual level readiness perceptions to the organizational level of analysis.ConclusionsThis article provides evidence in support of the ORIC measure. We believe this measure will enable testing of theories about determinants and consequences of organizational readiness and, ultimately, assist healthcare leaders to reduce the number of health organization change efforts that do not achieve desired benefits. Although ORIC shows promise, further assessment is needed to test for convergent, discriminant, and predictive validity.

Context matters: measuring implementation climate among individuals and groups

BackgroundIt has been noted that implementation climate is positively associated with implementation effectiveness. However, issues surrounding the measurement of implementation climate, or the extent to which organizational members perceive that innovation use is expected, supported and rewarded by their organization remain. Specifically, it is unclear whether implementation climate can be measured as a global construct, whether individual or group-referenced items should be used, and whether implementation climate can be assessed at the group or organizational level.MethodsThis research includes two cross-sectional studies with data collected via surveys at the individual level. The first study assessed the implementation climate perceptions of physicians participating in the National Cancer Institute’s (NCI) Community Clinical Oncology Program (CCOP), and the second study assessed the perceptions of children’s behavioral health clinicians implementing a treatment innovation. To address if implementation climate is a global construct, we used confirmatory factor analysis. To address how implementation climate should be measured and at what level, we followed a five-step framework outlined by van Mierlo and colleagues. This framework includes exploratory factor analysis and correlations to assess differences between individual and group-referenced items and intraclass correlations, interrater agreements, and exploratory factor analysis to determine if implementation climate can be assessed at the organizational level.ResultsThe confirmatory factor analysis demonstrated that implementation climate is a global construct consisting of items related to expectations, support and rewards. There are mixed results, however, as to whether implementation climate should be measured using individual or group-referenced items. In our first study, where physicians were geographically dispersed and practice independently, there were no differences based on the type of items used, and implementation climate was an individual level construct. However, in the second study, in which clinicians practice in a central location and interact more frequently, group-referenced items may be appropriate. In addition, implementation climate could be considered an organizational level construct.ConclusionsThe results are context-specific. Researchers should carefully consider the study setting when measuring implementation climate. In addition, more opportunities are needed to validate this measure and understand how well it predicts and explains implementation effectiveness.

Determining the predictors of innovation implementation in healthcare: A quantitative analysis of implementation effectiveness

BackgroundThe failure rates for implementing complex innovations in healthcare organizations are high. Estimates range from 30% to 90% depending on the scope of the organizational change involved, the definition of failure, and the criteria to judge it. The innovation implementation framework offers a promising approach to examine the organizational factors that determine effective implementation. To date, the utility of this framework in a healthcare setting has been limited to qualitative studies and/or group level analyses. Therefore, the goal of this study was to quantitatively examine this framework among individual participants in the National Cancer Institute’s Community Clinical Oncology Program using structural equation modeling.MethodsWe examined the innovation implementation framework using structural equation modeling (SEM) among 481 physician participants in the National Cancer Institute’s Community Clinical Oncology Program (CCOP). The data sources included the CCOP Annual Progress Reports, surveys of CCOP physician participants and administrators, and the American Medical Association Physician Masterfile.ResultsOverall the final model fit well. Our results demonstrated that not only did perceptions of implementation climate have a statistically significant direct effect on implementation effectiveness, but physicians’ perceptions of implementation climate also mediated the relationship between organizational implementation policies and practices (IPP) and enrollment (p <0.05). In addition, physician factors such as CCOP PI status, age, radiological oncologists, and non-oncologist specialists significantly influenced enrollment as well as CCOP organizational size and structure, which had indirect effects on implementation effectiveness through IPP and implementation climate.ConclusionsOverall, our results quantitatively confirmed the main relationship postulated in the innovation implementation framework between IPP, implementation climate, and implementation effectiveness among individual physicians. This finding is important, as although the model has been discussed within healthcare organizations before, the studies have been predominately qualitative in nature and/or at the organizational level. In addition, our findings have practical applications. Managers looking to increase implementation effectiveness of an innovation should focus on creating an environment that physicians perceive as encouraging implementation. In addition, managers should consider instituting specific organizational IPP aimed at increasing positive perceptions of implementation climate. For example, IPP should include specific expectations, support, and rewards for innovation use.

Organizational Designs for Achieving High Treatment Trial Enrollment: A Fuzzy-Set Analysis of the Community Clinical Oncology Program

PURPOSE To examine the organizational design features that were consistently associated in 2010 with high levels of patient enrollment onto National Cancer Institute (NCI) cancer treatment trials among the oncology practices and hospitals participating in the NCI Community Clinical Oncology Program (CCOP). METHODS Fuzzy-set qualitative comparative analysis was used to identify the recipes (ie, combinations of organizational design features) that CCOPs used to achieve high levels of patient enrollment onto NCI treatment trials in 2010. Four organizational design features were examined: number of open treatment trials with at least one patient enrolled, number of newly diagnosed patients with cancer, number of CCOP-affiliated physicians, and number of CCOP-affiliated hospitals or practices where patient enrollment could occur. Data were obtained from NCI data systems and CCOP grant progress reports. RESULTS Two recipes were consistently associated with high levels of patient enrollment onto NCI treatment trials in 2010: having many open treatment trials and many new patients with cancer, and having many open treatment trials and many affiliated hospitals or practices. Together, these recipes accounted for nearly two thirds of CCOP membership in the high-performance set in 2010. CONCLUSION No single organizational design feature, by itself, was consistently associated with high levels of patient enrollment onto NCI treatment trials in 2010. Having a large menu of active treatment trials may be necessary to achieve high-patient enrollment performance, but this is not sufficient unless combined with either large patient volume or many participating sites.

Organizational and physician factors associated with patient enrollment in cancer clinical trials

Background: Our purpose was to identify physicians’ individual characteristics, attitudes, and organizational contextual factors associated with higher enrollment of patients in cancer clinical trials among physician participants in the National Cancer Institute’s Community Clinical Oncology Program (CCOP). We hypothesized that physicians’ individual characteristics, such as age, medical specialty, tenure, CCOP organizational factors (i.e. policies and procedures to encourage enrollment), and attitudes toward participating in CCOP would directly determine enrollment. We also hypothesized that physicians’ characteristics and CCOP organizational factors would influence physicians’ attitudes toward participating in CCOP, which in turn would predict enrollment. Methods: We evaluated enrollment in National Cancer Institute–sponsored cancer clinical trials in 2011 among 481 physician participants using Structural Equation Modeling. The data sources include CCOP Annual Progress Reports, two surveys of CCOP administrators and physician participants, and the American Medical Association Masterfile. Results: Physicians with more positive attitudes toward participating in CCOP enrolled more patients than physicians with less positive attitudes. In addition, physicians who practiced in CCOPs that had more supportive policies and practices in place to encourage enrollment (i.e. offered trainings, provided support to screen and enroll patients, gave incentives to enroll patients, instituted minimum accrual expectations) also significantly enrolled more patients. Physician status as CCOP Principal Investigator had a positive direct effect on enrollment, while physician age and non-oncology medical specialty had negative direct effects on enrollment. Neither physicians’ characteristics nor CCOP organizational factors indirectly influenced enrollment through an effect on physician attitudes. Conclusion: We examined whether individual physicians’ characteristics and attitudes, as well as CCOP organizational factors, influenced patient enrollment in cancer clinical trials among CCOP physicians. Physician attitudes and CCOP organizational factors had positive direct effects, but not indirect effects, on physician enrollment of patients. Our results could be used to develop physician-directed strategies aimed at increasing involvement in clinical research. For example, administrators may want to ensure physicians have access to support staff to help screen and enroll patients or institute minimum accrual expectations. Our results also highlight the importance of recruiting physicians for volunteer clinical research programs whose attitudes and values align with programmatic goals. Given that physician involvement is a key determinant of patient enrollment in clinical trials, these interventions could expand the overall number of patients involved in cancer research. These strategies will be increasingly important as the CCOP network continues to evolve.

Using qualitative comparative analysis in a systematic review of a complex intervention

BackgroundSystematic reviews evaluating complex interventions often encounter substantial clinical heterogeneity in intervention components and implementation features making synthesis challenging. Qualitative comparative analysis (QCA) is a non-probabilistic method that uses mathematical set theory to study complex phenomena; it has been proposed as a potential method to complement traditional evidence synthesis in reviews of complex interventions to identify key intervention components or implementation features that might explain effectiveness or ineffectiveness. The objective of this study was to describe our approach in detail and examine the suitability of using QCA within the context of a systematic review.MethodsWe used data from a completed systematic review of behavioral interventions to improve medication adherence to conduct two substantive analyses using QCA. The first analysis sought to identify combinations of nine behavior change techniques/components (BCTs) found among effective interventions, and the second analysis sought to identify combinations of five implementation features (e.g., agent, target, mode, time span, exposure) found among effective interventions. For each substantive analysis, we reframed the review’s research questions to be designed for use with QCA, calibrated sets (i.e., transformed raw data into data used in analysis), and identified the necessary and/or sufficient combinations of BCTs and implementation features found in effective interventions.ResultsOur application of QCA for each substantive analysis is described in detail. We extended the original review findings by identifying seven combinations of BCTs and four combinations of implementation features that were sufficient for improving adherence. We found reasonable alignment between several systematic review steps and processes used in QCA except that typical approaches to study abstraction for some intervention components and features did not support a robust calibration for QCA.ConclusionsQCA was suitable for use within a systematic review of medication adherence interventions and offered insights beyond the single dimension stratifications used in the original completed review. Future prospective use of QCA during a review is needed to determine the optimal way to efficiently integrate QCA into existing approaches to evidence synthesis of complex interventions.

Achieving high cancer control trial enrollment in the community setting: An analysis of the Community Clinical Oncology Program

Determining the factors that lead to successful enrollment of patients in cancer control clinical trials is essential as cancer patients are often burdened with side effects such as pain, nausea, and fatigue. One promising intervention for increasing enrollment in cancer control trials is the National Cancer Institutes Community Clinical Oncology Program (CCOP). In this article, we examined CCOP staffing, policies, and procedures associated with enrollment in control trials. Data were obtained from three sources: the online CCOP, MB-CCOP, and Research Base Management System, CCOP Annual Progress Reports, and a survey of CCOP Administrators conducted in 2011. We analyzed cancer control trial accrual in 2011 among 46 CCOPs using multivariate regression. Three factors were significant predictors of accrual. First, having a team of staff dedicated to enrolling patients in control and prevention trials, compared to having no dedicated staff, was associated on average with an additional 30 patients enrolled in control trials (p<0.05). Second, CCOPs that recognized physicians for enrolling a large number of patients compared to CCOPs that did not recognize high enrolling physicians enrolled on average an additional 25 patients in control trials (p<0.05). Lastly, the number of cancer control trials available was also associated with enrollment (β=5.50, p<0.00). Our results indicate that CCOPs looking to increase enrollment in control trials should consider dedicating a team of staff to enroll patients in these types of trials. In addition, CCOPs or other volunteer research systems looking to increase physician participation should consider recognizing high enrolling physicians.

Assessing the feasibility of a virtual tumor board program: a case study.

EXECUTIVE SUMMARY Multidisciplinary tumor boards involve various providers (e.g., oncology physicians, nurses) in patient care. Although many community hospitals have local tumor boards that review all types of cases, numerous providers, particularly in rural areas and smaller institutions, still lack access to tumor boards specializing in a particular type of cancer (e.g., hematologic). Videoconferencing technology can connect providers across geographic locations and institutions; however, virtual tumor board (VTB) programs using this technology are uncommon. In this study, we evaluated the feasibility of a new VTB program at the University of North Carolina (UNC) Lineberger Comprehensive Cancer Center, which connects community‐based clinicians to UNC tumor boards representing different cancer types. Methods included observations, interviews, and surveys. Our findings suggest that participants were generally satisfied with the VTB. Cases presented to the VTB were appropriate, sufficient information was available for discussion, and technology problems were uncommon. UNC clinicians viewed the VTB as a service to patients and colleagues and an opportunity for clinical trial recruitment. Community‐based clinicians presenting at VTBs valued the discussion, even if it simply confirmed their original treatment plan or did not yield consensus recommendations. Barriers to participation for community‐based clinicians included timing of the VTB and lack of reimbursement. To maximize benefits of the VTB, these barriers should be addressed, scheduling and preparation processes optimized, and appropriate measures for evaluating impact identified.

Measuring constructs from the Consolidated Framework for Implementation Research in the context of increasing colorectal cancer screening at community health centers

Background The Consolidated Framework for Implementation Research (CFIR) is a comprehensive meta-framework widely applied to implementation related studies. Yet, few have used validated measures to operationalize constructs in CFIR in real-life settings. In this study, we operationalized selected CFIR constructs in an assessment to identify factors influencing implementation of evidence-based practices for increasing colorectal cancer screening in Community Health Centers (CHC).

Strategies for Successful Survivorship Care Plan Implementation: Results From a Qualitative Study

INTRODUCTION Care for US cancer survivors is often fragmented, contributing to poor health outcomes. Care and outcomes may improve when survivors and follow-up care providers receive survivorship care plans (SCPs), written documents containing information regarding cancer diagnosis, treatment, surveillance plans, and health promotion. However, implementing SCPs is challenging. As such, we sought to identify strategies for successfully implementing SCPs. METHODS We measured SCP implementation using performance data from cancer programs participating in the American Society of Clinical Oncology Quality Oncology Practice Initiative, an oncologist-led quality assessment and improvement program. We used semistructured interviews with cancer program employees (eg, physicians) to identify strategies for successfully implementing SCPs by comparing approaches in cancer programs that, according to Quality Oncology Practice Initiative performance indicators, developed and delivered SCPs to a relatively small proportion of eligible survivors and their follow-up care providers (ie, low performers; n = 6 participants in five programs) with approaches among programs with better performance (ie, moderate performers; n = 15 participants in nine programs). RESULTS Ten of 14 cancer programs developed SCPs for ≥ 50% of eligible survivors; two of 14 delivered SCPs to any survivors; and eight of 14 delivered SCPs to ≥ 25% of follow-up care providers. We found that moderate performers proactively addressed SCP requirements, leveraged requirements to improve survivorship care, set internal targets, automated implementation, had active leaders and champions, and tasked appropriate employees with SCP implementation. CONCLUSION SCP implementation remains challenging. We identified strategies for successfully implementing SCPs. Future research should examine how cancer programs have achieved these strategies; findings could contribute to an understanding of the changes needed to implement comprehensive survivorship care.