Alicia C. Bunger

Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda.

An unresolved issue in the field of implementation research is how to conceptualize and evaluate successful implementation. This paper advances the concept of “implementation outcomes” distinct from service system and clinical treatment outcomes. This paper proposes a heuristic, working “taxonomy” of eight conceptually distinct implementation outcomes—acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability—along with their nominal definitions. We propose a two-pronged agenda for research on implementation outcomes. Conceptualizing and measuring implementation outcomes will advance understanding of implementation processes, enhance efficiency in implementation research, and pave the way for studies of the comparative effectiveness of implementation strategies.

A compilation of strategies for implementing clinical innovations in health and mental health

Efforts to identify, develop, refine, and test strategies to disseminate and implement evidence-based treatments have been prioritized in order to improve the quality of health and mental health care delivery. However, this task is complicated by an implementation science literature characterized by inconsistent language use and inadequate descriptions of implementation strategies. This article brings more depth and clarity to implementation research and practice by presenting a consolidated compilation of discrete implementation strategies, based on a review of 205 sources published between 1995 and 2011. The resulting compilation includes 68 implementation strategies and definitions, which are grouped according to six key implementation processes: planning, educating, financing, restructuring, managing quality, and attending to the policy context. This consolidated compilation can serve as a reference to stakeholders who wish to implement clinical innovations in health and mental health care and can facilitate the development of multifaceted, multilevel implementation plans that are tailored to local contexts.

Public health program capacity for sustainability: a new framework

BackgroundPublic health programs can only deliver benefits if they are able to sustain activities over time. There is a broad literature on program sustainability in public health, but it is fragmented and there is a lack of consensus on core constructs. The purpose of this paper is to present a new conceptual framework for program sustainability in public health.MethodsThis developmental study uses a comprehensive literature review, input from an expert panel, and the results of concept-mapping to identify the core domains of a conceptual framework for public health program capacity for sustainability. The concept-mapping process included three types of participants (scientists, funders, and practitioners) from several public health areas (e.g., tobacco control, heart disease and stroke, physical activity and nutrition, and injury prevention).ResultsThe literature review identified 85 relevant studies focusing on program sustainability in public health. Most of the papers described empirical studies of prevention-oriented programs aimed at the community level. The concept-mapping process identified nine core domains that affect a program’s capacity for sustainability: Political Support, Funding Stability, Partnerships, Organizational Capacity, Program Evaluation, Program Adaptation, Communications, Public Health Impacts, and Strategic Planning. Concept-mapping participants further identified 93 items across these domains that have strong face validity—89% of the individual items composing the framework had specific support in the sustainability literature.ConclusionsThe sustainability framework presented here suggests that a number of selected factors may be related to a program’s ability to sustain its activities and benefits over time. These factors have been discussed in the literature, but this framework synthesizes and combines the factors and suggests how they may be interrelated with one another. The framework presents domains for public health decision makers to consider when developing and implementing prevention and intervention programs. The sustainability framework will be useful for public health decision makers, program managers, program evaluators, and dissemination and implementation researchers.

Context matters: measuring implementation climate among individuals and groups

BackgroundIt has been noted that implementation climate is positively associated with implementation effectiveness. However, issues surrounding the measurement of implementation climate, or the extent to which organizational members perceive that innovation use is expected, supported and rewarded by their organization remain. Specifically, it is unclear whether implementation climate can be measured as a global construct, whether individual or group-referenced items should be used, and whether implementation climate can be assessed at the group or organizational level.MethodsThis research includes two cross-sectional studies with data collected via surveys at the individual level. The first study assessed the implementation climate perceptions of physicians participating in the National Cancer Institute’s (NCI) Community Clinical Oncology Program (CCOP), and the second study assessed the perceptions of children’s behavioral health clinicians implementing a treatment innovation. To address if implementation climate is a global construct, we used confirmatory factor analysis. To address how implementation climate should be measured and at what level, we followed a five-step framework outlined by van Mierlo and colleagues. This framework includes exploratory factor analysis and correlations to assess differences between individual and group-referenced items and intraclass correlations, interrater agreements, and exploratory factor analysis to determine if implementation climate can be assessed at the organizational level.ResultsThe confirmatory factor analysis demonstrated that implementation climate is a global construct consisting of items related to expectations, support and rewards. There are mixed results, however, as to whether implementation climate should be measured using individual or group-referenced items. In our first study, where physicians were geographically dispersed and practice independently, there were no differences based on the type of items used, and implementation climate was an individual level construct. However, in the second study, in which clinicians practice in a central location and interact more frequently, group-referenced items may be appropriate. In addition, implementation climate could be considered an organizational level construct.ConclusionsThe results are context-specific. Researchers should carefully consider the study setting when measuring implementation climate. In addition, more opportunities are needed to validate this measure and understand how well it predicts and explains implementation effectiveness.

Administrative Coordination in Nonprofit Human Service Delivery Networks: The Role of Competition and Trust

Nonprofit human service organizations operating within the same regional network are often faced with dual pressure to compete as well as coordinate administrative operations (by sharing funding, staff, or space) to enhance efficiency. Emerging evidence has demonstrated that competing organizations coordinate, despite the risks. Trust, or perceived trustworthiness between two organizations may mitigate the negative influence of competition on coordination, however there have been few explicit tests of this hypothesis among nonprofit organizations. Drawing on quantitative data collected from a network of 36 nonprofit children’s behavioral health organizations, this article empirically tests how competition and perceived trustworthiness interact to influence administrative coordination. Results support the hypothesis that trustworthiness moderates the influence of competition on administrative coordination. Findings suggest that as competing nonprofit leaders build trust, the more their agencies coordinate their administrative functions. This study highlights the importance of leaders’ perceptions for organizational strategy.

Defining Service Coordination: A Social Work Perspective

ABSTRACT This pilot study explores the meaning of and factors that influence coordination as a first step toward conceptual refinement, theory development, and system interventions. Using data from treatment guidelines archived by the National Guideline Clearinghouse (n= 9) and semistructured interviews with social workers (n= 4), themes related to the definition, indicators, and perceptions of coordination were explored using a grounded theory approach. Data suggest the need for coordination is driven by complex client needs, but the quality of providers’ personal relationships influences coordination. Future research might examine the impact of standardization of roles, referral procedures, and treatment philosophies.

Treatment guidelines for substance use disorders and serious mental illnesses: do they address co-occurring disorders?

Practice guidelines are important tools for improving the delivery of evidence-based practices and reducing inappropriate variation in current treatment approaches. This study examined the degree to which guidelines targeted to the treatment of substance use disorders or serious mental illness address treatment of co-occurring disorders. Guidelines archived by the National Guideline Clearinghouse (NGC) were retrieved in December 2007 and content analyzed. Nineteen pertinent guidelines were identified, and 11 included recommendations regarding the assessment and/or treatment of co-occurring disorders. None of the guidelines making recommendations for treatment of co-occurring disorders included outcomes that clearly targeted both substance use and mental health disorders. Limitations and implications of this study are noted.

Can Learning Collaboratives Support Implementation by Rewiring Professional Networks

This study examined how a learning collaborative focusing on trauma-focused CBT (TF-CBT) impacted advice-seeking patterns between clinicians and three key learning sources: (1) training experts who share technical knowledge about TF-CBT, (2) peers from other participating organizations who share their implementation experiences, and (3) colleagues from their own agency who provide social and professional support. Based on surveys administered to 132 clinicians from 32 agencies, participants’ professional networks changed slightly over time by forming new advice-seeking relationships with training experts. While small, these changes at the clinician-level yielded substantial changes in the structure of the regional advice network.

Under What Conditions Does Caseworker-Caregiver Racial/Ethnic Similarity Matter for Housing Service Provision? An Application of Representative Bureaucracy Theory

In this article, we examine child welfare caseworkers’ housing-related service strategies when they serve culturally similar versus culturally dissimilar clients. Testing hypotheses drawn from representative bureaucracy theory and using data from the second cohort of the National Survey of Child and Adolescent Well-Being, we find that when non-Caucasian caseworkers share the same racial/ethnic background as caregivers, caseworkers use more active strategies to connect caregivers to needed housing services. The relationship between racial/ethnic matching and frontline workers’ repertoire of service strategies is most pronounced when the need for housing has been registered formally via referrals and case plans and thus legitimated institutionally. These results reinforce basic tenets of representative bureaucracy theory and provide evidence of the benefits of racial and ethnic diversity in the human service workforce. Our findings also highlight the need for research identifying institutional and frontline organizational factors that enhance the quality of service provision.

Building Service Delivery Networks: Partnership Evolution Among Children’s Behavioral Health Agencies in Response to New Funding

Meeting the complex needs of youth with behavioral health problems requires a coordinated network of community-based agencies. Although fiscal scarcity or retrenchment can limit coordinated services, munificence can stimulate service delivery partnerships as agencies expand programs, hire staff, and spend more time coordinating services. This study examines the 2-year evolution of referral and staff expertise sharing networks in response to substantial new funding for services within a regional network of children’s mental health organizations. Quantitative network survey data were collected from directors of 22 nonprofit organizations that receive funding from a county government-based behavioral health service fund. Both referral and staff expertise sharing networks changed over time, but results of a stochastic actor-oriented model of network dynamics suggest the nature of this change varies for these networks. Agencies with higher numbers of referral and staff expertise sharing partners tend to maintain these ties and/or develop new relationships over the 2 years. Agencies tend to refer to agencies they trust, but trust was not associated with staff expertise sharing ties. However, agencies maintain or form staff expertise sharing ties with referral partners, or with organizations that provide similar services. In addition, agencies tend to reciprocate staff expertise sharing, but not referrals. Findings suggest that during periods of resource munificence and service expansion, behavioral health organizations build service delivery partnerships in complex ways that build upon prior collaborative history and coordinate services among similar types of providers. Referral partnerships can pave the way for future information sharing relationships.