Lisa Saldana

Observational measure of implementation progress in community based settings: The Stages of implementation completion (SIC)

BackgroundAn increasingly large body of research is focused on designing and testing strategies to improve knowledge about how to embed evidence-based programs (EBP) into community settings. Development of strategies for overcoming barriers and increasing the effectiveness and pace of implementation is a high priority. Yet, there are few research tools that measure the implementation process itself. The Stages of Implementation Completion (SIC) is an observation-based measure that is used to track the time to achievement of key implementation milestones in an EBP being implemented in 51 counties in 53 sites (two counties have two sites) in two states in the United States.MethodsThe SIC was developed in the context of a randomized trial comparing the effectiveness of two implementation strategies: community development teams (experimental condition) and individualized implementation (control condition). Fifty-one counties were randomized to experimental or control conditions for implementation of multidimensional treatment foster care (MTFC), an alternative to group/residential care placement for children and adolescents. Progress through eight implementation stages was tracked by noting dates of completion of specific activities in each stage. Activities were tailored to the strategies for implementing the specific EBP.ResultsPreliminary data showed that several counties ceased progress during pre-implementation and that there was a high degree of variability among sites in the duration scores per stage and on the proportion of activities that were completed in each stage. Progress through activities and stages for three example counties is shown.ConclusionsBy assessing the attainment time of each stage and the proportion of activities completed, the SIC measure can be used to track and compare the effectiveness of various implementation strategies. Data from the SIC will provide sites with relevant information on the time and resources needed to implement MTFC during various phases of implementation. With some modifications, the SIC could be appropriate for use in evaluating implementation strategies in head-to-head randomized implementation trials and as a monitoring tool for rolling out other EBPs.

Engaging and Recruiting Counties in an Experiment on Implementing Evidence-based Practice in California

There is a growing consensus that implementation of evidence-based intervention and treatment models holds promise to improve the quality of services in child public service systems such as mental health, juvenile justice, and child welfare. Recent policy initiatives to integrate such research-based services into public service systems have created pressure to expand knowledge about implementation methods. Experimental strategies are needed to test multi-level models of implementation in real world contexts. In this article, the initial phase of a randomized trial that tests two methods of implementing Multidimensional Treatment Foster Care (an evidence-based intervention that crosses child public service systems) in 40 non-early adopting California counties is described. Results are presented that support the feasibility of using a randomized design to rigorously test contrasting implementation models and engaging system leaders to participate in the trial.

Predicting Program Start-Up Using the Stages of Implementation Measure

Recent efforts to better understand the process of implementation have been hampered by a lack of tools available to define and measure implementation progress. The Stages of Implementation Completion (SIC) was developed as part of an implementation trial of MTFC in 53 sites, and identifies the duration of time spent on implementation activities and the proportion of activities completed. This article examines the ability of the first three stages of the SIC (Engagement, Consideration of Feasibility, Readiness Planning) to predict successful program start-up. Results suggest that completing SIC stages completely, yet relatively quickly, predicts the likelihood of successful implementation.

Supporting Implementation: The Role of Community Development Teams to Build Infrastructure

Evidence-based methods for assisting consumers, such as counties, in successfully implementing practices are lacking in the field of implementation science. To fill this gap, the Community Development Teams (CDT) approach was developed to assist counties in developing peer networks focused on problem-solving and resource sharing to enhance their possibility of successful implementation. The CDT is an interactive, solution-focused approach that shares many elements of the Interactive Systems Framework (ISF) for Dissemination and Implementation. An ongoing randomized implementation trial of Multidimensional Treatment Foster Care (MTFC) was designed to test the hypothesis that such interactive implementation methods are more successful at helping counties achieve successful and sustainable MTFC programs than standard individualized implementation methods. Using the Stages of Implementation Completion measure, developed for this study, the potential benefit of these interactive methods is examined at different stages of the implementation process ranging from initial engagement to program competency.

The stages of implementation completion for evidence-based practice: protocol for a mixed methods study

BackgroundThis protocol describes the ‘development of outcome measures and suitable methodologies for dissemination and implementation approaches,’ a priority for implementation research. Although many evidence-based practices (EBPs) have been developed, large knowledge gaps remain regarding how to routinely move EBPs into usual care. The lack of understanding of ‘what it takes’ to install EBPs has costly public health consequences, including a lack of availability of the most beneficial services, wasted efforts and resources on failed implementation attempts, and the potential for engendering reluctance to try implementing new EBPs after failed attempts.The Stages of Implementation Completion (SIC) is an eight-stage tool of implementation process and milestones, with stages spanning three implementation phases (pre-implementation, implementation, sustainability). Items delineate the date that a site completes implementation activities, yielding an assessment of duration (time to complete a stage), proportion (of stage activities completed), and a general measure of how far a site moved in the implementation process.Methods/DesignWe propose to extend the SIC to EBPs operating in child service sectors (juvenile justice, schools, substance use, child welfare). Both successful and failed implementation attempts will be scrutinized using a mixed methods design. Stage costs will be measured and examined. Both retrospective data (from previous site implementation efforts) and prospective data (from newly adopting sites) will be analyzed. The influence of pre-implementation on implementation and sustainability outcomes will be examined (Aim 1). Mixed methods procedures will focus on increasing understanding of the process of implementation failure in an effort to determine if the SIC can provide early detection of sites that are unlikely to succeed (Aim 2). Study activities will include cost mapping of SIC stages and an examination of the relationship between implementation costs and implementation performance (Aim 3).DiscussionThis project fills a gap in the field of implementation science by addressing the measurement gap between the implementation process and the associated costs. The goal of this project is to provide tools that will help increase the uptake of EBPs, thereby increasing the availability of services to youth and decreasing wasted resources from failed implementation efforts.

Evaluation of two implementation strategies in 51 child county public service systems in two states: results of a cluster randomized head-to-head implementation trial

BackgroundMuch is to be learned about what implementation strategies are the most beneficial to communities attempting to adopt evidence-based practices. This paper presents outcomes from a randomized implementation trial of Multidimensional Treatment Foster Care (MTFC) in child public service systems in California and Ohio, including child welfare, juvenile justice, and mental health.MethodsFifty-one counties were assigned randomly to one of two different implementation strategies (Community Development Teams (CDT) or independent county implementation strategy (IND)) across four cohorts after being matched on county characteristics. We compared these two strategies on implementation process, quality, and milestone achievements using the Stages of Implementation Completion (SIC) (Implement Sci 6(1):1-8, 2011).ResultsA composite score for each county, combining the final implementation stage attained, the number of families served, and quality of implementation, was used as the primary outcome. No significant difference between CDT and IND was found for the composite measure. Additional analyses showed that there was no evidence that CDT increased the proportion of counties that started-up programs (i.e., placed at least one family in MTFC). For counties that did implement MTFC, those in the CDT condition served over twice as many youth during the study period as did IND. Of the counties that successfully achieved program start-up, those in the CDT condition completed the implementation process more thoroughly, as measured by the SIC. We found no significant differences by implementation condition on the time it took for first placement, achieving competency, or number of stages completed.ConclusionsThis trial did not lead to higher rates of implementation or faster implementation but did provide evidence for more robust implementation in the CDT condition compared to IND implementation once the first family received MTFC services. This trial was successful from a design perspective in that no counties dropped out, even though this study took place during an economic recession. We believe that this methodologic approach of measurement utilizing the SIC, which is comprised of the three dimensions of quality, quantity, and timing, is appropriate for a wide range of implementation and translational studies.Trial registrationTrial ID: NCT00880126 (ClinicalTrials.gov).

A Strategy for Assessing Costs of Implementing New Practices in the Child Welfare System: Adapting the English Cost Calculator in the United States

In decisions to adopt and implement new practices or innovations in child welfare, costs are often a bottom-line consideration. The cost calculator, a method developed in England that can be used to calculate unit costs of core case work activities and associated administrative costs, is described as a potentially helpful tool for assisting child welfare administrators to evaluate the costs of current practices relative to their outcomes and could impact decisions about whether to implement new practices. The process by which the cost calculator is being adapted for use in US child welfare systems in two states is described and an illustration of using the method to compare two intervention approaches is provided.

Implementation and evaluation of linked parenting models in a large urban child welfare system

During the past decade, there have been increased efforts to implement evidence-based practices into child welfare systems to improve outcomes for children in foster care and their families. In this paper, the implementation and evaluation of a policy-driven large system-initiated reform is described. Over 250 caseworkers and supervisors were trained and supported to implement two evidence-based parent focused interventions in five private agencies serving over 2,000 children and families. At the request of child welfare system leaders, a third intervention was developed and implemented to train the social work workforce to use evidence-based principles in everyday interactions with caregivers (including foster, relative, adoptive, and biological parents). In this paper, we describe the policy context and the targeted outcomes of the reform. We discuss the theory of the interventions and the logistics of how they were linked to create consistency and synergy. Training and ongoing consultation strategies used are described as are some of the barriers and opportunities that arose during the implementation. The strategy for creating a path to sustainability is also discussed. The reform effort was evaluated using both qualitative and quantitative methods; the evaluation design, research questions and preliminary results are provided.

A method for assessing implementation success of a peer-led suicide prevention program

Background Peer leader programs that prepare opinion leaders to spread healthy practices through their social networks reduce high-risk sex behaviors and show promise in preventing adolescent substance use and suicidal behavior. However, knowledge of implementation processes is very limited. To address this limitation, we drew on the Stages of Implementation Completion (SIC) framework to measure a key phase of peer leader implementation.

Incremental net benefit of early intervention for preschool-aged children with emotional and behavioral problems in foster care

Of 1 million cases of child maltreatment identified every year in the United States, one-fifth result in foster care. Many of these children suffer from significant emotional and behavioral conditions. Decision-makers must allocate highly constrained budgets to serve these children. Recent evidence suggests that Multidimensional Treatment Foster Care for Preschoolers can reduce negative outcomes for these children, but the relative benefits and costs of the program have not been evaluated. The objective of this study was to assess net benefit, over 24 months, of Multidimensional Treatment Foster Care for Preschoolers compared to regular foster care. Data were from a randomized controlled trial of 117 young children entering a new foster placement. A subsample exhibited placement instability (n = 52). Intervention services including parent training, lasted 9-12 months. Multidimensional Treatment Foster Care for Preschoolers significantly increased permanent placements for the placement instability sample. Average total cost for the new intervention sample was significantly less than for regular foster care (full sample: