Becci A. Akin

Usability testing, initial implementation, and formative evaluation of an evidence-based intervention: Lessons from a demonstration project to reduce long-term foster care

The field of child welfare faces an undersupply of evidence-based interventions to address long-term foster care. The Permanency Innovations Initiative is a five-year federal demonstration project intended to generate evidence to reduce long stays in foster care for those youth who encounter the most substantial barriers to permanency. This article describes a systematic and staged approach to implementation and evaluation of a PII project that included usability testing as one of its key activities. Usability testing is an industry-derived practice which analyzes early implementation processes and evaluation procedures before they are finalized. This article describes the iterative selection, testing, and analysis of nine usability metrics that were designed to assess three important constructs of the projects initial implementation and evaluation: intervening early, obtaining consent, and engaging parents. Results showed that seven of nine metrics met a predetermined target. This study demonstrates how findings from usability testing influenced the initial implementation and formative evaluation of an evidence-supported intervention. Implications are discussed for usability testing as a quality improvement cycle that may contribute to better operationalized interventions and more reliable, valid, and replicable evidence.

Selecting an EBP to Reduce Long-Term Foster Care: Lessons from a University–Child Welfare Agency Partnership

A growing implementation literature outlines broad evidence-based practice implementation principles and pitfalls. Less robust is knowledge about the real-world process by which a state or agency chooses an evidence-based practice to implement and evaluate. Using a major U.S. initiative to reduce long-term foster care as the case, this article describes three major aspects of the evidence-based practice selection process: defining a target population, selecting an evidence-based practice model and purveyor, and tailoring the model to the practice context. Use of implementation science guidelines and lessons learned from a unique private-public-university partnership are discussed.

Examining the role of methamphetamine in permanency: A competing risks analysis of reunification, guardianship, and adoption.

Parental methamphetamine use has drawn significant attention in recent years. Despite prior research that shows that parental substance abuse is a risk factor for lengthy foster care stay, little is known about the effect of specific types of substance use on permanency. This study sought to compare the impact of parental methamphetamine use to alcohol use, other drug use, and polysubstance use on the timing of 3 types of permanency: reunification, guardianship, and adoption. Using an entry cohort of 16,620 children who had entered foster care during a 5-year period, competing risks event history models were conducted for each permanency type. Findings showed that, after controlling for several case characteristics, parent illicit drug use significantly impacted the timing of the 3 types of permanency, but alcohol use did not. Methamphetamine, other drug, and polysubstance with methamphetamine use were associated with lower rates of reunification and higher rates of adoption. Guardianship was also predicted by other drug and polysubstance use without methamphetamine; however, methamphetamine use was not associated with guardianship. Notably, the methamphetamine groups comprised the youngest children and had the shortest median time to adoption. Results suggest that type of parental substance use is predictive of permanency exits and that parental illicit drug use may require tailored strategies for improving permanency outcomes. Further implications of the findings are discussed.

Formative Evaluation of an Evidence-Based Intervention to Reduce Long-Term Foster Care: Assessing Readiness for Summative Evaluation

As part of the federal Permanency Innovations Initiative, this local grantee implemented an evidence-based intervention to improve outcomes for children with serious emotional disturbance. The project approach integrates the tenets of implementation science and program evaluation. Formative evaluation used a small sample (n = 76) to assess implementation integrity and statistical association to recommend whether the intervention was ready for summative evaluation (SE). Results showed that most process expectations were met. Statistical findings suggested that the intervention was associated with higher rates of reunification for treatment families as compared to families receiving services-as-usual. Thus, the intervention was recommended for SE.

Solution-Focused Brief Therapy With Substance-Using Individuals: A Randomized Controlled Trial Study

Objective: This study examined the effectiveness of solution-focused brief therapy (SFBT) intervention on substance abuse and trauma-related problems. Methods: A randomized controlled trial design was used to evaluate the effectiveness of SFBT in primary substance use treatment services for child welfare involved parents in outpatient treatment for substance use disorders. Mixed linear models were used to test within- and between-group changes using intent-to-treat analysis (N = 64). Hedges’s g effect sizes were also calculated to examine magnitude of treatment effects. Results: Both groups decreased on the Addiction Severity Index-Self-Report and the Trauma Symptom Checklist-40. The between group effect sizes were not statistically significant on either measures, thus SFBT produced similar results as the research supported treatments the control group received. Conclusion: Results support the use of SFBT in treating substance use and trauma and provide an alternative approach that is more strengths based and less problem focused.

Effect of a Parenting Intervention on Foster Care Reentry After Reunification Among Substance-Affected Families: A Quasi-Experimental Study:

Although parental substance abuse has been identified as a risk factor for poor foster care outcomes, current research on effective interventions is limited. A few studies have shown that parenting interventions improved parenting skills and family functioning and decreased time to reunification among children in foster care due to parental substance abuse. However, more research is needed to evaluate whether these interventions positively impact reentry rates. Using propensity score analyses to establish a matched comparison group, survival analyses evaluated the relationship between participation in a parenting intervention, the Strengthening Families Program (SFP), and reentry among a sample of 493 children previously reunified with their parents. The overall reentry rate was 20.9%. Analyses indicated that there was no difference in reentry rates between the SFP (23.7%) and comparison groups (18.6%). Significant predictors of reentry were child behavior problems, family poverty, and reunification between 15 and 18 months from removal.

Randomized Trial of PMTO in Foster Care: Six-Month Child Well-Being Outcomes

Objective: This study tested the effectiveness of Parent Management Training, Oregon (PMTO) model on child social–emotional well-being. Methods: Using a randomized controlled design and three measures of social–emotional well-being, the study investigated effectiveness of PMTO with families of children in foster care with serious emotional disturbance (SED). Participants included children (3–16 years) and parents who were randomly assigned to PMTO (n = 461) or services as usual (n = 457). Study condition was known to participants and assessors. Six months after baseline, analysis of covariance models examined the intervention’s overall effect and time interactions using intent-to-treat analysis. Follow-up analyses identified salient predictors of well-being. Results: PMTO demonstrated small but significant positive effects on three primary outcomes: social–emotional functioning (Cohen’s d = .31), problem behaviors (Cohen’s d = .09), and prosocial skills (Cohen’s d = .09). Conclusion: Results suggest that PMTO was effective at improving short-term social–emotional well-being in a high-risk population of children with SED.

Randomized Study of PMTO in Foster Care: Six-Month Parent Outcomes

Objective: This study examined the effects of Parent Management Training, Oregon (PMTO) model on parenting effectiveness and caregiver functioning. Method: Children in foster care with emotional and behavioral problems were randomized to PMTO (n = 461) or services as usual (n = 457) in a nonblinded study design. Using an intent-to-treat approach, analysis of covariance models tested the intervention’s overall effect and time interactions for parenting and caregiver functioning. Additional analyses were conducted to identify significant predictors of outcomes. Results: PMTO did not significantly affect parenting practices; however, positive effects were observed on caregiver functioning in mental health (odds ratio [OR] = 2.01), substance use (OR = 1.67), social supports (OR = 2.37), and readiness for reunification (OR = 1.64). While no time interactions were detected, several child, parent, and case characteristics were associated with improvements in 6-month outcomes. Conclusion: This study extends evidence on PMTO to biological families of children in foster care, including those with older youth.

A study in contrasts: Supports and barriers to successful implementation of two evidence-based parenting interventions in child welfare

Although evidence-based interventions (EBIs) are spreading to child welfare, research on real-world dynamics of implementation within this setting is scarce. Using a six-factor implementation framework to examine implementation of two evidence-based parenting interventions, we sought to build greater understanding of key facilitators and barriers by comparing successful versus failed EBI implementation in a child welfare setting. Semi-structured interviews were conducted with a purposive sample of 15 frontline practitioners and state-level managers. Interviews were transcribed verbatim and data analysis used a modified analytic approach. Our results showed the successful EBI was viewed more positively on all six factors; however, implementation was multidimensional, multilevel, and mixed with accomplishments and challenges. An accumulation of strengths across implementation factors proved beneficial. Implementation frameworks may be advantageous in organizing and explaining the numerous factors that may influence successful versus failed implementation. While encountering obstacles is largely inevitable, understanding which factors have shaped the success or failure of EBI implementations in child welfare settings may optimize future implementations in this context.

Engaging substance-affected families in child welfare: parent perspectives of a parenting intervention at program initiation and completion

ABSTRACT Families affected by substance abuse are at an increased risk of child welfare involvement and poor child welfare outcomes. One strategy to improve outcomes among these families is evidence-based parenting interventions. While these interventions show the potential for advancing the child welfare field, they have not been widely and rigorously evaluated with birth parents, especially those affected by substance abuse, who face marked vulnerability and marginalization. We sought to draw on parents’ expertise and to understand their first and changing impressions throughout involvement in a parenting intervention, as well as overall impressions at program completion. Semi-structured interviews were conducted with a purposive sample of 10 parents who were involved in child welfare and a family drug treatment court. All parents were affected by parental substance use and had recently completed the Strengthening Families Program. Interviews were transcribed verbatim and analyzed for prominent themes. The study’s findings showed that parents described four key themes that influenced their perspectives of the parenting intervention: program reputation as communicated by peers, relevance and applicability of the program, children’s involvement and enjoyment of the program, and program structure issues that pointed to the need for high-quality group facilitation. Implications for practice and research are discussed.