Carlos Gómez Gallo

Partnerships for the Design, Conduct, and Analysis of Effectiveness, and Implementation Research: Experiences of the Prevention Science and Methodology Group

What progress prevention research has made comes through strategic partnerships with communities and institutions that host this research, as well as professional and practice networks that facilitate the diffusion of knowledge about prevention. We discuss partnership issues related to the design, analysis, and implementation of prevention research and especially how rigorous designs, including random assignment, get resolved through a partnership between community stakeholders, institutions, and researchers. These partnerships shape not only study design, but they determine the data that can be collected and how results and new methods are disseminated. We also examine a second type of partnership to improve the implementation of effective prevention programs into practice. We draw on social networks to studying partnership formation and function. The experience of the Prevention Science and Methodology Group, which itself is a networked partnership between scientists and methodologists, is highlighted.

A computational future for preventing HIV in minority communities: how advanced technology can improve implementation of effective programs.

Abstract:African Americans and Hispanics in the United States have much higher rates of HIV than non-minorities. There is now strong evidence that a range of behavioral interventions are efficacious in reducing sexual risk behavior in these populations. Although a handful of these programs are just beginning to be disseminated widely, we still have not implemented effective programs to a level that would reduce the population incidence of HIV for minorities. We proposed that innovative approaches involving computational technologies be explored for their use in both developing new interventions and in supporting wide-scale implementation of effective behavioral interventions. Mobile technologies have a place in both of these activities. First, mobile technologies can be used in sensing contexts and interacting to the unique preferences and needs of individuals at times where intervention to reduce risk would be most impactful. Second, mobile technologies can be used to improve the delivery of interventions by facilitators and their agencies. Systems science methods including social network analysis, agent-based models, computational linguistics, intelligent data analysis, and systems and software engineering all have strategic roles that can bring about advances in HIV prevention in minority communities. Using an existing mobile technology for depression and 3 effective HIV prevention programs, we illustrated how 8 areas in the intervention/implementation process can use innovative computational approaches to advance intervention adoption, fidelity, and sustainability.

Blending Qualitative and Computational Linguistics Methods for Fidelity Assessment: Experience with the Familias Unidas Preventive Intervention.

AbstractCareful fidelity monitoring and feedback are critical to implementing effective interventions. A wide range of procedures exist to assess fidelity; most are derived from observational assessments (Schoenwald and Garland, Psycholog Assess 25:146–156, 2013). However, these fidelity measures are resource intensive for research teams in efficacy/effectiveness trials, and are often unattainable or unmanageable for the host organization to rate when the program is implemented on a large scale. We present a first step towards automated processing of linguistic patterns in fidelity monitoring of a behavioral intervention using an innovative mixed methods approach to fidelity assessment that uses rule-based, computational linguistics to overcome major resource burdens. Data come from an effectiveness trial of the Familias Unidas intervention, an evidence-based, family-centered preventive intervention found to be efficacious in reducing conduct problems, substance use and HIV sexual risk behaviors among Hispanic youth. This computational approach focuses on “joining,” which measures the quality of the working alliance of the facilitator with the family. Quantitative assessments of reliability are provided. Kappa scores between a human rater and a machine rater for the new method for measuring joining reached 0.83. Early findings suggest that this approach can reduce the high cost of fidelity measurement and the time delay between fidelity assessment and feedback to facilitators; it also has the potential for improving the quality of intervention fidelity ratings.

Human Subjects Protection and Technology in Prevention Science: Selected Opportunities and Challenges

Internet-connected devices are changing the way people live, work, and relate to one another. For prevention scientists, technological advances create opportunities to promote the welfare of human subjects and society. The challenge is to obtain the benefits while minimizing risks. In this article, we use the guiding principles for ethical human subjects research and proposed changes to the Common Rule regulations, as a basis for discussing selected opportunities and challenges that new technologies present for prevention science. The benefits of conducting research with new populations, and at new levels of integration into participants’ daily lives, are presented along with five challenges along with technological and other solutions to strengthen the protections that we provide: (1) achieving adequate informed consent with procedures that are acceptable to participants in a digital age; (2) balancing opportunities for rapid development and broad reach, with gaining adequate understanding of population needs; (3) integrating data collection and intervention into participants’ lives while minimizing intrusiveness and fatigue; (4) setting appropriate expectations for responding to safety and suicide concerns; and (5) safeguarding newly available streams of sensitive data. Our goal is to promote collaboration between prevention scientists, institutional review boards, and community members to safely and ethically harness advancing technologies to strengthen impact of prevention science.

“Home Practice Is the Program”: Parents’ Practice of Program Skills as Predictors of Outcomes in the New Beginnings Program Effectiveness Trial

An examination of the content and processes of evidence-based programs is critical for empirically evaluating theories about how programs work, the “action theory” of the program (West et al. in American Journal of Community Psychology, 21, 571–605, 1993). The New Beginnings Program (NBP; Wolchik et al., 2007), a parenting-after-divorce preventive intervention, theorizes that program-induced improvements in parenting across three domains: positive relationship quality, effective discipline, and protecting children from interparental conflict, will reduce the negative outcomes that are common among children from divorced families. The process theory is that home practice of program skills related to these parenting domains is the primary mechanism leading to positive change in parenting. This theory was tested using multi-rater data from 477 parents in the intervention condition of an effectiveness trial of the NBP (Sandler et al. 2016a, 2016b). Four research questions were addressed: Does home practice of skills predict change in the associated parenting outcomes targeted by the program? Is the effect above and beyond the influence of attendance at program sessions? What indicators of home practice (i.e., attempts, fidelity, efficacy, and competence) are most predictive of improvements in parenting? Do these indicators predict parenting improvements in underserved subpopulations (i.e., fathers and Latinos)? Structural Equation Modeling analyses indicated that parent-reported efficacy and provider-rated parent competence of home practice predicted improvements in the targeted parenting domains according to both parent and child reports. Moreover, indicators of home practice predicted improvements in parenting for fathers and Latinos, although patterns of effects varied by parenting outcome.

Automatic classification of communication logs into implementation stages via text analysis.

BackgroundTo improve the quality, quantity, and speed of implementation, careful monitoring of the implementation process is required. However, some health organizations have such limited capacity to collect, organize, and synthesize information relevant to its decision to implement an evidence-based program, the preparation steps necessary for successful program adoption, the fidelity of program delivery, and the sustainment of this program over time. When a large health system implements an evidence-based program across multiple sites, a trained intermediary or broker may provide such monitoring and feedback, but this task is labor intensive and not easily scaled up for large numbers of sites.We present a novel approach to producing an automated system of monitoring implementation stage entrances and exits based on a computational analysis of communication log notes generated by implementation brokers. Potentially discriminating keywords are identified using the definitions of the stages and experts’ coding of a portion of the log notes. A machine learning algorithm produces a decision rule to classify remaining, unclassified log notes.ResultsWe applied this procedure to log notes in the implementation trial of multidimensional treatment foster care in the California 40-county implementation trial (CAL-40) project, using the stages of implementation completion (SIC) measure. We found that a semi-supervised non-negative matrix factorization method accurately identified most stage transitions. Another computational model was built for determining the start and the end of each stage.ConclusionsThis automated system demonstrated feasibility in this proof of concept challenge. We provide suggestions on how such a system can be used to improve the speed, quality, quantity, and sustainment of implementation. The innovative methods presented here are not intended to replace the expertise and judgement of an expert rater already in place. Rather, these can be used when human monitoring and feedback is too expensive to use or maintain. These methods rely on digitized text that already exists or can be collected with minimal to no intrusiveness and can signal when additional attention or remediation is required during implementation. Thus, resources can be allocated according to need rather than universally applied, or worse, not applied at all due to their cost.

The Cascading Effects of Multiple Dimensions of Implementation on Program Outcomes: a Test of a Theoretical Model

This study tests a theoretical cascade model in which multiple dimensions of facilitator delivery predict indicators of participant responsiveness, which in turn lead to improvements in targeted program outcomes. An effectiveness trial of the 10-session New Beginnings Program for divorcing families was implemented in partnership with four county-level family courts. This study included 366 families assigned to the intervention condition who attended at least one session. Independent observers provided ratings of program delivery (i.e., fidelity to the curriculum and process quality). Facilitators reported on parent attendance and parents’ competence in home practice of program skills. At pretest and posttest, children reported on parenting and parents reported child mental health. We hypothesized effects of quality on attendance, fidelity and attendance on home practice, and home practice on improvements in parenting and child mental health. Structural Equation Modeling with mediation and moderation analyses were used to test these associations. Results indicated quality was significantly associated with attendance, and attendance moderated the effect of fidelity on home practice. Home practice was a significant mediator of the links between fidelity and improvements in parent-child relationship quality and child externalizing and internalizing problems. Findings provide support for fidelity to the curriculum, process quality, attendance, and home practice as valid predictors of program outcomes for mothers and fathers. Future directions for assessing implementation in community settings are discussed.

A statistical method for synthesizing mediation analyses using the product of coefficient approach across multiple trials

Mediation analysis often requires larger sample sizes than main effect analysis to achieve the same statistical power. Combining results across similar trials may be the only practical option for increasing statistical power for mediation analysis in some situations. In this paper, we propose a method to estimate: (1) marginal means for mediation path a, the relation of the independent variable to the mediator; (2) marginal means for path b, the relation of the mediator to the outcome, across multiple trials; and (3) the between-trial level variance–covariance matrix based on a bivariate normal distribution. We present the statistical theory and an R computer program to combine regression coefficients from multiple trials to estimate a combined mediated effect and confidence interval under a random effects model. Values of coefficients a and b, along with their standard errors from each trial are the input for the method. This marginal likelihood based approach with Monte Carlo confidence intervals provides more accurate inference than the standard meta-analytic approach. We discuss computational issues, apply the method to two real-data examples and make recommendations for the use of the method in different settings.

Computational and technical approaches to improve the implementation of prevention programs

A potential new arena that could lead to major advances in implementation science involves the integration of computational and technologic approaches with behavioral and organizational sciences. While some attention has been given to the use of systems science methods, specifically agent-based modeling, social network analysis, and system dynamics, there is actually a much broader set of tools that could be used to improve adoption, assessment of fidelity, and sustainability. This panel provides a broad perspective and illustrates how such tools can aid implementation especially given the unique challenges in the prevention field.