Mark W. Lipsey

The Primary Factors that Characterize Effective Interventions with Juvenile Offenders: A Meta-Analytic Overview

Abstract Previous meta-analyses have identified many effective interventions for reducing the recidivism of juvenile offenders and various program factors that are associated with the best outcomes. Most of that work has been focused on only one intervention area and thus has limited scope. Notable exceptions are two relatively comprehensive meta-analyses that have identified a small number of factors or principles that appear to characterize the most effective programs. This paper presents a new analysis of data from one of those meta-analyses designed to test a broader range of intervention factors in a manner that allows identification of both the general principles and the distinct intervention types associated with the greatest reductions in recidivism. Only three factors emerged as major correlates of program effectiveness: a “therapeutic” intervention philosophy, serving high risk offenders, and quality of implementation. With other variables statistically controlled, relatively few differences were found in the effectiveness of different types of therapeutic interventions.

A practitioner's guide to meta-analysis

ConclusionMeta-analysis can be a powerful and useful technique. In the short term, the results of meta-analysis can increase understanding and influence the future endeavors of researchers interested in a particular field. Over the long term, meta-analysis provides a vehicle by which large bodies of research can be integrated and focused on public policy issues.Administrative structures currently exist to foster the relationship between research findings and public policy. As one example, Saxe (1986) has described the activities of the Office of Technology Assessment (OTA) established by Congress to provide lawmakers with scientific information in such a way as to be useful for policy decision-making. OTA is commissioned by Congress to provide assessments of scientific findings, and meta-analyses are an ideal resource in this regard. Saxe (1986) notes “In almost all cases, the emphasis is on making sense of already collected data, thus literature reviews and methods for synthesizing research results are relied upon” (pp. 61–62).Whether meta-analysis actually plays an important role in public policy depends on many factors, not the least of which is the care with which meta-analyses are conducted. Hopefully, our comments here will help investigators conduct methodologically rigorous meta-analyses of research relevant to the field of community psychology.

Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions.

Abstract Two complementary approaches to developing empirical benchmarks for achievement effect sizes in educational interventions are explored. The first approach characterizes the natural developmental progress in achievement made by students from one year to the next as effect sizes. Data for seven nationally standardized achievement tests show large annual gains in the early elementary grades followed by gradually declining gains in later grades. A given intervention effect will therefore look quite different when compared to the annual progress for different grade levels. The second approach explores achievement gaps for policy-relevant subgroups of students or schools. Data from national- and district-level achievement tests show that, when represented as effect sizes, student gaps are relatively small for gender and much larger for economic disadvantage and race/ethnicity. For schools, the differences between weak schools and average schools are surprisingly modest when expressed as student-level effect sizes. A given intervention effect viewed in terms of its potential for closing one of these performance gaps will therefore look very different depending on which gap is considered.

Cognitive-Behavioral Programs for Offenders

A systematic review using meta-analytic techniques was conducted with 14 studies selected to provide the best evidence on the effectiveness of cognitive-behavioral programs for reducing the reoffense recidivism of criminal offenders. The results indicated that, overall, cognitive-behavioral programs are effective, and the best of them are capable of producing sizable reductions in recidivism. Many of the available studies, however, investigate research-oriented demonstration programs; the effects found for routine practical programs were notably smaller. Moreover, the research coverage of both juvenile and adult programs in institutional and noninstitutional settings is uneven and leaves troublesome gaps in the evidence.

Wilderness challenge programs for delinquent youth: a meta-analysis of outcome evaluations

Evaluations of wilderness challenge programs were meta-analyzed to assess the impact on delinquent behavior. The overall mean effect size for delinquency outcomes was 0.18 (N=22), equivalent to a recidivism rate of 29% for program participants vs 37% for comparison subjects. Program length was not related to outcome among short-term programs (up to 6 weeks) but extended programs (over 10 weeks) showed smaller effects overall. However, the most influential program characteristics were the intensity of the physical activities and whether the program included a distinct therapeutic component. Programs involving relatively intense activities or with therapeutic enhancements produced the greatest reductions in delinquent behavior.

Those Confounded Moderators in Meta-Analysis: Good, Bad, and Ugly

One of the more revealing aspects of meta-analysis is the relationship between effect sizes and moderator variables representing differences among studies in their methods, samples, and interventions. However, interesting moderator variables are generally related to each other as well as to effect sizes. This confounding among moderator variables can make the results of any analysis focusing on a single moderator variable misleading. Despite the ambiguity of such results, this form of analysis is common in meta-analytic studies of intervention. The hazards and complexities of investigating and interpreting confounded moderator variables are illustrated in this article by the authors examining the difference in effect sizes associated with randomized versus nonrandomized designs in a large meta-analytic database of delinquency intervention studies.

Can Intervention Rehabilitate Serious Delinquents

Much contemporary discussion of the future of the juvenile court revolves around the balance between rehabilitation and punishment, especially with regard to the most serious juvenile offenders. Political forces increasingly press in the direction of punitive approaches, while the historical orientation of the court has been rehabilitative. This article addresses the question of whether rehabilitative treatment can be effective for the most serious offenders. Meta-analysis techniques were used to synthesize the large body of empirical research on the effects of rehabilitative programs in community and institutional settings. The results show that well-designed rehabilitative strategies do reduce recidivism for such offenders and cannot be dismissed on the grounds that they are ineffective.

The comparative effectiveness of outpatient treatment for adolescent substance abuse: A meta-analysis

Meta-analysis was used to synthesize research on the effects of outpatient treatment on substance use outcomes for adolescents with substance use disorders. An extensive literature search located 45 eligible experimental or quasi-experimental studies reporting 73 treatment-comparison group pairs, with many of the comparison groups also receiving some treatment. The first analysis examined 250 effect sizes for the substance use outcomes of adolescents receiving different types of treatment relative to the respective comparison groups. As a category, family therapy programs were found to be more effective than their comparison conditions, whereas no treatment programs were less effective. However, not all treatment types were compared with each other in the available research, making it difficult to assess the comparative effectiveness of the different treatments. To provide a more differentiated picture of the relative improvement in substance use outcomes for different treatments, a second analysis examined 311 pre-post effect sizes measuring changes in substance use for adolescents in the separate treatment and comparison arms of the studies. The adolescents in almost all types of treatment showed reductions in substance use. The greatest improvements were found for family therapy and mixed and group counseling. Longer treatment duration was associated with smaller improvements, but other treatment characteristics and participant characteristics had little relationship to the pre-post changes in substance use. Based on these findings family therapy is the treatment with the strongest evidence of comparative effectiveness, although most types of treatment appear to be beneficial in helping adolescents reduce their substance use.

Brief alcohol interventions for adolescents and young adults: a systematic review and meta-analysis

This study reports findings from a meta-analysis summarizing the effectiveness of brief alcohol interventions for adolescents (age 11-18) and young adults (age 19-30). We identified 185 eligible study samples using a comprehensive literature search and synthesized findings using random-effects meta-analyses with robust standard errors. Overall, brief alcohol interventions led to significant reductions in alcohol consumption and alcohol-related problems among adolescents (g = 0.27 and g = 0.19) and young adults (g = 0.17 and g = 0.11). These effects persisted for up to 1 year after intervention and did not vary across participant demographics, intervention length, or intervention format. However, certain intervention modalities (e.g., motivational interviewing) and components (e.g., decisional balance, goal-setting exercises) were associated with larger effects. We conclude that brief alcohol interventions yield beneficial effects on alcohol-related outcomes for adolescents and young adults that are modest but potentially worthwhile given their brevity and low cost.

Driving toward theory in program evaluation: More models to choose from

Abstract The advantages of a theory-based approach to evaluation are numerous despite the relative lack of explicit theorizing about program process typical of contemporary evaluation. To be useful program theory does not have to come in the form of sophisticated structural equation modeling. More differentiated notions of the forms program theory might take and how it might be developed in practical context may help extend the range of applicability of the Chen and Rossi “theory-driven” approach. To this end, four different theory forms are described and analyzed: causal modeling, the “basic two-step,” stage-state theory, and substantive models. In addition, three practical approaches to theory development are discussed: building on prior theory and research, conducting exploratory research, and extracting the implicit program theory from program personnel.