Terri D. Pigott

Advances in meta-analysis

Rationale and overview of the book.- Complexities of primary research and meta-analysis.- Fixed, random, and mixed models.- Correlations.- Effect sizes.- Proportion/odds ratio data.- Dependence in primary research and meta-analysis.- Correlations.- Effect sizes (group comparisons and change measures).- Proportion/odds ratio data.- Bayesian approaches.- Missing data.- Combining across designs.- Qualitative studies in synthesis.- Quality assessments.- Studies on quality weighting.- Summary/Conclusions.

Identifying, documenting, and examining heterogeneity in systematic reviews of complex interventions.

This article describes approaches for planning, dealing, and analyzing heterogeneity in a systematic review of complex interventions. Approaches aim to generate a priori hypotheses of the mechanism of action of a complex intervention to identify the key variables that might contribute to variation among studies and guide statistical analysis. In addition to characteristics related to the population, intervention, and outcomes, we describe study-related variables, such as the way the interventions have been implemented and the context and conduct of studies. These approaches will guide reviewers planning a meta-analysis and provide a rationale for not meta-analyzing data if there is too much variability. Potential difficulties in applying meta-analytical techniques to examine statistical association among study results and sources of potential heterogeneity are described; these include the selection of a fixed or random-effects model, the risk of multiple testing and confounding when studies include different aspects of a complex intervention or different subsamples of the intended participant pool.

Indicated Truancy Interventions for Chronic Truant Students A Campbell Systematic Review

Objective: Examine the effects of indicated interventions on attendance with chronic truant students. Method: Systematic review and meta-analytic methods, following Campbell Collaboration guidelines, were utilized. A comprehensive search identified 5 randomized and 11 quasi-experimental studies. Results: The mean effect on attendance outcomes was moderate, positive, and significant, g = .46, 95% confidence interval [.30, .62], p < .05, translating into an improvement in attendance by an average of 4.69 days; however, significant heterogeneity was observed. Moderator analyses found no significant differences in mean effects between studies on variables tested. Discussion: Chronic truant students benefit from interventions targeting attendance behaviors; however, no program stood out as being more effective than others. Mean rates of absenteeism at posttest in most studies remained above acceptable levels, indicating a need for more effective interventions. The paucity of research, gaps, and deficiencies affirm the need for strengthening the evidence base. Recommendations for practice, policy, and research are discussed.

A Meta-Analysis of School-Based Interventions Aimed to Prevent or Reduce Violence in Teen Dating Relationships

The incidence of violence in dating relationships has a significant impact on young people, including decreased mental and physical health. This review is the first to provide a quantitative synthesis of empirical evaluations of school-based programs implemented in middle and high schools that sought to prevent or reduce incidents of dating violence. After a systematic search and screening procedure, a meta-analysis of 23 studies was used to examine the effects of school-based programs. Results indicated school-based programs influence dating violence knowledge ( g ¯ = 0 . 22 , 95% confidence interval [0.05, 0.39]) and attitudes ( g ¯ = 0 . 14 , 95% confidence interval [0.10, 0.19]); however, to date, the results for dating violence perpetration and victimization indicate programs are not affecting these behaviors to a significant extent. The results of this review are encouraging, but they also highlight the need for modifications to dating violence prevention programs including the incorporation of skill-building components and a need to address the role of bystanders.

The Use of Meta-Analytic Statistical Significance Testing.

Meta-analysis multiplicity, the concept of conducting multiple tests of statistical significance within one review, is an underdeveloped literature. We address this issue by considering how Type I errors can impact meta-analytic results, suggest how statistical power may be affected through the use of multiplicity corrections, and propose how meta-analysts should analyze multiple tests of statistical significance. The context for this study is a meta-review of meta-analyses published in two leading review journals in education and psychology. Our review of 130 meta-analyses revealed a strong reliance on statistical significance testing without consideration of Type I errors or the use of multiplicity corrections. In order to provide valid conclusions, meta-analysts must consider these issues prior to conducting the review.

Treatment for School Refusal among Children and Adolescents: A Systematic Review and Meta-Analysis.

Objective: School refusal is a psychosocial problem associated with adverse short- and long-term consequences for children and adolescents. The authors conducted a systematic review and meta-analysis to examine the effects of psychosocial treatments for children and adolescents with school refusal. Method: A comprehensive search process was used to find eligible randomized controlled trials and quasi-experimental studies assessing the effects of psychosocial treatments on anxiety or attendance outcomes. Data were quantitatively synthesized using meta-analytic methods. Results: Eight studies including 435 children and adolescents with school refusal were included in this review. Significant effects were found for attendance but not for anxiety. Conclusions: Evidence indicates that improvements in school attendance occur for children and adolescents with school refusal who receive psychosocial treatment. The lack of evidence of short-term effects on anxiety points to the need for long-term follow-up studies to determine whether increased attendance ultimately leads to reduced anxiety.

AHRQ series on complex intervention systematic reviews—paper 6: PRISMA-CI extension statement and checklist

BACKGROUND Complex interventions are widely used in health systems, public health, education, and communities and are increasingly the subject of systematic reviews. Oversimplification and inconsistencies in reporting about complex interventions can limit the usability of review findings. RATIONALE Although guidance exists to ensure that reports of individual studies and systematic reviews adhere to accepted scientific standards, their design-specific focus leaves important reporting gaps relative to complex interventions in health care. This paper provides a stand-alone extension to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting tool for complex interventions-PRISMA-CI-to help authors, publishers, and readers understand and apply to systematic reviews of complex interventions. DISCUSSION PRISMA-CI development followed the Enhancing the QUAlity and Transparency Of health Research Network guidance for extensions and focused on adding or modifying only essential items that are truly unique to complex interventions and are not covered by broader interpretation of current PRISMA guidance. PRISMA-CI provides an important structure and guidance for systematic reviews and meta-analyses for the highly prevalent and dynamic field of complex interventions.

AHRQ series on complex intervention systematic reviews-paper 7: PRISMA-CI elaboration and explanation

BACKGROUND Complex interventions are widely used in health care, public health, education, criminology, social work, business, and welfare. They have increasingly become the subject of systematic reviews and are challenging to effectively report. The Complex Interventions Methods Workgroup developed an extension to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Complex Interventions (PRISMA-CI). RATIONALE Following the EQUATOR Network guidance for Preferred Reporting Items for Systematic Reviews and Meta-Analysis extensions, this Explanation and Elaboration (EE) document accompanies the PRISMA-CI checklist to promote consistency in reporting of systematic reviews of complex interventions. DISCUSSIONS The EE document explains the meaning and rationale for each unique PRISMA-CI checklist item and provides examples to assist systematic review authors in operationalizing PRISMA-CI guidance. The Complex Interventions Workgroup developed PRISMA-CI as an important start toward increased consistency in reporting of systematic reviews of complex interventions. Because the field is rapidly expanding, the Complex Interventions Methods Workgroup plans to re-evaluate periodically for the need to add increasing specificity and examples as the field matures.

AHRQ series on complex intervention systematic reviews—paper 5: advanced analytic methods

BACKGROUND AND OBJECTIVE Advanced analytic methods for synthesizing evidence about complex interventions continue to be developed. In this paper, we emphasize that the specific research question posed in the review should be used as a guide for choosing the appropriate analytic method. METHODS We present advanced analytic approaches that address four common questions that guide reviews of complex interventions: (1) How effective is the intervention? (2) For whom does the intervention work and in what contexts? (3) What happens when the intervention is implemented? and (4) What decisions are possible given the results of the synthesis? CONCLUSION The analytic approaches presented in this paper are particularly useful when each primary study differs in components, mechanisms of action, context, implementation, timing, and many other domains.

The Question of School Resources and Student Achievement: A History and Reconsideration

One question posed continually over the past century of education research is to what extent school resources affect student outcomes. From the turn of the century to the present, a diverse set of actors, including politicians, physicians, and researchers from a number of disciplines, have studied whether and how money that is provided for schools translates into increased student achievement. The authors discuss the historical origins of the question of whether school resources relate to student achievement, and report the results of a meta-analysis of studies examining that relationship. They find that policymakers, researchers, and other stakeholders have addressed this question using diverse strategies. The way the question is asked, and the methods used to answer it, is shaped by history, as well by the scholarly, social, and political concerns of any given time. The diversity of methods has resulted in a body of literature too diverse and too inconsistent to yield reliable inferences through meta-analysis. The authors suggest that a collaborative approach addressing the question from a variety of disciplinary and practice perspectives may lead to more effective interventions to meet the needs of all students.