Christopher Rhoads

The Implications of “Contamination” for Experimental Design in Education

Experimental designs that randomly assign entire clusters of individuals (e.g., schools and classrooms) to treatments are frequently advocated as a way of guarding against contamination of the estimated average causal effect of treatment. However, in the absence of contamination, experimental designs that randomly assign intact clusters to treatments are less efficient than designs that randomly assign individual units within clusters. The current article considers the case of contamination processes that tend to make experimental and control subjects appear more similar than they truly are. The article demonstrates that, for most parameter values of practical interest, the statistical power of a randomized block (RB) design remains higher than the power of a cluster randomized (CR) design even when contamination causes the effect size to decrease by as much as 10%–60%. Furthermore, from the standpoint of point estimation, RB designs will tend to be preferred when true effect sizes are small and when the number of clusters in the experiment is not too large, but CR designs will tend to be preferred when true effect sizes are large or when the number of clusters in the experiment is large.

Use of MP3 players to increase asthma knowledge in inner-city African-American adolescents.

Background: Low-income African-American adolescents suffer a disproportionate burden of asthma morbidity.Purpose: To evaluate the ability of our intervention, the Adolescents’ Disease Empowerment and Persistency Technology (ADEPT) for asthma, to increase asthma knowledge in our target population.Methods: This was a 14-week (2-week run-in and 12-week treatment) randomized, double-blind, placebo-controlled pilot study in which 28 inner-city African-American adolescents with asthma, between 10 and 18 years of age, were randomized to receive (1) celebrity asthma messages (experimental group), or (2) general health messages (control group) between music tracks on an MP3 player. The asthma messages were recorded by famous athletes, musicians, and other celebrities popular among this group of teenagers. Asthma knowledge, assessed by the ZAP Asthma Knowledge instrament, was collected pre- and post-intervention.Results: Mean improvement in ZAP score was significantly higher in the experimental group (8.1%, SD 7.2%) than the control group (0.4%, SD 7.2%) (p = 0.05).Conclusion: These findings suggest that this may be an innovative and promising new approach to improving asthma outcomes in this difficult-to-reach population.

Correcting an analysis of variance for clustering

A great deal of educational and social data arises from cluster sampling designs where clusters involve schools, classrooms, or communities. A mistake that is sometimes encountered in the analysis of such data is to ignore the effect of clustering and analyse the data as if it were based on a simple random sample. This typically leads to an overstatement of the precision of results and too liberal conclusions about precision and statistical significance of mean differences. This paper gives simple corrections to the test statistics that would be computed in an analysis of variance if clustering were (incorrectly) ignored. The corrections are multiplicative factors depending on the total sample size, the cluster size, and the intraclass correlation structure. For example, the corrected F statistic has Fishers F distribution with reduced degrees of freedom. The corrected statistic reduces to the F statistic computed by ignoring clustering when the intraclass correlations are zero. It reduces to the F statistic computed using cluster means when the intraclass correlations are unity, and it is in between otherwise. A similar adjustment to the usual statistic for testing a linear contrast among group means is described.

Statistical power analysis

Statistical power analysis involves determining statistical power of a design given a statistical significance level, sample size, and effect size, or determining the sample size necessary to obtain a desired level of power for a specified effect size and significance level. Power analysis is described for research designs with and without covariates. It is also described for research designs that involve multilevel sampling and assignment of intact groups to treatments. A table is provided to assist in the computation of statistical power for a given effect size and sample size or required sample size for a given effect size and desired power.

Under What Circumstances Does External Knowledge About the Correlation Structure Improve Power in Cluster Randomized Designs

Abstract Recent publications have drawn attention to the idea of utilizing prior information about the correlation structure to improve statistical power in cluster randomized experiments. Because power in cluster randomized designs is a function of many different parameters, it has been difficult for applied researchers to discern a simple rule explaining when prior correlation information will substantially improve power. This article provides bounds on the maximum possible improvement in power as a function of a single parameter, the number of clusters at the highest level of a multilevel experiment. The maximum improvement in power is less than 0.05 unless the number of clusters at the highest level is less than 20. Thus, the utility of using prior correlation information is limited to experiments with very small cluster-level sample sizes. Situations where small cluster-level sample sizes could still result in experiments with good statistical power are discussed, as is the relative utility of prior information about intracluster correlations as compared with covariate information that can explain cluster level variability in the outcome.

Promoting students’ science literacy skills through a simulation of international negotiations: The GlobalEd 2 Project

Abstract Problem-based learning (PBL) is an instructional design approach for promoting student learning in context-rich settings. GlobalEd 2 (GE2) is PBL intervention that combines face-to-face and online environments into a 12-week simulation of international negotiations of science delegates on global science issues. Although GE2 focuses on science, it is implemented in a social studies classroom. This manuscript describes the GE2 environment and evaluates its impact on middle school students’ scientific literacy compared to a comparison group receiving normal educational practice. Hierarchical Linear Modeling (HLM) analyses on GE2 and comparison groups demonstrates the significant positive impact of GE2 on two measures of scientific literacy (Socio-scientific Literacy and Scientific Inquiry), among middle-grade students from two states. Implications regarding instructional practice and future research are discussed.

Challenges to Using the Regression Discontinuity Design in Educational Evaluations Lessons From the Transition to Algebra Study

Interest in the regression discontinuity (RD) design as an alternative to randomized control trials (RCTs) has grown in recent years. There is little practical guidance, however, on conditions that...Interest in the regression discontinuity (RD) design as an alternative to randomized control trials (RCTs) has grown in recent years. There is little practical guidance, however, on conditions that would lead to a successful RD evaluation or the utility of studies with underpowered RD designs. This article describes the use of RD design to evaluate the impact of a supplemental algebra-readiness curriculum, Transition to Algebra, on students’ mathematics outcomes. Lessons learned highlight the need for evaluators to understand important data requirements for strong RD evaluation studies, the need to collaborate with informed and committed partners to ensure successful RD design implementation, the value of embedding an RCT within an RD design whenever possible, and the need for caution when contemplating an RD design with a small sample. Underpowered RD studies—unlike underpowered RCTs—may not produce useful evaluation results, particularly if other RD data requirements are not met.

Coherent Power Analysis in Multilevel Studies Using Parameters From Surveys

Researchers designing multisite and cluster randomized trials of educational interventions will usually conduct a power analysis in the planning stage of the study. To conduct the power analysis, researchers often use estimates of intracluster correlation coefficients and effect sizes derived from an analysis of survey data. When there is heterogeneity in treatment effects across the clusters in the study, these parameters will need to be adjusted to produce an accurate power analysis for a hierarchical trial design. The relevant adjustment factors are derived and presented in the current article. The adjustment factors depend upon the covariance between treatment effects and cluster-specific average values of the outcome variable, illustrating the need for better information about this parameter. The results in the article also facilitate understanding of the relative power of multisite and cluster randomized studies conducted on the same population by showing how the parameters necessary to compute power in the two types of designs are related. This is accomplished by relating parameters defined by linear mixed model specifications to parameters defined in terms of potential outcomes.

The Implications of Contamination for Educational Experiments With Two Levels of Nesting

ABSTRACT Experimental evaluations that involve the educational system usually involve a hierarchical structure (students are nested within classrooms that are nested within schools, etc.). Concerns about contamination, where research subjects receive certain features of an intervention intended for subjects in a different experimental group, have often led researchers to randomize units at a higher level of the educational hierarchy. Existing work on two-level designs suggests that situations where contamination should lead to randomization at a higher level are likely to be rare. This article extends these results to the case of three-level designs. In order to understand the implications of mathematical results, existing information about the size of intracluster correlation coefficients (ICCs) in educational studies with three levels and about the extent of treatment effect heterogeneity across schools is discussed. Better empirical estimates of ICCs, treatment effect heterogeneity, and plausible contamination values are necessary to make full use of the results in this article. However, it seems likely that situations where contamination should lead to randomization at a higher level in three-level designs are rare.

Optimal Design for Two-Level Random Assignment and Regression Discontinuity Studies

An important concern when planning research studies is to obtain maximum precision of an estimate of a treatment effect given a budget constraint. When research designs have a multilevel or hierarchical structure changes in sample size at different levels of the design will impact precision differently. Furthermore, there will typically be differential costs of enrolling additional units at different levels of the hierarchy. The optimal design problem in multilevel research studies involves determining the optimal sample size at each level of the design given specified design parameters and a specified marginal cost of recruitment at each level. The current work extends existing results by considering optimal design for (a) unbalanced random assignment designs and (b) regression discontinuity designs.