Michael Vaden-Kiernan

Sample Selection in Randomized Experiments: A New Method Using Propensity Score Stratified Sampling.

Abstract Randomized experiments are often seen as the “gold standard” for causal research. Despite the fact that experiments use random assignment to treatment conditions, units are seldom selected into the experiment using probability sampling. Very little research on experimental design has focused on how to make generalizations to well-defined populations or on how units should be selected into an experiment to facilitate generalization. This article addresses the problem of sample selection in experiments by providing a method for selecting the sample so that the population and sample are similar in composition. The method begins by requiring that the inference population and eligibility criteria for the study are well defined before study recruitment begins. When the inference population and population of eligible units differs, the article provides a method for sample recruitment based on stratified selection on a propensity score. The article situates the problem within the example of how to select districts for two scale-up experiments currently in recruitment.

Site Selection in Experiments: An Assessment of Site Recruitment and Generalizability in Two Scale-Up Studies.

ABSTRACT Recently, statisticians have begun developing methods to improve the generalizability of results from large-scale experiments in education. This work has included the development of methods for improved site selection when random sampling is infeasible, including the use of stratification and targeted recruitment strategies. This article provides the next step in this literature—a template for assessing generalizability after a study is completed. In this template, first records from the recruitment process are analyzed, comparing differences between those who agreed to be in the study and those who did not. Second, the final sample is compared to the original inference population and different possible subsets, with the goal of determining where the results best generalize (and where they do not). Throughout, these methods are situated in the post hoc analysis of results from two scale-up studies. The article ends with a discussion of the use of these methods more generally when reporting results from randomized trials.

Findings From a Multiyear Scale-Up Effectiveness Trial of Open Court Reading

ABSTRACT This multiyear scale-up effectiveness study of Open Court Reading (OCR) involved approximately 4,500 students and more than 1,000 teachers per year in Grades K–5 from 49 elementary schools in seven districts across the country. Using a school-level cluster randomized trial design, we assessed the implementation and effectiveness of Open Court Reading over two years. Implementation study results demonstrated adequate to high levels of fidelity across the treatment schools. Intent-to-treat analyses revealed no statistically significant main effects on students’ reading performance in Year 1 and a small negative effect (d = – .09) in Year 2. There were positive impacts for particular subgroups, including kindergarten (d = .12) and Hispanic (d = .10) students in the first year. However, there were negative impacts for first grade (d = –.13), females (d = –.11), students who were not eligible for free or reduced-price lunch (d = –.19), and non-English language learners (d = –.10) in the second year of the study. Thus, relative to the “business-as-usual” reading curricula, no positive overall impacts of OCR and mixed impacts for student subgroups were found.