Guanglei Hong

Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics

Grade retention has been controversial for many years, and current calls to end social promotion have lent new urgency to this issue. On the one hand, a policy of retaining in grade those students making slow progress might facilitate instruction by making classrooms more homogeneous academically. On the other hand, grade retention might harm high-risk students by limiting their learning opportunities. Analyzing data from the US Early Childhood Longitudinal Study Kindergarten cohort with the technique of multilevel propensity score stratification, we find no evidence that a policy of grade retention in kindergarten improves average achievement in mathematics or reading. Nor do we find evidence that the policy benefits children who would be promoted under the policy. However, the evidence does suggest that children who are retained learn less than they would have had they instead been promoted. The negative effect of grade retention on those retained has little influence on the overall mean achievement of children attending schools with a retention policy because the fraction of children retained in those schools is quite small. Nevertheless, the effect of retention on the retainees is considerably large.

Evaluating Kindergarten Retention Policy: A Case Study of Causal Inference for Multilevel Observational Data

This article considers the policy of retaining low-achieving children in kindergarten rather than promoting them to first grade. Under the stable unit treatment value assumption (SUTVA) as articulated by Rubin, each child at risk of retention has two potential outcomes: Y(1) if retained and Y(0) if promoted. But SUTVA is questionable, because a childs potential outcomes will plausibly depend on which school that child attends and also on treatment assignments of other children. We develop a causal model that allows school assignment and peer treatments to affect potential outcomes. We impose an identifying assumption that peer effects can be summarized through a scalar function of the vector of treatment assignments in a school. Using a large, nationally representative sample, we then estimate (1) the effect of being retained in kindergarten rather than being promoted to the first grade in schools having a low retention rate, (2) the retention effect in schools having a high retention rate, and (3) the effect of being promoted in a low-retention school as compared to being promoted in a high-retention school. This third effect is not definable under SUTVA. We use multilevel propensity score stratification to approximate a two-stage experiment. At the first stage, intact schools are blocked on covariates and then, within blocks, randomly assigned to a policy of retaining comparatively more or fewer children in kindergarten. At the second stage, “at-risk” students within schools are blocked on covariates and then assigned at random to be retained. We find evidence that retainees learned less on average than did similar children who were promoted, a result found in both high-retention and low-retention schools. We do not detect a peer treatment effect on low-risk students.

Marginal Mean Weighting Through Stratification: Adjustment for Selection Bias in Multilevel Data

Defining causal effects as comparisons between marginal population means, this article introduces marginal mean weighting through stratification (MMW-S) to adjust for selection bias in multilevel educational data. The article formally shows the inherent connections among the MMW-S method, propensity score stratification, and inverse-probability-of-treatment weighting (IPTW). Both MMW-S and IPTW are suitable for evaluating multiple concurrent treatments, and hence have broader applications than matching, stratification, or covariance adjustment for the propensity score. Furthermore, mathematical consideration and a series of simulations reveal that the MMW-S method has incorporated some important strengths of the propensity score stratification method, which generally enhance the robustness of MMW-S estimates in comparison with IPTW estimates. To illustrate, the author applies the MMW-S method to evaluations of within-class homogeneous grouping in early elementary reading instruction.

Causal Inference for Time-Varying Instructional Treatments.

The authors propose a strategy for studying the effects of time-varying instructional treatments on repeatedly observed student achievement. This approach responds to three challenges: (a) The yearly reallocation of students to classrooms and teachers creates a complex structure of dependence among responses; (b) a child’s learning outcome under a certain treatment may depend on the treatment assignment of other children, the skill of the teacher, and the classmates and teachers encountered in the past years; and (c) time-varying confounding poses special problems of endogeneity. The authors address these challenges by modifying the stable unit treatment value assumption to identify potential outcomes and causal effects and by integrating inverse probability of treatment weighting into a four-way value-added hierarchical model with pseudolikelihood estimation. Using data from the Longitudinal Analysis of School Change and Performance, the authors apply these methods to study the impact of “intensive math instruction” in Grades 4 and 5.

Early-Grade Retention and Children’s Reading and Math Learning in Elementary Years

Many schools have adopted early-grade retention as an intervention strategy for children displaying academic or behavioral problems. Previous analyses of the Early Childhood Longitudinal Study Kindergarten Cohort data have found evidence of negative effects of kindergarten retention on academic learning during the repeated year. Will kindergarten retainees recover their lost ground and excel in the long run? What are the effects of first grade retention? According to the analytic results of this study, the negative effects of kindergarten retention on retainees’ reading and math outcomes at the end of the treatment year substantially fade by fifth grade. Meanwhile, first grade retention shows negative effects that stay almost constant from 1 year after treatment to 3 years later. In general, we find no evidence that early-grade retention brings benefits to the retainees’ reading and math learning toward the end of the elementary years.

Reading Instruction Time and Homogeneous Grouping in Kindergarten: An Application of Marginal Mean Weighting through Stratification.

A kindergartner’s opportunities to develop reading and language arts skills are constrained by the amount of time allocated to reading instruction. In the meantime, the student’s engagement in learning tasks may increase if the instruction has been adapted to his or her prior ability through homogeneous grouping. This study investigates whether the grouping effects on kindergartners’ reading growth depend on the amount of reading instruction time and the intensity of grouping. To answer the study’s research questions requires causal inferences about concurrent multivalued instructional treatments. The authors develop a procedure of applying the method of marginal mean weighting through stratification to multilevel educational data. Results from the Early Childhood Longitudinal Study Kindergarten cohort data set lend support to the theoretical hypothesis that when teachers allocate a substantial amount of time to reading instruction, homogeneous grouping helps kindergartners to gain more in reading. The authors find no effect of homogeneous grouping when the total amount of reading time is limited. They also find that the benefit of increasing reading instruction time becomes evident only if kindergarten teachers adapt instruction through homogeneous grouping.

Differential Effects of Literacy Instruction Time and Homogeneous Ability Grouping in Kindergarten Classrooms Who Will Benefit? Who Will Suffer?

This study challenges the belief that homogeneous ability grouping benefits high-ability students in cognitive and social-emotional development at the expense of their low-ability peers. From a developmental point of view, the authors hypothesize that homogeneous grouping may improve the learning behaviors and may benefit the literacy learning of kindergartners at all ability levels through adaptive instruction under adequate instructional time. The benefits are expected to be more evident for medium- and low-ability children than for high-ability children. However, when instructional time is limited, low-ability children may suffer from high-intensity grouping, defined as grouping taking up a large proportion of instructional time. The authors also examine whether low-ability kindergartners develop lower self-esteem as a result of homogeneous grouping. Analyzing Early Childhood Longitudinal Study kindergarten cohort data, the authors find no overall advantage of homogeneous grouping for high-ability students. For medium-ability students’ literacy growth, homogeneous grouping appears to be optimal when teachers spend more than 1 hour per day on literacy instruction; high-intensity grouping shows additional advantage for improving these students’ general learning behaviors. For low-ability kindergartners, homogeneous grouping with ample instruction time seems to improve their general learning behaviors, whereas low-intensity grouping with ample instruction time seems to reduce internalizing problem behaviors. Yet for low-ability students’ literacy growth, a detrimental effect of high-intensity grouping is found when instructional time is limited. These findings contradict results from past research and have important implications for educational theories and practice.

Weighting Methods for Assessing Policy Effects Mediated by Peer Change.

Abstract The conventional approaches to mediation analysis such as path analysis and structural equation modeling typically involve specifying two structural models, one for the mediator and the other for the outcome. We employ an alternative approach that avoids some strong identification assumptions invoked by the conventional approaches. By applying a new weighting procedure to the observed data, we estimate the average potential outcome if the entire population were treated, the average potential outcome if the entire population were untreated, and the average potential outcome if the entire population were treated and if every individual units mediator value would counterfactually remain at the same level as it would be when untreated. The estimated differences among these average potential outcomes provide estimates of the total effect, the natural direct effect, and the natural indirect effect. Applying this approach to multilevel educational data, we evaluate the total effect of the algebra-for-all policy in the Chicago Public Schools by comparing the math achievement of two ninth-grade cohorts. We further investigate whether the policy effect was mediated by the policy-induced change in class peer ability. Combining weighting with prognostic score-based difference-in-differences adjustment enables us to reduce both measured and unmeasured confounding.

Ratio-of-Mediator-Probability Weighting for Causal Mediation Analysis in the Presence of Treatment-by-Mediator Interaction.

Conventional methods for mediation analysis generate biased results when the mediator–outcome relationship depends on the treatment condition. This article shows how the ratio-of-mediator-probability weighting (RMPW) method can be used to decompose total effects into natural direct and indirect effects in the presence of treatment-by-mediator interactions. The indirect effect can be further decomposed into a pure indirect effect and a natural treatment-by-mediator interaction effect. Similar to other techniques for causal mediation analysis, RMPW generates causally valid results when the sequential ignorability assumptions hold. Yet unlike the model-based alternatives, including path analysis, structural equation modeling, and their latest extensions, RMPW requires relatively few assumptions about the distribution of the outcome, the distribution of the mediator, and the functional form of the outcome model. Correct specification of the propensity score models for the mediator remains crucial when parametric RMPW is applied. This article gives an intuitive explanation of the RMPW rationale, a mathematical proof, and simulation results for the parametric and nonparametric RMPW procedures. We apply the technique to identifying whether employment mediated the relationship between an experimental welfare-to-work program and maternal depression. A detailed delineation of the analytic procedures is accompanied by online Stata code as well as a stand-alone RMPW software program to facilitate users’ analytic decision making.

Heterogeneous Agents, Social Interactions, and Causal Inference

Most causal analyses in the social sciences depend on the assumption that each participant possesses a single potential outcome under each possible treatment assignment. Rubin (J Am Stat Assoc 81:961–962, 1986) labeled this the “stable unit treatment value assumption” (SUTVA). Under SUTVA, the individual-specific impact of a treatment depends neither on the mechanism by which the treatment is assigned nor on the treatment assignments of other individuals. However, in the social world, heterogeneous agents enact most interventions of interest: Teachers implement curricula, psychologists enact family therapy, and precinct captains supervise community policing. Moreover, the potential outcomes of one participant will often depend on the treatment assignment of other participants (classmates, family members, neighbors). This chapter presents a model that relaxes the conventional SUTVA by incorporating agents and social interactions. We define a treatment setting for an individual participant as a local environment constituted by a set of agents and participants along with their treatment assignments. Our model assigns a single potential outcome to each participant in each of such treatment settings. In a cluster-randomized trial, if no interference exists between clusters and if cluster composition remains intact, the treatment setting is fixed for all participants in a cluster and SUTVA becomes reasonable. However, when participants are assigned to treatments within clusters, we need a model for within-cluster interference among participants. When clusters are spatially contiguous, social interactions generate interference between clusters. We also incorporate new models for interference as a part of the meditation mechanism. In general, when SUTVA is relaxed, new causal questions come to light. We illustrate these ideas using studies of grade retention in elementary school, community policing in cities, school-wide interventions for behavioral improvement, and system-wide curricular changes for promoting math learning.