Takako Nomi

College Preparatory Curriculum for All: Academic Consequences of Requiring Algebra and English I for Ninth Graders in Chicago

There is a national movement to universalize the high school curriculum so that all students graduate prepared for college. The present work evaluates a policy in Chicago that ended remedial classes and mandated college preparatory course work for all students. Based on an interrupted time-series cohort design with multiple comparisons, this study found that the policy reduced inequities in ninth grade course work by entering ability, race/ethnicity, and special education status. Although more students completed ninth grade with credits in algebra and English I, failure rates increased, grades slightly declined, test scores did not improve, and students were no more likely to enter college. In sum, few benefits resulted from universalizing college preparatory course work among freshmen, but dropout rates did not increase. Possible explanations are discussed.

The Unintended Consequences of an Algebra-for-All Policy on High-Skill Students Effects on Instructional Organization and Students’ Academic Outcomes

In 1997, Chicago implemented a policy that required algebra for all ninth-grade students, eliminating all remedial coursework. This policy increased opportunities to take algebra for low-skill students who had previously enrolled in remedial math. However, little is known about how schools respond to the policy in terms of organizing math classrooms to accommodate curricular changes. The policy unintentionally affected high-skill students who were not targeted by the policy—those who would enroll in algebra in its absence. Using an interrupted time-series design combined with within-cohort comparisons, this study shows that schools created more mixed-ability classrooms when eliminating remedial math classes, and peer skill levels declined for high-skill students. Consequently, their test scores also declined.

The Effects of Within-Class Ability Grouping on Academic Achievement in Early Elementary Years

Abstract By incorporating two theoretical frameworks this study examines how school characteristics shape first-grade reading ability-grouping practices, and how this, in turn, affects students’ reading achievement. The author uses the data from the Early Childhood Longitudinal Study and applies the propensity-score method to examine whether first-grade ability grouping improves student achievement, whether ability grouping increases achievement inequalities, and whether its effects vary by student initial abilities and/or school contexts. Findings support an argument that ability grouping is an organizational response to problems of diversity in the student body. Schools that use ability grouping are likely to have heterogeneous ability compositions. They are also public, low-performing, low socioeconomic status, and high-minority schools. In these schools, ability grouping has no effects or negative effects, particularly for low-ability students. In contrast, ability grouping may improve achievement for all students in schools with advantageous characteristics, mostly private schools, and may reduce achievement inequalities, because low-ability students benefit the most from this practice.

Sorting and Supporting: Why Double-Dose Algebra Led to Better Test Scores but More Course Failures

In 2003, Chicago schools required students entering ninth grade with below-average math scores to take two periods of algebra. This led to higher test scores for students with both above- and below-average skills, yet failure rates increased for above-average students. We examine the mechanisms behind these surprising results. Sorting by incoming skills benefitted the test scores of high-skill students partially through higher demands and fewer disruptive peers. But more students failed because their skills were low relative to classroom peers. For below-average students, improvements in pedagogy and more time for learning offset problems associated with low-skill classrooms. In some cases, classrooms were not sorted, but below-average students took an extra support class simultaneously. Test scores also improved in such classes.

Statistical Analysis for Multisite Trials Using Instrumental Variables With Random Coefficients

Abstract Multisite trials can clarify the average impact of a new program and the heterogeneity of impacts across sites. Unfortunately, in many applications, compliance with treatment assignment is imperfect. For these applications, we propose an instrumental variable (IV) model with person-specific and site-specific random coefficients. Site-specific IV coefficients can be interpreted as site-average effects of program participation or as site-average effects of participation for the compliers. The validity of these interpretations depends on the analysts assumptions. Within the framework of a two-level hierarchical linear model, we propose three ways to estimate the mean and variance of these site-specific effects: (a) estimate the impact of program participation and its standard error in each site, then combine these site-specific statistics to estimate the mean and variance of the true site effects; (b) estimate the mean and variance of the effect of treatment assignment on the outcome and the mean and variance of the effect of treatment assignment on program participation; then combine these results to obtain estimates of the mean and variance of the effect of program participation; and (c) use Site by Treatment interactions as multiple instruments. If we assume the IV coefficients to be homogenous across sites, the three approaches are equivalent to variants of familiar two-stage least squares estimates with site fixed effects. Estimates based on our model are valid under a weaker assumption: that site-average levels of compliance are independent of site-average effects of program participation. To illustrate our approach, we evaluate a district-wide policy intended to increase math instructional time in math for low-achieving students. Finally, we discuss how Method (c) can be extended to incorporate multiple mediators.

Weighting Methods for Assessing Policy Effects Mediated by Peer Change.

Abstract The conventional approaches to mediation analysis such as path analysis and structural equation modeling typically involve specifying two structural models, one for the mediator and the other for the outcome. We employ an alternative approach that avoids some strong identification assumptions invoked by the conventional approaches. By applying a new weighting procedure to the observed data, we estimate the average potential outcome if the entire population were treated, the average potential outcome if the entire population were untreated, and the average potential outcome if the entire population were treated and if every individual units mediator value would counterfactually remain at the same level as it would be when untreated. The estimated differences among these average potential outcomes provide estimates of the total effect, the natural direct effect, and the natural indirect effect. Applying this approach to multilevel educational data, we evaluate the total effect of the algebra-for-all policy in the Chicago Public Schools by comparing the math achievement of two ninth-grade cohorts. We further investigate whether the policy effect was mediated by the policy-induced change in class peer ability. Combining weighting with prognostic score-based difference-in-differences adjustment enables us to reduce both measured and unmeasured confounding.

Making a Success of "Algebra for All": The Impact of Extended Instructional Time and Classroom Peer Skill in Chicago.

In 2003, Chicago launched “Double-Dose Algebra,” requiring students with pretest scores below the national median to take two periods of math–algebra and supplemental coursework. In many schools, assignment to Double Dose changed the peer composition of the algebra classroom. Using school-specific instrumental variables within a regression-discontinuity design (RDD), we find that attending a lower skill classroom reduced math achievement for median-skill students. As a result, the Double-Dose policy had little or no effect for median-skill students in schools that exposed them to low-skill classrooms. However, the effects of Double Dose were substantially positive in schools that did not do so. We consider policy implications and interpretations of the results from RDDs.

Intensive Math Instruction and Educational Attainment

We study an intensive math instruction policy that assigned low-skilled ninth graders to an algebra course that doubled instructional time, altered peer composition and emphasized problem solving skills. A regression discontinuity design shows substantial positive impacts of double-dose algebra on credits earned, test scores, high school graduation, and college enrollment rates. Test score effects underpredict attainment effects, highlighting the importance of long-run evaluation of such a policy. Perhaps because the intervention focused on verbal exposition of mathematical concepts, the impact was largest for students with below-average reading skills, emphasizing the need to target interventions toward appropriately skilled students.

Rejoinder: Probing Assumptions, Enriching Analysis

Thanks very much to Howard Bloom, Derek Neal, and Mike Seltzer for insightful comments on our article. Their commentary focused foremost on the advantages and disadvantages of using Options A, B, or C in using instrumental variables in multisite trials. Of interest are the quantities one can estimate and test in each case, strength of assumptions required, and robustness to outliers, and strategies for probing assumptions. The other key theme regards approaches to enrich the study of mechanisms whereby programs come to have effects. A central concern in our article is the interpretation one can make of δs , the average effect of program participation in each site;δ, the overall average effect of program participation; and τ 2 δ , the between-site variance in site-specific effects δs . Derek Neal makes the excellent point that, under comparatively weak assumptions, Option B identifies δ as the average effect of program participation on the compliers. The key assumption is monotonicity (which says that assignment to the program does not reduce program participation for any subset of the population) as defined in our article. In contrast, Option A under precision weighting and Option C require that, to support the same interpretation of δ, we invoke a stronger assumption, namely, that site-average compliance is independent of the site-average effect on the compliers. We agree with this point, and wish to emphasize that estimation of τ 2 δ under Option also B does not require the independence assumption concerning between-site covariance and effect so long as δs is regarded as the local average treatment effect in site s under the monotonicity assumption. We worry that the interpretation of δ as a complier-average treatment effect (or local average treatment effect) becomes problematic when we move beyond the case of binary program assignment T and binary mediator M. In particular, when M is measured on an interval or ratio scale, we often would like to estimate a dose-response relationship. For example, in the Tennessee class size experiment, where students and teachers were assigned to T = 1 (a small class) or T = 0 (a large class), we want to know the impact of the realized class size, M (Krueger, 1999; Krueger & Whitmore, 2001; Shin & Raudenbush, 2011). Here the monotonicity assumption does not support a dose-response interpretation of the association between M and Y but rather generates a compliance-weighted average effect,