Roberto Agodini

Are Experiments the Only Option? A Look at Dropout Prevention Programs

By comparing experimental and propensity-score impact estimates of dropout prevention programs, we examine whether propensity-score methods produce unbiased estimates of program impacts. We find no consistent evidence that such methods replicate experimental impacts in our setting. This finding holds even when the data available for matching are extensive. Our findings suggest that evaluators who plan to use nonexperimental methods, such as propensity-score matching, need to consider carefully how programs recruit individuals and why individuals enter programs, as unobserved factors may exert powerful influences on outcomes that are not easily captured using nonexperimental methods.

An Experimental Evaluation of Four Elementary School Math Curricula.

Abstract This article examines the effectiveness of four elementary school math curricula: (a) Investigations in Number, Data, and Space; (b) Math Expressions; (c) Saxon Math; and (d) Scott Foresman–Addison Wesley Mathematics (SFAW). These curricula are distinct from one another and represent many of the diverse approaches used to teach elementary school math in the United States. The results are based on 39 schools that were randomly assigned to use the various curricula at the first-grade level. The results show that average spring first-grade math achievement of Math Expressions and Saxon students was 0.30 SD higher than Investigations students and 0.24 SD higher than SFAW students. These effect sizes mean that an average-performing students percentile rank in math would be 9 to 12 points higher if the school used Math Expressions or Saxon, instead of Investigations or SFAW. We also conducted correlational analyses that examine whether curriculum-group differences in math instructional time and content coverage account for the differences in curriculum effects and found that a portion of the relative effects is due to differences in content coverage.

An experimental assessment of the effects of reading software products on reading test scores

Responding to a Congressional mandate in the No Child Left Behind Act, the effectiveness of nine reading instructional software applications was investigated using an experimental design and a large sample of American schools, teachers, and students. The software packages were selected based on evidence of effectiveness from previous research. Five software applications were for first‐grade students and four were for fourth‐grade students. Standardized tests were administered in the fall and spring of the 2004–2005 school year, classrooms were observed, and school records data were collected. The findings indicated that instructional activities were altered by the presence of software, but test scores were not statistically significantly higher in classrooms in which the software applications were used. For a subset of teachers and schools, an additional year of follow‐up was conducted to examine whether effectiveness was moderated by a year of teacher experience. The results were mixed, with measured eff...

How Teacher and Classroom Characteristics Moderate the Effects of Four Elementary Math Curricula

Research shows that some elementary math curricula are more effective than others at increasing student achievement. Most studies in this research base typically included teachers and classrooms that differ, so the results represent average curriculum effects, which raises an important question: Are curricula that are effective on average also effective among different types of teachers and classrooms? We examined whether curriculum effects are moderated by three characteristics that influence curriculum implementation and therefore may influence effects: (a) teachers’ knowledge, (b) teachers’ attitude toward math instruction, and (c) the extent to which teachers need to differentiate instruction in their classrooms. We examined these moderators for four elementary math curricula that use different pedagogical approaches and specifically for first- and second-grade achievement. We found that the two curricula that were more effective on average also were either as effective or more effective than the other two across all the contexts examined.