Jonathan Schweig

Approaches for Combining Multiple Measures of Teacher Performance: Reliability, Validity, and Implications for Evaluation Policy

A key question facing teacher evaluation systems is how to combine multiple measures of complex constructs into composite indicators of performance. We use data from the Measures of Effective Teaching (MET) study to investigate the measurement properties of composite indicators obtained under various conjunctive, disjunctive (or complementary), and weighted (or compensatory) models. We find that accuracy varies across models and cut-scores and that models with similar accuracy may yield different teacher classifications. Accuracy and consistency are greatest if composites are constructed to maximize reliability and lowest if they seek to optimally predict student test scores. We discuss the implications of the results for the validity of inferences about the performance of individual teachers, and more generally for the design of teacher evaluation systems.

Is There a Magnet-School Effect? A Multisite Study of MSAP-Funded Magnet Schools

ABSTRACT Magnet schools are one of the largest sectors of choice schools in the United States. In this study, we explored the heterogeneity in magnet-school effects on student achievement by examining 24 magnet schools, funded under the Magnet Schools Assistance Program (MSAP), in 5 school districts across 4 states. The magnet effects were synthesized across schools with a multilevel variance-known analysis, using the school-level effects estimated with a propensity score matched regression approach. Results indicated significant variation in magnet effects on student outcomes, with some magnet schools showing positive effects, and others showing negative effects. This variation can be explained by program implementation and magnet support.

Intention-to-treat analysis in partially nested randomized controlled trials with real-world complexity

ABSTRACT Demands for scientific knowledge of what works in educational policy and practice has driven interest in quantitative investigations of educational outcomes, and randomized controlled trials (RCTs) have proliferated under these conditions. In educational settings, even when individuals are randomized, both experimental and control students are often grouped into particular classrooms and schools and share common learning experiences. Analyses that account for these clusters are common. A less common design involves one clustered experimental arm and one unclustered experimental arm, sometimes called a partially clustered design. Analysts do not always use methods that yield valid statistical inferences for such partially clustered designs. Additionally, published methods for handling partially clustered designs may not be flexible enough to handle real-world complications, including treatment non-compliance. In this paper, we illustrate how models that accommodate partial clustering may be used in educational research. We explore the performance of these models using a series of Monte Carlo simulations informed by data taken from a large-scale RCT studying the impacts of a programme designed to decrease summer learning loss. We find that clustering and non-compliance can have substantial impacts on statistical inferences about intent-to-treat effects, and demonstrate methods that show promise for addressing these complications.

Study Suggests: Kids Who Attend More Thrive More

Excerpted from Learning from Summer: Effects of Voluntary Summer Learning Programs on Low-Income Urban Youth, Catherine H. Augustine et al., RAND Corporation, RR-1557, 2016. The RAND Corporation is a research organization that develops solutions to public policy challenges to help make communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the public interest. www.rand.org/t/RR1557 These findings are correlational but very likely due to the summer learning programs. The differences in the performance between the “high attenders” and the control group are the equivalent of about 20%–25% of a year’s learning in language arts and math at this age. These benefits persisted throughout the 5th-grade school year. Kids with high attendance performed better in math and reading after two summers compared with students in the control group, who were not invited to participate in the programs.