Louis T. Mariano

The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment Data

Open-ended or “constructed” student responses to test items have become a stock component of standardized educational assessments. Digital imaging of examinee work now enables a distributed rating process to be flexibly managed, and allocation designs that involve as many as six or more ratings for a subset of responses are now feasible. In this article we develop Patz’s (1996) hierarchical rater model (HRM) for polytomous item response data scored by multiple raters, and show how it can be used to scale examinees and items, to model aspects of consensus among raters, and to model individual rater severity and consistency effects. The HRM treats examinee responses to open-ended items as unobsered discrete varibles, and it explicitly models the “proficiency” of raters in assigning accurate scores as well as the proficiency of examinees in providing correct responses. We show how the HRM “fits in” to the generalizability theory framework that has been the traditional tool of analysis for rated item response data, and give some relationships between the HRM, the design effects correction of Bock, Brennan and Muraki (1999), and the rater bundle model of Wilson and Hoskens (2002). Using simulated and real data, we compare the HRM to the conventional IRT Facets model for rating data (e.g., Linacre, 1989; Engelhard, 1994, 1996), and we explore ways that information from HRM analyses may improved the quality of the rating process.

Bayesian Methods for Scalable Multivariate Value-Added Assessment.

There is increased interest in value-added models relying on longitudinal student-level test score data to isolate teachers’ contributions to student achievement. The complex linkage of students to teachers as students progress through grades poses both substantive and computational challenges. This article introduces a multivariate Bayesian formulation of the longitudinal model developed by McCaffrey, Lockwood, Koretz, Louis, and Hamilton (2004) that explicitly parameterizes the long-term effects of past teachers on student outcomes in future years and shows how the Bayesian approach makes estimation feasible even for large data sets. The article presents empirical results using reading and mathematics achievement data from a large urban school district, providing estimates of teacher effect persistence and examining how different assumptions about persistence impact estimated teacher effects. It also examines the impacts of alternative methods of accounting for missing teacher links and of joint versus marginal modeling of reading and mathematics.

A Model for Teacher Effects From Longitudinal Data Without Assuming Vertical Scaling

There is an increasing interest in using longitudinal measures of student achievement to estimate individual teacher effects. Current multivariate models assume each teacher has a single effect on student outcomes that persists undiminished to all future test administrations (complete persistence [CP]) or can diminish with time but remains perfectly correlated (variable persistence [VP]). However, when state assessments do not use a vertical scale or the evolution of the mix of topics present across a sequence of vertically aligned assessments changes as students advance in school, these assumptions of persistence may not be consistent with the achievement data. We develop the “generalized persistence” (GP) model, a Bayesian multivariate model for estimating teacher effects that accommodates longitudinal data that are not vertically scaled by allowing less than perfect correlation of a teacher’s effects across test administrations. We illustrate the model using mathematics assessment data.

Covariates of the Rating Process in Hierarchical Models for Multiple Ratings of Test Items

When constructed response test items are scored by more than one rater, the repeated ratings allow for the consideration of individual rater bias and variability in estimating student proficiency. Several hierarchical models based on item response theory have been introduced to model such effects. In this article, the authors demonstrate how these models may be extended to include covariates of the rating process. For example, how do features of an essay grader’s training affect his or her performance? The authors show how to include covariates by embedding a linear model at appropriate levels of the model hierarchy. Depending on the level, such covariates may be thought of as determining fixed effects or random effects on the rating process. The authors also discuss the appropriate design matrix for such covariates, discuss how to incorporate needed identifiability constraints, and illustrate the methods using data from a rating study of a student assessment.

The Academic Effects of Summer Instruction and Retention in New York City

This article examines the impacts of summer instruction and test-based grade retention in New York City. We use a research design that exploits test score cutoffs used in assignment to these treatments. We find modest positive effects of summer instruction on English language arts (ELA) achievement for students assigned to summer instruction because of poor ELA performance but find little evidence of positive effects of summer instruction on math outcomes. After netting out estimates of differential test score growth within grades across years, the estimated effects of grade retention are substantial and positive through seventh grade on both math and ELA outcomes, suggesting that the additional year of instruction in fifth grade leads to improvements in subsequent grade achievement.

The Causal Effects of Grade Retention on Behavioral Outcomes.

ABSTRACT This study examines the impact of grade retention on behavioral outcomes under a comprehensive assessment-based student promotion policy in New York City. To isolate the causal effect of grade retention, we implement a fuzzy regression discontinuity (RD) design that exploits the fact that grade retention is largely determined by whether a student scores below a cutoff on a standardized test score. We use data on students subject to the policy over a nine-year span to examine impacts on attendance and disciplinary event outcomes. We do not find evidence of systematic effects of retention on behavioral outcomes in either direction. We do find sporadic nonsustained significant effects of retention on behavioral outcomes. When present, these isolated nonpersistent effects tend to be beneficial when found for retained elementary school students and mixed for retained middle school students.

The Effects of Grade Retention on High School Outcomes: Evidence from New York City Schools

This study examines the causal impact of grade retention on high school attainment outcomes. We use administrative data on New York City public school students and a regression discontinuity design based on test score cutoffs used to determine retention eligibility. Grade retention reduces high school credit accumulation and the likelihood of taking math and English Regents exams. For middle school students, we also find that retention increases dropout and reduces the likelihood of completing Regents exam graduation requirements. We also explore potential mechanisms and find that retention increases placement into special education and makes future retentions less likely.

How Does Repeating a Grade Impact Students' High School Persistence and Behavior? The Case of New York City

Test-based grade promotion and retention policies have been put into place by 17 states and numerous school districts in the past 20 years. These policies require students to repeat a grade if they do not meet a minimum academic performance level. There is much controversy surrounding grade retention polices. Advocates reason that having low-performing students repeat a grade will provide them with more instruction and time to develop the grade-level academic skills necessary for success in later grades. Critics point to a large body of 20th-century research showing that grade retention is associated with increased behavioral problems and higher high school dropout rates. These prior research studies were able to identify characteristics common to retained students—for example, that retained students also tended to have behavioral issues. They were not designed to detect cause-and-effect (or causal) relationships— for example, that grade retention actually caused an increase in behavioral issues. It is important to try to understand the actual cause-and-effect consequences of retention: This is an intensive, high-cost intervention with the potential to shape students’ academic progress. To inform the ongoing development of grade promotion and retention policies, researchers from the RAND Corporation and the University of California, Davis, teamed to examine the long-term causal effects of retention in grades 3 through 8 on student behavior and high school outcomes. The researchers used 12 years of student administrative data from the New York City Department of Education (NYC DOE) to answer two key questions: 1. What is the effect of grade retention on behavioral problems, such as suspensions and absenteeism? 2. What is the effect of grade retention on high school persistence and attainment, such as dropout rates, credit accumulation, and graduation outcomes?