J. R. Lockwood

Models for Value-Added Modeling of Teacher Effects

The use of complex value-added models that attempt to isolate the contributions of teachers or schools to student development is increasing. Several variations on these models are being applied in the research literature, and policy makers have expressed interest in using these models for evaluating teachers and schools. In this article, we present a general multivariate, longitudinal mixed-model that incorporates the complex grouping structures inherent to longitudinal student data linked to teachers. We summarize the principal existing modeling approaches, show how these approaches are special cases of the proposed model, and discuss possible extensions to model more complex data structures. We present simulation and analytical results that clarify the interplay between estimated teacher effects and repeated outcomes on students over time. We also explore the potential impact of model misspecifications, including missing student covariates and assumptions about the accumulation of teacher effects over time, on key inferences made from the models. We conclude that mixed models that account for student correlation over time are reasonably robust to such misspecifications when all the schools in the sample serve similar student populations. However, student characteristics are likely to confound estimated teacher effects when schools serve distinctly different populations.

Uncertainty in Rank Estimation: Implications for Value-Added Modeling Accountability Systems

Accountability for public education often requires estimating and ranking the quality of individual teachers or schools on the basis of student test scores. Although the properties of estimators of teacher-or-school effects are well established, less is known about the properties of rank estimators. We investigate performance of rank (percentile) estimators in a basic, two-stage hierarchical model capturing the essential features of the more complicated models that are commonly used to estimate effects. We use simulation to study mean squared error (MSE) performance of percentile estimates and to find the operating characteristics of decision rules based on estimated percentiles. Each depends on the signal-to-noise ratio (the ratio of the teacher or school variance component to the variance of the direct, teacher- or school-specific estimator) and only moderately on the number of teachers or schools. Results show that even when using optimal procedures, MSE is large for the commonly encountered variance ratios, with an unrealistically large ratio required for ideal performance. Percentile-specific MSE results reveal interesting interactions between variance ratios and estimators, especially for extreme percentiles, which are of considerable practical import. These interactions are apparent in the performance of decision rules for the identification of extreme percentiles, underscoring the statistical and practical complexity of the multiple goal inferences faced in value-added modeling. Our results highlight the need to assess whether even optimal percentile estimators perform sufficiently well to be used in evaluating teachers or schools.

Bayesian Methods for Scalable Multivariate Value-Added Assessment.

There is increased interest in value-added models relying on longitudinal student-level test score data to isolate teachers’ contributions to student achievement. The complex linkage of students to teachers as students progress through grades poses both substantive and computational challenges. This article introduces a multivariate Bayesian formulation of the longitudinal model developed by McCaffrey, Lockwood, Koretz, Louis, and Hamilton (2004) that explicitly parameterizes the long-term effects of past teachers on student outcomes in future years and shows how the Bayesian approach makes estimation feasible even for large data sets. The article presents empirical results using reading and mathematics achievement data from a large urban school district, providing estimates of teacher effect persistence and examining how different assumptions about persistence impact estimated teacher effects. It also examines the impacts of alternative methods of accounting for missing teacher links and of joint versus marginal modeling of reading and mathematics.

Using Structured Classroom Vignettes to Measure Instructional Practices in Mathematics

Large-scale educational studies frequently require accurate descriptions of classroom practices to judge implementation and impact. However, it can be difficult to obtain these descriptions in a timely, efficient manner. To address this problem, the authors developed a vignette-based measure of one aspect of mathematics instructional practice, reform-oriented instruction. Teachers read contextualized descriptions of teaching practices that varied in terms of reform-oriented instruction, and rated the degree to which the options corresponded to their own likely behaviors. Responses from 80 fourth-grade teachers yielded fairly consistent responses across two parallel vignettes and moderate correlations with other scales of reform-oriented instruction derived from classroom observations, surveys, and logs. The results suggested that the vignettes measure important aspects of reform-oriented instruction that are not captured by other measurement methods. Based on this work, it appears that vignettes can be a useful tool for research on instructional practice, but cognitive interviews with participating teachers provided insight into possible improvements to the items.

Correcting for Test Score Measurement Error in ANCOVA Models for Estimating Treatment Effects

A common strategy for estimating treatment effects in observational studies using individual student-level data is analysis of covariance (ANCOVA) or hierarchical variants of it, in which outcomes (often standardized test scores) are regressed on pretreatment test scores, other student characteristics, and treatment group indicators. Measurement error in the prior test scores, which typically is both large and heteroscedastic, is regularly overlooked in empirical analyses and may erode the ability of regression models to adjust for student factors and may result in biased treatment effect estimates. We develop extensions of method-of-moments, Simulation-Extrapolation, and latent regression approaches to correcting for measurement error using the conditional standard errors of measure of test scores, and demonstrate their effectiveness relative to simpler alternatives using both simulation and a case study of teacher value-added effect estimation using longitudinal data from a large suburban school district.

A Model for Teacher Effects From Longitudinal Data Without Assuming Vertical Scaling

There is an increasing interest in using longitudinal measures of student achievement to estimate individual teacher effects. Current multivariate models assume each teacher has a single effect on student outcomes that persists undiminished to all future test administrations (complete persistence [CP]) or can diminish with time but remains perfectly correlated (variable persistence [VP]). However, when state assessments do not use a vertical scale or the evolution of the mix of topics present across a sequence of vertically aligned assessments changes as students advance in school, these assumptions of persistence may not be consistent with the achievement data. We develop the “generalized persistence” (GP) model, a Bayesian multivariate model for estimating teacher effects that accommodates longitudinal data that are not vertically scaled by allowing less than perfect correlation of a teacher’s effects across test administrations. We illustrate the model using mathematics assessment data.

Where You Come from or Where You Go? Distinguishing between School Quality and the Effectiveness of Teacher Preparation Program Graduates

In this paper we consider the challenges involved in evaluating teacher preparation programs when controlling for school contextual bias. Including school fixed effects in the achievement models used to estimate preparation program effects controls for school environment by relying on differences among student outcomes within the same schools to identify the program effects. However, identification of preparation program effects using school fixed effects requires teachers from different programs to teach in the same school. Even if program effects are identified, the precision of the estimated effects will depend on the degree to which graduates from different programs overlap across schools. In addition, if the connections between preparation programs result from the overlap of atypical graduates or from graduates teaching in atypical school environments, use of school effects could produce bias. Using statewide data from Florida, we show that teachers tend to teach in schools near the programs in which they received their training, but there is still sufficient overlap across schools to identify preparation program effects. We show that the ranking of preparation programs varies significantly depending on whether or not school environment is taken into account via school fixed effects. We find that schools and teachers that are integral to connecting preparation programs are atypical, with disproportionately high percentages of Hispanic teachers and students compared to the state averages. Finally, we find significant variance inflation in the estimated program effects when controlling for school fixed effects, and that the size of the variance inflation factor depends crucially on the length of the window used to compare graduates teaching in the same schools.

Fitting Value-Added Models in R:

Value-added models of student achievement have received widespread attention in light of the current test-based accountability movement. These models use longitudinal growth modeling techniques to identify effective schools or teachers based upon the results of changes in student achievement test scores. Given their increasing popularity, this article demonstrates how to perform the data analysis necessary to fit a general value-added model using the nlme package available for the R statistics environment. We demonstrate techniques for inspecting the data prior to fitting the model, walk a practitioner through a sample analysis, and discuss general extensions commonly found across the literature that may be incorporated to enhance the basic model presented, including the estimation of multiple outcomes and teacher effects.

A Longitudinal Investigation of the Relationship between Teachers’ Self-Reports of Reform-Oriented Instruction and Mathematics and Science Achievement:

In the past two decades, several major initiatives were launched to improve mathematics and science education. One prominent feature in these efforts was a new approach to teaching mathematics and science, referred to as reform-oriented teaching. Although past studies suggest this approach may improve student achievement, the relationships between reform-oriented pedagogy and achievement were weak. The weak relationships may be partially attributable to the limited time frame in which reform-oriented teaching was examined (typically a 1-year period). This study explored the relationship between mathematics and science achievement and reform-oriented teaching over a 3-year period. Results suggested greater exposure to reform-oriented instruction was generally not significantly associated with higher student achievement but the effects became stronger with prolonged exposure to reform-oriented practices. Reform-oriented instruction showed stronger, positive relationships with open-ended measures than with multiple-choice tests in both mathematics and science and with problem-solving skills than with procedural skills in mathematics.

Alternative Statistical Frameworks for Student Growth Percentile Estimation

This article suggests two alternative statistical approaches for estimating student growth percentiles (SGP). The first is to estimate percentile ranks of current test scores conditional on past test scores directly, by modeling the conditional cumulative distribution functions, rather than indirectly through quantile regressions. This would remove the need for post hoc procedures required to ensure monotonicity of the estimated quantile functions, and for inversion of those functions to obtain SGP. We provide a brief empirical example demonstrating this approach and its potential benefits for handling discreteness of the observed test scores. The second suggestion is to estimate SGP directly from longitudinal item-level data, using multidimensional item response theory models, rather than from test scale scores. This leads to an isomorphism between using item-level data from one test to make inferences about latent student achievement, and using item-level data from multiple tests administered over time to make inferences about latent SGP. This framework can be used to solve the bias problems for current SGP methods caused by measurement error in both the current and past test scores, and provides straightforward assessments of uncertainty in SGP. We note practical problems that need to be addressed to implement our suggestions.