Matthew A. Kraft

When Rater Reliability Is Not Enough Teacher Observation Systems and a Case for the Generalizability Study

In recent years, interest has grown in using classroom observation as a means to several ends, including teacher development, teacher evaluation, and impact evaluation of classroom-based interventions. Although education practitioners and researchers have developed numerous observational instruments for these purposes, many developers fail to specify important criteria regarding instrument use. In this article, the authors argue that for classroom observation to succeed in its aims, improved observational systems must be developed. These systems should include not only observational instruments but also scoring designs capable of producing reliable and cost-efficient scores and processes for rater recruitment, training, and certification. To illustrate how such a system might be developed and improved, the authors provide an empirical example that applies generalizability theory to data from a mathematics observational instrument.

Can Professional Environments in Schools Promote Teacher Development? Explaining Heterogeneity in Returns to Teaching Experience

Although wide variation in teacher effectiveness is well established, much less is known about differences in teacher improvement over time. We document that average returns to teaching experience mask large variation across individual teachers and across groups of teachers working in different schools. We examine the role of school context in explaining these differences using a measure of the professional environment constructed from teachers responses to state-wide surveys. Our analyses show that teachers working in more supportive professional environments improve their effectiveness more over time than teachers working in less supportive contexts. On average, teachers working in schools at the 75th percentile of professional environment ratings improved 38% more than teachers in schools at the 25th percentile after 10 years.

Promise and Paradox: Measuring Students' Non-Cognitive Skills and the Impact of Schooling.

We used self-report surveys to gather information on a broad set of non-cognitive skills from 1,368 eighth graders. At the student level, scales measuring conscientiousness, self-control, grit, and growth mindset are positively correlated with attendance, behavior, and test-score gains between fourth grade and eighth grade. Conscientiousness, self-control, and grit are unrelated to test-score gains at the school level, however, and students attending over-subscribed charter schools score lower on these scales than do students attending district schools. Exploiting admissions lotteries, we find positive impacts of charter school attendance on achievement and attendance but negative impacts on these non-cognitive skills. We provide suggestive evidence that these paradoxical results are driven by reference bias or the tendency for survey responses to be influenced by social context.

The Effect of Teacher–Family Communication on Student Engagement: Evidence From a Randomized Field Experiment

Abstract In this study, we evaluate the efficacy of teacher communication with parents and students as a means of increasing student engagement. We estimate the causal effect of teacher communication by conducting a randomized field experiment in which sixth- and ninth-grade students were assigned to receive a daily phone call home and a text/written message during a mandatory summer school program. We find that frequent teacher–family communication immediately increased student engagement as measured by homework completion rates, on-task behavior, and class participation. On average, teacher–family communication increased the odds that students completed their homework by 40%, decreased instances in which teachers had to redirect students’ attention to the task at hand by 25%, and increased class participation rates by 15%. Drawing upon surveys and interviews with participating teachers and students, we identify three primary mechanisms through which communication likely affected engagement: stronger teacher–student relationships, expanded parental involvement, and increased student motivation.

Cognitive Skills, Student Achievement Tests, and Schools:

Cognitive skills predict academic performance, so schools that improve academic performance might also improve cognitive skills. To investigate the impact schools have on both academic performance and cognitive skills, we related standardized achievement-test scores to measures of cognitive skills in a large sample (N = 1,367) of eighth-grade students attending traditional, exam, and charter public schools. Test scores and gains in test scores over time correlated with measures of cognitive skills. Despite wide variation in test scores across schools, differences in cognitive skills across schools were negligible after we controlled for fourth-grade test scores. Random offers of enrollment to oversubscribed charter schools resulted in positive impacts of such school attendance on math achievement but had no impact on cognitive skills. These findings suggest that schools that improve standardized achievement-test scores do so primarily through channels other than improving cognitive skills.

The Underutilized Potential of Teacher-to-Parent Communication: Evidence from a Field Experiment

Parental involvement is correlated with student performance, though the causal relationship is less well established. This experiment examined an intervention that delivered weekly one-sentence individualized messages from teachers to the parents of high school students in a credit recovery program. Messages decreased the percentage of students who failed to earn course credit from 15.8% to 9.3%—a 41% reduction. This reduction resulted primarily from preventing drop-outs, rather than from reducing failure or dismissal rates. The intervention shaped the content of parent–child conversations with messages emphasizing what students could improve, versus what students were doing well, producing the largest effects. We estimate the cost of this intervention per additional student credit earned to be less than one-tenth the typical cost per credit earned for the district. These findings underscore the value of educational policies that encourage and facilitate teacher-to-parent communication to empower parental involvement in their childrens education.

Validating Arguments for Observational Instruments: Attending to Multiple Sources of Variation

Measurement scholars have recently constructed validity arguments in support of a variety of educational assessments, including classroom observation instruments. In this article, we note that users must examine the robustness of validity arguments to variation in the implementation of these instruments. We illustrate how such an analysis might be used to assess a validity argument constructed for the Mathematical Quality of Instruction instrument, focusing in particular on the effects of varying the rater pool, subject matter content, observation procedure, and district context. Variation in the subject matter content of lessons did not affect rater agreement with master scores, but the evaluation of other portions of the validity argument varied according to the composition of the rater pool, observation procedure, and district context. These results demonstrate the need for conducting such analyses, especially for classroom observation instruments that are subject to multiple sources of variation.

Can Principals Promote Teacher Development as Evaluators? A Case Study of Principals’ Views and Experiences

Purpose: New teacher evaluation systems have expanded the role of principals as instructional leaders, but little is known about principals’ ability to promote teacher development through the evaluation process. We conducted a case study of principals’ perspectives on evaluation and their experiences implementing observation and feedback cycles to better understand whether principals feel as though they are able to promote teacher development as evaluators. Research Method: We conducted interviews with a stratified random sample of 24 principals in an urban district that recently implemented major reforms to its teacher evaluation system. We analyzed these interviews by drafting thematic summaries, coding interview transcripts, creating data-analytic matrices, and writing analytic memos. Findings: We found that the evaluation reforms provided a common framework and language that helped facilitate principals’ feedback conversations with teachers. However, we also found that tasking principals with primary responsibility for conducting evaluations resulted in a variety of unintended consequences which undercut the quality of evaluation feedback they provided. We analyze five broad solutions to these challenges: strategically targeting evaluations, reducing operational responsibilities, providing principal training, hiring instructional coaches, and developing peer evaluation systems. Implications: The quality of feedback teachers receive through the evaluation process depends critically on the time and training evaluators have to provide individualized and actionable feedback. Districts that task principals with primary responsibility for conducting observation and feedback cycles must attend to the many implementation challenges associated with this approach in order for next-generation evaluation systems to successfully promote teacher development.

School Organizational Contexts, Teacher Turnover, and Student Achievement: Evidence From Panel Data.

We study the relationship between school organizational contexts, teacher turnover, and student achievement in New York City (NYC) middle schools. Using factor analysis, we construct measures of four distinct dimensions of school climate captured on the annual NYC School Survey. We identify credible estimates by isolating variation in organizational contexts within schools over time. We find that improvements in school leadership especially, as well as in academic expectations, teacher relationships, and school safety are all independently associated with corresponding reductions in teacher turnover. Increases in school safety and academic expectations also correspond with student achievement gains. These results are robust to a range of threats to validity suggesting that our findings are consistent with an underlying causal relationship.

Educating Amid Uncertainty The Organizational Supports Teachers Need to Serve Students in High-Poverty, Urban Schools

Purpose: We examine how uncertainty, both about students and the context in which they are taught, remains a persistent condition of teachers’ work in high-poverty, urban schools. We describe six schools’ organizational responses to these uncertainties, analyze how these responses reflect open- versus closed-system approaches, and examine how this orientation affects teachers’ work. Research Methods: We draw on interviews with a diverse set of 95 teachers and administrators across a purposive sample of six high-poverty, urban schools in one district. We analyzed these interviews by drafting thematic summaries, coding interview transcripts, creating data-analytic matrices, and writing analytic memos. Findings: We find that students introduced considerable uncertainty into teachers’ work. Although most teachers we spoke with embraced the challenges of their work and the expanded responsibilities that it entailed, they recognized that their individual efforts were not sufficient to succeed. Teachers consistently spoke about the need for organizational responses that addressed the environmental uncertainty of working with students from low-income families whose experience in school often has been unsuccessful. We describe four types of organizational responses—coordinated instructional supports, systems to promote order and discipline, socioemotional supports for students, and efforts to engage parents—and illustrate how these responses affected teachers’ ability to manage the uncertainty introduced by their environment. Conclusions: Traditional public schools are open systems and require systematic organizational responses to address the uncertainty introduced by their environments. Uncoordinated individual efforts alone are not sufficient to meet the needs of students in high-poverty urban communities.