John P. Papay

Different Tests, Different Answers The Stability of Teacher Value-Added Estimates Across Outcome Measures

Recently, educational researchers and practitioners have turned to value-added models to evaluate teacher performance. Although value-added estimates depend on the assessment used to measure student achievement, the importance of outcome selection has received scant attention in the literature. Using data from a large, urban school district, I examine whether value-added estimates from three separate reading achievement tests provide similar answers about teacher performance. I find moderate-sized rank correlations, ranging from 0.15 to 0.58, between the estimates derived from different tests. Although the tests vary to some degree in content, scaling, and sample of students, these factors do not explain the differences in teacher effects. Instead, test timing and measurement error contribute substantially to the instability of value-added estimates across tests.

Can Professional Environments in Schools Promote Teacher Development? Explaining Heterogeneity in Returns to Teaching Experience

Although wide variation in teacher effectiveness is well established, much less is known about differences in teacher improvement over time. We document that average returns to teaching experience mask large variation across individual teachers and across groups of teachers working in different schools. We examine the role of school context in explaining these differences using a measure of the professional environment constructed from teachers responses to state-wide surveys. Our analyses show that teachers working in more supportive professional environments improve their effectiveness more over time than teachers working in less supportive contexts. On average, teachers working in schools at the 75th percentile of professional environment ratings improved 38% more than teachers in schools at the 25th percentile after 10 years.

The Consequences of High School Exit Examinations for Low-Performing Urban Students: Evidence From Massachusetts

In specifying a minimum passing score on examinations that students must pass to obtain a high school diploma, states divide a continuous performance measure into dichotomous categories. Thus, students with scores near the cutoff either pass or fail despite having essentially equal skills. The authors evaluate the causal effects of barely passing or failing a high school exit examination on the probability of graduation using a regression discontinuity design. For most Massachusetts students, barely failing their first 10th grade mathematics or English language arts (ELA) examination does not affect their probability of graduating. However, low-income urban students who just fail the mathematics examination have an 8 percentage point lower graduation rate than observationally similar students who just pass. There is no analogous impact from just passing or failing the ELA exit examination. For these urban, low-income students, barely failing the mathematics test does not affect the likelihood of on-time grade promotion, but it does cause students to be 4 percentage points more likely to drop out of school in the year following the test. Low-income urban students are just as likely to retake the test as equally skilled suburban students, but they have less success on retest.

Does an Urban Teacher Residency Increase Student Achievement? Early Evidence From Boston:

Boston Teacher Residency (BTR) is an innovative practice-based preparation program in which candidates work alongside a mentor teacher for a year before becoming a teacher of record in the Boston Public Schools (BPS). The authors found that BTR graduates are more racially diverse than other BPS novices, more likely to teach math and science, and more likely to remain teaching in the district through Year 5. Initially, BTR graduates for whom value-added performance data are available are no more effective at raising student test scores than other novice teachers in English language arts and less effective in math. The effectiveness of BTR graduates in math improves rapidly over time, however, such that by their 4th and 5th years they outperform veteran teachers. Simulations of the program’s overall effect through retention and effectiveness suggest that it is likely to improve student achievement in the district only modestly over the long run.

Is PAR a Good Investment? Understanding the Costs and Benefits of Teacher Peer Assistance and Review Programs

Peer Assistance and Review (PAR) is a local labor–management initiative designed to improve teacher quality. In PAR, expert “consulting teachers” mentor, support, and evaluate novice and underperforming veteran teachers. Evaluations under PAR can lead to dismissals. The authors examine the costs and benefits of PAR, both financial and organizational. Although PAR is an expensive reform, costing US

Educating Amid Uncertainty The Organizational Supports Teachers Need to Serve Students in High-Poverty, Urban Schools

3,000 to US

Leading the Local: Teachers Union Presidents Chart Their Own Course

10,000 per teacher served, it affords the district a range of financial savings and organizational benefits that offset program costs. The authors argue that limiting the scope of an educational cost-benefit study to only quantifiable elements artificially constrains understanding what a reform actually requires and offers.

The Impact of Test Score Labels on Human-Capital Investment Decisions

Purpose: We examine how uncertainty, both about students and the context in which they are taught, remains a persistent condition of teachers’ work in high-poverty, urban schools. We describe six schools’ organizational responses to these uncertainties, analyze how these responses reflect open- versus closed-system approaches, and examine how this orientation affects teachers’ work. Research Methods: We draw on interviews with a diverse set of 95 teachers and administrators across a purposive sample of six high-poverty, urban schools in one district. We analyzed these interviews by drafting thematic summaries, coding interview transcripts, creating data-analytic matrices, and writing analytic memos. Findings: We find that students introduced considerable uncertainty into teachers’ work. Although most teachers we spoke with embraced the challenges of their work and the expanded responsibilities that it entailed, they recognized that their individual efforts were not sufficient to succeed. Teachers consistently spoke about the need for organizational responses that addressed the environmental uncertainty of working with students from low-income families whose experience in school often has been unsuccessful. We describe four types of organizational responses—coordinated instructional supports, systems to promote order and discipline, socioemotional supports for students, and efforts to engage parents—and illustrate how these responses affected teachers’ ability to manage the uncertainty introduced by their environment. Conclusions: Traditional public schools are open systems and require systematic organizational responses to address the uncertainty introduced by their environments. Uncoordinated individual efforts alone are not sufficient to meet the needs of students in high-poverty urban communities.

High-School Exit Examinations and the Schooling Decisions of Teenagers: Evidence From Regression-Discontinuity Approaches

Teachers unions are among the most powerful, yet least studied, actors in public education today. Although public attention focuses on the influence of national unions, the policies that most affect teachers and schooling are bargained by local unions and school boards. Interviews with 30 recently elected local union presidents reveal that these leaders balance competing interests and obligations, from the concerns and priorities of their members to pressures from state and national affiliates. Although these presidents reaffirm the traditional union agenda, most also advocate an expanded agenda of teacher professionalism and more collaborative approaches to collective bargaining and contract management.

An Idea Whose Time Had Come: Negotiating Teacher Evaluation Reform in New Haven, Connecticut.

Students receive abundant information about their educational performance, but how this information affects future educational-investment decisions is not well understood. Increasingly, results from state-mandated standardized tests are an important source of information. Students receive a score and a label that summarizes their performance on these tests. Using a regression-discontinuity design, we find persistent effects of earning a more positive label on the college-going decisions of urban, low-income students. These findings are important not only for understanding students’ educational-investment decisions and the consequences of state testing practices but also for researchers using the regression-discontinuity design.