David Blazar

Validating Arguments for Observational Instruments: Attending to Multiple Sources of Variation

Measurement scholars have recently constructed validity arguments in support of a variety of educational assessments, including classroom observation instruments. In this article, we note that users must examine the robustness of validity arguments to variation in the implementation of these instruments. We illustrate how such an analysis might be used to assess a validity argument constructed for the Mathematical Quality of Instruction instrument, focusing in particular on the effects of varying the rater pool, subject matter content, observation procedure, and district context. Variation in the subject matter content of lessons did not affect rater agreement with master scores, but the evaluation of other portions of the validity argument varied according to the composition of the rater pool, observation procedure, and district context. These results demonstrate the need for conducting such analyses, especially for classroom observation instruments that are subject to multiple sources of variation.

Attending to General and Mathematics-Specific Dimensions of Teaching: Exploring Factors Across Two Observation Instruments

ABSTRACT New systems that seek to evaluate teachers with regard to their classroom quality often rely on observation instruments that capture general instructional pedagogies. However, decades of research suggest that content-specific dimensions of instruction also are important to differentiate teachers and improve student outcomes. We explore the degree of overlap between a general and a content-specific instrument when capturing upper elementary teachers’ mathematics instruction. To do so, we conducted exploratory and confirmatory factor analyses on data from more than 2,000 videotaped lessons scored using both the Classroom Assessment Scoring System, a general instrument, and the Mathematical Quality of Instruction, a content-specific instrument. Findings indicate that there is some overlap between instruments but that preferred factor structures include both general and content-specific practices.

What Does It Mean to Be Ranked a “High” or “Low” Value-Added Teacher? Observing Differences in Instructional Quality Across Districts

Education agencies are evaluating teachers using student achievement data. However, very little is known about the comparability of test-based or “value-added” metrics across districts and the extent to which they capture variability in classroom practices. Drawing on data from four urban districts, we found that teachers were categorized differently when compared within versus across districts. In addition, analyses of scores from two observation instruments, as well as qualitative viewing of lesson videos, identified stark differences in instructional practices across districts among teachers who received similar within-district value-added rankings. These patterns were not explained by observable background characteristics of teachers, suggesting that factors beyond labor market sorting likely played a key role.

Resources for Teaching

Policymakers and researchers have for many years advocated disparate approaches to ensuring teachers deliver high-quality instruction, including requiring that teachers complete specific training requirements, possess a minimum level of content knowledge, and use curriculum materials and professional development resources available from schools and districts. In this paper, we investigate the extent to which these factors, which we conceptualize as resources for teaching, predict instructional quality in upper elementary mathematics classrooms. Results show that teachers’ mathematical knowledge and their district context explained a moderate share of the variation in mathematics-specific teaching dimensions; other factors, such as teacher experience, preparation, non-instructional work hours, and measures of the school environment, explained very little variation in any dimension.

Exploring Mechanisms of Effective Teacher Coaching A Tale of Two Cohorts From a Randomized Experiment

Although previous research has shown that teacher coaching can improve teaching practices and student achievement, little is known about specific features of effective coaching programs. We estimate the impact of MATCH Teacher Coaching (MTC) on a range of teacher practices using a blocked randomized trial and explore how changes in the coaching model across two cohorts are related to program effects. Findings indicate large positive effects in Cohort 1 but no effects in Cohort 2. After ruling out explanations related to the research design, a set of exploratory analyses suggest that differential treatment effects may be attributable to differences in coach effectiveness, coaching dosage, and the focus of coaching across cohorts.

The Effect of Teacher Coaching on Instruction and Achievement: A Meta-Analysis of the Causal Evidence:

Teacher coaching has emerged as a promising alternative to traditional models of professional development. We review the empirical literature on teacher coaching and conduct meta-analyses to estimate the mean effect of coaching programs on teachers’ instructional practice and students’ academic achievement. Combining results across 60 studies that employ causal research designs, we find pooled effect sizes of 0.49 standard deviations (SD) on instruction and 0.18 SD on achievement. Much of this evidence comes from literacy coaching programs for prekindergarten and elementary school teachers in the United States. Although these findings affirm the potential of coaching as a development tool, further analyses illustrate the challenges of taking coaching programs to scale while maintaining effectiveness. Average effects from effectiveness trials of larger programs are only a fraction of the effects found in efficacy trials of smaller programs. We conclude by discussing ways to address scale-up implementation challenges and providing guidance for future causal studies.

Grade Assignments and the Teacher Pipeline: A Low-Cost Lever to Improve Student Achievement?

Research on teacher stability typically focuses on the extent to which teachers remain in the same school, district, or the teaching profession from one year to the next. I investigate another facet of stability—whether teachers remain in the grade they teach. Drawing on administrative data from a large district in California, I find that high shares of teachers switch grades. Disproportionately, these are early career teachers who come from low-achieving or high-minority schools. Teachers who switch grades leave schools at higher rates than their colleagues and exhibit lower impacts on their students’ achievement. For teachers who switch to a nonadjacent grade, these negative effects can wipe out any gains due to increased experience and can persist in the year after the switch occurs.

Individualized Coaching to Improve Teacher Practice Across Grades and Subjects New Experimental Evidence

This article analyzes a coaching model focused on classroom management skills and instructional practices across grade levels and subject areas. We describe the design and implementation of MATCH Teacher Coaching among an initial cohort of 59 teachers working in New Orleans charter schools. We evaluate the effect of the program on teachers’ instructional practices using a block randomized trial and find that coached teachers scored 0.59 standard deviations higher on an index of effective teaching practices comprised of observation scores, principal evaluations, and student surveys. We discuss implementation challenges and make recommendations for researcher–practitioner partnerships to address key remaining questions.

Relationships between Observations of Elementary Mathematics Instruction and Student Achievement: Exploring Variability across Districts

Much debate surrounding teacher quality has focused on students’ standardized test scores, but recent federal and state initiatives have emphasized the use of multiple measures to evaluate teacher quality, including classroom observations. In this study, we explore differences across school districts in the relationship between student achievement outcomes and the observed quality of teachers’ instruction. Using data from 298 elementary mathematics teachers in five urban US districts, we examine relationships between teachers’ performance on the Mathematical Quality of Instruction observation instrument and their students’ scores on both state standardized and researcher-developed tests. We find that these relationships differ across school districts. We explore the extent to which differences in skills and expectations for students across tests may explain this variability. An improved understanding of the relationship between classroom observations and student tests may help districts to better support teachers in developing their instructional effectiveness.

Does Test Preparation Mean Low-Quality Instruction?.

Critics of test-based accountability warn that test preparation has a negative influence on teachers’ instruction due to a focus on procedural skills. Others advocate that the adoption of more rigorous assessments may be a way to incentivize more ambitious test preparation instruction. Drawing on classroom observations and teacher surveys, we do find that test preparation activities predict lower quality and less ambitious mathematics instruction in upper-elementary classrooms. However, the magnitudes of these relationships appear smaller than the prevailing narrative has warned. Further, our findings call into question the hypothesis that test rigor can serve as a lever to elevate test preparation to ambitious teaching. Therefore, improving the quality of mathematics instruction in the midst of high-stakes testing likely will require that policymakers and school leaders undertake comprehensive efforts that look beyond the tests themselves.