Ben Kelcey

Teachers' Knowledge about Early Reading: Effects on Students' Gains in Reading Achievement.

Abstract This study developed a new survey of teachers’ knowledge about early reading and examined the effects of teachers’ knowledge on students’ reading achievement in Grades 1 to 3 in a large sample of Michigan schools. Using statistical models that controlled for teachers’ personal and professional characteristics, students’ prior reading achievement, and the clustering of high-knowledge teachers in schools and school districts with particular demographic composition, we found that the effects of teachers’ knowledge about early reading on students’ reading achievement were small. In 1st grade, students in classrooms headed by higher knowledge teachers performed better on year-end tests of reading comprehension but not word analysis. In 2nd and 3rd grades, the effects of teachers’ knowledge on either measure of students’ reading achievement were not statistically significant. Although the study suggests new forms of statistical analysis that might produce better estimates of the effects of teachers’ knowledge on students’ reading achievement, further research is needed to improve the conceptual and psychometric properties of measures of teachers’ knowledge of reading and to investigate the relation of their knowledge and their instructional practices.

Assessing the Effects of Teachers’ Reading Knowledge on Students’ Achievement Using Multilevel Propensity Score Stratification

This study investigated the relationship of teachers’ reading knowledge with students’ reading achievement using a direct teacher knowledge assessment rather than indirect proxies (e.g., certification). To address the inequitable distribution of teachers’ knowledge resulting from differences in teachers’ backgrounds and the disparities in how schools attract and cultivate knowledge, the study developed multilevel propensity score methods to identify comparable teachers on the basis of both teacher and school backgrounds. Results suggest that schools are complexly associated with differences in teachers’ knowledge and that comparisons which ignore the relevance of schools may be misleading. By comparing teachers with similar personal and school backgrounds, results show measured knowledge is significantly associated with students’ achievement in reading comprehension but not word analysis. The findings support policies which leverage school capacities to develop the specialized knowledge needed for teaching reading.

Considerations for Designing Group Randomized Trials of Professional Development With Teacher Knowledge Outcomes

Despite recent shifts in research emphasizing the value of carefully designed experiments, the number of studies of teacher professional development with rigorous designs has lagged behind its student outcome counterparts. We outline a framework for the design of group randomized trials (GRTs) with teachers’ knowledge as the outcome and consider mathematics and reading knowledge outcomes designed to assess the types of content problems that teachers encounter in practice. To estimate design parameters, we draw on a national sample of teachers for mathematics and a state Reading First sample to estimate for reading. Our results suggest that there is substantial clustering of teachers’ knowledge within schools and professional development GRTs will likely need increased sample sizes to account for this clustering.

Teachers’ Support of Students’ Vocabulary Learning During Literacy Instruction in High Poverty Elementary Schools:

The purpose of this study was to examine third-grade teachers’ support for students’ vocabulary learning in high poverty schools characterized by underachievement in reading. We examined the prevalence and nature of discourse actions teachers used to support vocabulary learning in different literacy lessons (e.g., phonics); these actions varied in the cognitive demands placed on the students. Results showed that teachers rarely engaged students in cognitively challenging work on word meanings. Various lesson features and student and teacher characteristics were associated with teachers’ support for students’ vocabulary learning (e.g., teachers’ knowledge about reading). A major finding was that the extent of teachers’ support of their students’ vocabulary learning was significantly related to gains in reading comprehension across the year.

Strategies for Improving Power in School-Randomized Studies of Professional Development.

Objectives: Group-randomized designs are well suited for studies of professional development because they can accommodate programs that are delivered to intact groups (e.g., schools), the collaborative nature of professional development, and extant teacher/school assignments. Though group designs may be theoretically favorable, prior evidence has suggested that they may be challenging to conduct in professional development studies because well-powered designs will typically require large sample sizes or expect large effect sizes. Using teacher knowledge outcomes in mathematics, we investigated when and the extent to which there is evidence that covariance adjustment on a pretest, teacher certification, or demographic covariates can reduce the sample size necessary to achieve reasonable power. Method: Our analyses drew on multilevel models and outcomes in five different content areas for over 4,000 teachers and 2,000 schools. Using these estimates, we assessed the minimum detectable effect sizes for several school-randomized designs with and without covariance adjustment. Results: The analyses suggested that teachers’ knowledge is substantially clustered within schools in each of the five content areas and that covariance adjustment for a pretest or, to a lesser extent, teacher certification, has the potential to transform designs that are unreasonably large for professional development studies into viable studies.

Approximate measurement invariance in cross-classified rater-mediated assessments

An important assumption underlying meaningful comparisons of scores in rater-mediated assessments is that measurement is commensurate across raters. When raters differentially apply the standards established by an instrument, scores from different raters are on fundamentally different scales and no longer preserve a common meaning and basis for comparison. In this study, we developed a method to accommodate measurement noninvariance across raters when measurements are cross-classified within two distinct hierarchical units. We conceptualized random item effects cross-classified graded response models and used random discrimination and threshold effects to test, calibrate, and account for measurement noninvariance among raters. By leveraging empirical estimates of rater-specific deviations in the discrimination and threshold parameters, the proposed method allows us to identify noninvariant items and empirically estimate and directly adjust for this noninvariance within a cross-classified framework. Within the context of teaching evaluations, the results of a case study suggested substantial noninvariance across raters and that establishing an approximately invariant scale through random item effects improves model fit and predictive validity.

Experimental Power for Indirect Effects in Group-randomized Studies with Group-level Mediators

ABSTRACT Mediation analyses have provided a critical platform to assess the validity of theories of action across a wide range of disciplines. Despite widespread interest and development in these analyses, literature guiding the design of mediation studies has been largely unavailable. Like studies focused on the detection of a total or main effect, an important design consideration is the statistical power to detect indirect effects if they exist. Understanding the sensitivity to detect indirect effects is exceptionally important because it directly influences the scale of data collection and ultimately governs the types of evidence group-randomized studies can bring to bear on theories of action. However, unlike studies concerned with the detection of total effects, literature has not established power formulas for detecting multilevel indirect effects in group-randomized designs. In this study, we develop closed-form expressions to estimate the variance of and the power to detect indirect effects in group-randomized studies with a group-level mediator using two-level linear models (i.e., 2-2-1 mediation). The results suggest that when carefully planned, group-randomized designs may frequently be well positioned to detect mediation effects with typical sample sizes. The resulting power formulas are implemented in the R package PowerUpR and the PowerUp!-Mediator software (causalevaluation.org).

Intraclass Correlation Coefficients for Designing Cluster-Randomized Trials in Sub-Saharan Africa Education

Objective: Over the past two decades, the lack of reliable empirical evidence concerning the effectiveness of educational interventions has motivated a new wave of research in education in sub-Saharan Africa (and across most of the world) that focuses on impact evaluation through rigorous research designs such as experiments. Often these experiments draw on the random assignment of entire clusters, such as schools, to accommodate the multilevel structure of schooling and the theory of action underlying many school-based interventions. Planning effective and efficient school randomized studies, however, requires plausible values of the intraclass correlation coefficient (ICC) and the variance explained by covariates during the design stage. The purpose of this study was to improve the planning of two-level school-randomized studies in sub-Saharan Africa by providing empirical estimates of the ICC and the variance explained by covariates for education outcomes in 15 countries. Method: Our investigation drew on large-scale representative samples of sixth-grade students in 15 countries in sub-Saharan Africa and includes over 60,000 students across 2,500 schools. We examined two core education outcomes: standardized achievement in reading and mathematics. We estimated a series of two-level hierarchical linear models with students nested within schools to inform the design of two-level school-randomized trials. Results: The analyses suggested that outcomes were substantially clustered within schools but that the magnitude of the clustering varied considerably across countries. Similarly, the results indicated that covariance adjustment generally reduced clustering but that the prognostic value of such adjustment varied across countries.

Sample Size Planning for Cluster-Randomized Interventions Probing Multilevel Mediation

Multilevel mediation analyses play an essential role in helping researchers develop, probe, and refine theories of action underlying interventions and document how interventions impact outcomes. However, little is known about how to plan studies with sufficient power to detect such multilevel mediation effects. In this study, we describe how to prospectively estimate power and identify sufficient sample sizes for experiments intended to detect multilevel mediation effects. We outline a simple approach to estimate the power to detect mediation effects with individual- or cluster-level mediators using summary statistics easily obtained from empirical literature and the anticipated magnitude of the mediation effect. We draw on a running example to illustrate several different types of mediation and provide an accessible introduction to the design of multilevel mediation studies. The power formulas are implemented in the R package PowerUpR and the PowerUp software (causalevaluation.org).

Designing Large-Scale Multisite and Cluster-Randomized Studies of Professional Development

ABSTRACT We develop a theoretical and empirical basis for the design of teacher professional development studies. We build on previous work by (a) developing estimates of intraclass correlation coefficients for teacher outcomes using two- and three-level data structures, (b) developing estimates of the variance explained by covariates, and (c) modifying the conventional optimal design framework to include differential covariate costs so as to capture the point at which the cost of collecting a covariate overtakes the reduction in variance it supplies. We illustrate the use of these estimates to explore the absolute and relative sensitivity of multilevel designs in teacher professional development studies. The results from these analyses are intended to guide researchers in making more-informed decisions about the tradeoffs and considerations involved in selecting study designs for assessing the impacts of professional development programs.