Jerome V. D'Agostino

Predicting Teacher Performance With Test Scores and Grade Point Average: A Meta-Analysis:

A meta-analysis was conducted to examine the degree to which teachers’ test scores and their performance in preparation programs as measured by their collegiate grade point average (GPA) predicted their teaching competence. Results from 123 studies that yielded 715 effect sizes were analyzed, and the mediating effects of test and GPA type, criterion type, teaching level, service level, and decade of data collection were considered. It was found that test scores were at best modestly related to teaching competence and that performance in preparation programs was a significantly better predictor of teaching skill. Results revealed that test scores likely do not provide additional information beyond preservice performance to safeguard the public from incompetent teaching.

Instructional Sensitivity of a State's Standards-Based Assessment.

Abstract The accuracy of achievement test score inferences largely depends on the sensitivity of scores to instruction focused on tested objectives. Sensitivity requirements are particularly challenging for standards-based assessments because a variety of plausible instructional differences across classrooms must be detected. For this study, we developed a new method for capturing the alignment between how teachers bring standards to life in their classrooms and how the standards are defined on a test. Teachers were asked to report the degree to which they emphasized the states academic standards, and to describe how they taught certain objectives from the standards. Two curriculum experts judged the alignment between how teachers brought the objectives to life in their classrooms and how the objectives were operationalized on the state test. Emphasis alone did not account for achievement differences among classrooms. The best predictors of classroom achievement were the match between how the standards were taught and tested, and the interaction between emphasis and match, indicating that test scores were sensitive to instruction of the standards, but in a narrow sense.

High School Exit Exam Scores and University Performance

Many U.S. students must pass a standards-based exit exam to earn a high school diploma. The degree to which exit exams and state standards properly signal to students their preparedness for postsecondary schooling has been questioned. The alignment of test scores with college grades for students at the University of Arizona (n = 2,667) who took the Arizona high school exams was ascertained in this study. The pass/fail signal accuracy of test scores varied depending on subject: The writing cut score was well aligned with collegiate performance, the reading cut score was below expectations, and the mathematics cut score was set quite rigorously. High school content and performance standards might not be as diluted as prior research has suggested.

Using a Partitioned Treatment Design to Examine the Effect of Project WET

Although young people in 50 U.S. states and 21 countries learn about water resources through Project WET (Water Education for Teachers), few researchers have conducted summative evaluations of the program. The authors employed a partitioned, or differential, treatments design in which two groups of 6th-grade students received overlapping but unique lesson components. Using hierarchical linear modeling, the authors found that classrooms from both groups had similar pre- to posttest gains on a test of the common material, but each group outperformed the other group on tests of the unique material a group experienced.

Reading Recovery: Exploring the Effects on First-Graders' Reading Motivation and Achievement.

ABSTRACT This study examined the effects of Reading Recovery on childrens motivational levels, and how motivation may contribute to the effect of the intervention on literacy achievement. Prior studies concluded that Reading Recovery was positively associated with increased student motivation levels, but most of those studies were limited methodologically. The achievement and motivation levels before and after the intervention of Reading Recovery students and similarly low-performing first-grade students were compared using structural equation modeling. It was found that Reading Recovery had a .31 treatment effect on achievement after controlling for baseline achievement and motivational differences among the treatment and comparison students. Reading Recovery also was associated with greater average levels of posttest motivation, and motivation was found to mediate the treatment-achievement relationship. This study highlights how important it is for early reading interventions to consider the role motivation plays in literacy acquisition.

An International Meta-Analysis of Reading Recovery.

ABSTRACT Reading Recovery is one of the most researched literacy programs worldwide. Although there have been at least 4 quantitative reviews of its effectiveness, none have considered all rigorous group-comparison studies from all implementing nations from the late 1970s to 2015. Using a hierarchical linear modeling (HLM) v-known analysis, we examined if effects differed in the United States versus other nations, if experiments yielded larger effects than quasi-experiments, if the effects changed over time, and if the type of outcome mediated the impact estimates. We also considered the sustained effects of the intervention. After reviewing 203 primary studies, we identified 16 that met our criteria, such as treatment fidelity and experimental or high-quality quasi-experimental design. Based on a random effects model, the estimated overall effect was .59, with larger effects for outcomes based on the Observation Survey (Clay, 2013), and stronger effects in certain literacy domains, such as text reading, print knowledge, and general literacy. Although United States studies produced a larger point estimate (.61) compared to other countries (.52), and experiments (.69) yielded a larger estimate than quasi-experiments (.43), neither difference was statistically significant. Overall, effects did not change over time, but effects based on the Observation Survey did improve significantly from earlier to later studies. We also found that the long-term effect may diminish, but there were too few studies to estimate the sustained impact with confidence. The .59 overall effect places Reading Recovery in the top 10% in terms of impact of early literacy programs reviewed by the What Works Clearinghouse.

Increasing the Role of Educational Psychology Theory in Program Development and Evaluation

Program evaluation coursework is commonly housed in educational psychology departments. Often, however, it is presented as a topic separate from learning, motivation, development, and curricular design. Some students who major in evaluation and methodology are not encouraged to take courses in these substantive areas of educational psychology, and the field of educational psychology is not presented to students as an integration of sophisticated research methods, curricular design, and the study of psychological processes that occur during learning and teaching. This article argues that evaluation has become more concerned with theory, but little work has been done in articulating, and therefore using, substantive theory to develop and evaluate social interventions. Few evaluators possess the skills and knowledge to become more involved in this area, but educational psychologists, if properly prepared in evaluation methods and substantive areas, would be in great demand to improve programs designed to address social problems.

Examining the Sustained Effects of Reading Recovery

ABSTRACT Though the immediate effect of Reading Recovery (RR) is both strong and well established, the longer term or sustained effect has been less studied and the evidence regarding it has been less conclusive. Michigan Reading Recovery students (n = 328) were compared to control students (n = 264) while in first (2009–2010), third (2011), and fourth grades (2012), using propensity score matching to generate 3 levels of eligibility. Although the immediate effect measured at mid-year of first grade on the Observation Survey was large (1.17), the effect by the end of first grade on the same measure was .51, and by third grade, the effect was .16 on the state reading test. The overall effect completely diminished by fourth grade, but it was significant (.35) for the most eligible students in reading, and for moderately eligible (.34) and most eligible students (.35) in writing. The sustained effect overall was present but diminished by third grade, and was sustained into fourth grade for those students at greater risk. The findings suggest that RR instruction should be better tailored to the initial literacy profiles of individual students to maximize the longevity of the effect for all participants.