Gary T. Henry

Beyond Use: Understanding Evaluation’s Influence on Attitudes and Actions

Although use is a core construct in the field of evaluation, neither the change processes through which evaluation affects attitudes, beliefs, and actions, nor the interim outcomes that lie between the evaluation and its ultimate goal—social betterment—have been sufficiently developed. We draw a number of these change mechanisms, such as justification, persuasion, and policy diffusion, from the social science research literature, and organize them into a framework that has three levels: individual, interpersonal, and collective. We illustrate how these change processes can be linked together to form “pathways” or working hypotheses that link evaluation processes to outcomes that move us along the road toward the goal of social betterment. In addition, we join with Kirkhart (2000) in moving beyond use, to focus our thinking on evaluation influence. Influence, combined with the set of mechanisms and interim outcomes presented here, offers a better way for thinking about, communicating, and adding to the evidence base about the consequences of evaluation and the relationship of evaluation to social betterment.

The Mechanisms and Outcomes of Evaluation Influence

Past literature has identified several putative precursors of use, as well as alternative forms of use. However, important shortcomings still exist in previous work on use. In particular, inadequate attention has been given to the underlying processes that may mediate the effects of evaluation on attitude and action. In essence, a key part of the theory of change for evaluation itself is missing. To help fill this gap, we describe a framework designed to capture key mechanisms through which evaluation may have its effects. The framework includes change processes that have been validated in various social science literatures. It identifies three levels of analysis (individual, interpersonal and collective), each with four kinds of processes (general influence, attitudinal, motivational and behavioral). With a more comprehensive view of the mechanisms underlying evaluation’s influence, the field can move forward in relation to its understanding and facilitation of evaluation’s role in the service of social betterment.

Stayers and Leavers Early-Career Teacher Effectiveness and Attrition

Research on teacher development reports significant early-career increases in teacher effectiveness, but the extent to which this is attributable to the development of teachers who persist or to the attrition of less effective teachers is unclear. In this study of novice teachers in North Carolina public schools, the authors investigated the development of teachers’ effectiveness during their first five years in the classroom and contrasted the effectiveness of teachers who stayed with that of those who left. Across grade levels, teachers’ effectiveness increased significantly in their second year of teaching but flattened after three years. The teachers who left the profession were less effective, on average, than those who stayed at least five years, but this finding is somewhat less consistent than the findings of an initial jump in effectiveness and diminishing returns to on-the-job development.

Is HOPE Enough? Impacts of Receiving and Losing Merit-Based Financial Aid

In 1993, the creation of Georgia’s HOPE Scholarship Program accelerated interest in understanding the effects of merit-based student financial aid. This article compares a sample of “borderline” HOPE recipients (students just above the eligibility threshold) with similar nonrecipients to examine differences on four college performance outcomes. The HOPE Scholarship recipients accumulated more credit hours, achieved slightly higher grade point averages, and were more likely to have graduated after 4 years of college. In addition, HOPE recipients who attended 4-year institutions of higher education were more likely to persist in college. Most merit aid recipients lost their scholarships, however, which slightly reduced recipients’advantages on grade point average and credit hour accumulation. Differences in persistence and graduation are significant only for those who maintain eligibility for the scholarship, suggesting that scholarship retention is critical if merit aid programs are to help achieve several of the broad goals of higher education.

Early Education Policy Alternatives: Comparing Quality and Outcomes of Head Start and State Prekindergarten:

The debates over the 2003 reauthorization of Head Start highlighted a controversy about the devolution of federal early education policy. At the center of the debate is the concern that state control of early education programs will reduce the quality and effectiveness of federal support for children living in poverty, and their families. The current fragmentation of early education policy, with both federal Head Start programs and state-subsidized prekindergarten programs operating in close proximity, presents an opportunity to compare the programs’ quality and effectiveness within a region of common support. In this study, propensity score techniques were used to match a probability sample of Head Start participants in Georgia with a group of children who were eligible for Head Start but who attended the state prekindergarten program in Georgia. The two groups were statistically similar at the beginning of their preschool year on three of four direct assessments (p < .05), but by the beginning of kindergarten the children attending the state prekindergarten program posted higher developmental outcomes on five of six direct assessments (p < .05) and 14 of 17 ratings by kindergarten teachers (p < .05). This study indicates that economically disadvantaged children who attended Georgia’s universal prekindergarten entered kindergarten at least as well prepared as similar children who attended the Head Start program.

Toward an Integrative Framework for Evaluation Practice.

Evaluation has been beset with serious divisions, including the paradigm wars and the seeming segmentation of evaluation practice into distinct evaluation theories and approaches. In this paper, we describe key aspects of an integrative framework that may help evaluators move beyond such divisions. We offer a new scheme for categorizing evaluation methods within four inquiry modes, which are “families” or clusters of methods: description, classification, causal analysis, and values inquiry. In addition, we briefly describe a set of alternative evaluation purposes. We argue that, together with a form of realist philosophy, the framework of inquiry modes and evaluation purposes (1) provides a common lexicon for evaluators, which may help the field in moving beyond past divisions, and (2) offers a useful approach to evaluation planning.

The Effects of Experience and Attrition for Novice High-School Science and Mathematics Teachers

New Teacher Syndrome The employment retention of high-school science and math teachers in the United States has diminished over recent decades, such that now many students are being taught by novice teachers. Henry et al. (p. 1118) have analyzed the effectiveness of teachers by examining data collected from North Carolina public schools. Less-effective teachers seem to have a greater tendency to leave teaching and the greatest gains in teacher effectiveness are when the teacher is new—within the first 3 years on the job. Some subject areas are more likely to be affected by the preponderance of novice teachers than other subjects, including math and science. New teachers face a steep learning curve, and those who fall off tend to leave teaching. Because of the current high proportion of novice high-school teachers, many students’ mastery of science and mathematics depends on the effectiveness of early-career teachers. In this study, which used value-added models to analyze high-school teachers’ effectiveness in raising test scores on 1.05 million end-of-course exams, we found that the effectiveness of high-school science and mathematics teachers increased substantially with experience but exhibited diminishing rates of return by their fourth year; that teachers of algebra 1, algebra 2, biology, and physical science who continued to teach for at least 5 years were more effective as novice teachers than those who left the profession earlier; and that novice teachers of physics, chemistry, physical science, geometry, and biology exhibited steeper growth in effectiveness than did novice non–science, technology, engineering, and mathematics teachers.

Review: Promoting Realism in Evaluation Realistic Evaluation and the Broader Context

Realism, as a foundational philosophy and as an applied approach to inquiry, is increasingly visible in social science. Ray Pawson and Nick Tilley have provided an important service to the field of evaluation by applying some of the basic concepts of realism to frame an approach that they refer to as ‘realistic evaluation’. As they note, their contribution largely involves taking concepts of realism that are attracting attention in many social sciences and making them accessible to evaluators. We applaud them for their efforts, and look forward to their continued development and promotion of a realism-based view of evaluation.

Incorporating Teacher Effectiveness Into Teacher Preparation Program Evaluation

New federal and state policies require that teacher preparation programs (TPP) be held accountable for the effectiveness of their graduates as measured by test score gains of the students they teach. In this article, the authors review the approaches taken in several states that have already estimated TPP effects and analyze the proposals for incorporating students’ test score gains into the evaluations of TPP by states that have received federal Race to the Top funds. The authors organize their review to focus on three types of decisions that are required to implement these new accountability requirements: (a) selection of teachers, students, subjects, and years of data; (b) methods for estimating teachers’ effects on student test score gains; and (c) reporting and interpretation of effects. The purpose of the review is to inform the teacher preparation community on the state of current and near term practice for adding measures of teacher effectiveness to TPP accountability practices.

The Effects of Teacher Entry Portals on Student Achievement

The current teacher workforce is younger, less experienced, more likely to turnover, and more diverse in preparation experiences than the workforce of two decades ago. Research shows that inexperienced teachers are less effective, but we know little about the effectiveness of teachers with different types of preparation. In this study, we classify North Carolina public school teachers into portals—fixed and mutually exclusive categories that capture teachers’ formal preparation and qualifications upon first entering the profession—and estimate the adjusted average test score gains of students taught by teachers from each portal. Compared with undergraduate-prepared teachers from in-state public universities, (a) out-of-state undergraduate-prepared teachers are less effective in elementary grades and high school, (b) alternative entry teachers are less effective in high school, and (c) Teach For America corps members are more effective in STEM subjects and secondary grades.