Ernest R. House

Assumptions Underlying Evaluation Models

One way of understanding evaluation is to compare the numerous evaluation models with one another. There are many possibilities for comparison, but perhaps the most significant comparisons are those among the underlying theoretical assumptions on which the models are based. In this way, one might see how logically similar the models are to one another and determine what logical possibilities do and do not exist.

An Instrument for Assessing Instructional Climate through Low-Inference Student Judgments

Instructional climate is an aspect of environmental press defined by the characteristic demands of the classroom environment as perceived by the students to whom they are directed. The concept of environmental press was described in 1938 by Henry Murray. From the characteristic modes of response of individuals are inferred needs whose strength and relationships characterize the personality. In a similar way the strengths and relationships of characteristic stresses, pressures, rewards, and other influences of the environment compose the environmental press. In 1956 Stern, Stein, and Bloom elaborated the environmental press concept by applying it to assessment studies and showing that an improvement in the prediction of performance was possible by defining the psychological demands of the situation in which the performance takes place. The College Characteristics Index developed in 1957 by Pace and Stern applied the concept of environmental press to college

A Framework for Appraising Educational Reforms

Transaction-cost economics provides a framework for appraising educational reforms, i.e., deciding whether these reforms are likely to succeed in the “real life” of schools. This framework conceives of reform as a contract between reformers and stake-holders (teachers, students, parents), and these transactions are marked by bounded rationality, opportunism, and the protection of specific assets (e.g., accumulated knowledge and skills) on the part of participants. Not acknowledging these attributes turns reforms into unproductive planning exercises—mere promises without results—or stakeholders struggling to protect their assets, resulting in the failure of the reform. Several current educational reforms are judged on the basis of their transaction costs and consequent prospects for success.

Three Perspectives on School Reform

Most research on school reform over the past several decades is characterized by three perspectives-the technological, political, and cultural (House, 1979; House, 1981). Studies based on these three perspectives account for a vast amount of the scholarly literature. An adequate understanding of school reform necessarily involves all three perspectives, though many reformers emphasize only one, a partial knowledge which often results in reform failure because of neglect of the other powerful factors. According to our analysis, successful school reform must be based on all three aspects. In this chapter, we outline the three perspectives and suggest how successful reforms embody an appreciation of all three.

Democracy and Evaluation

I am truly honored to be invited to speak to the EES. I have followed the progress of the Society and the Evaluation journal with great interest as they have established a different, significant voice in evaluation. Today I will discuss the conference themes, which are among the most critical issues facing evaluation. One word of caution: there is an old European saying, a very old saying: ‘Beware Greeks bearing gifts’. I would update that to ‘Beware Americans offering advice’, including myself, especially these days. A long shadow has fallen across the United States. The American tradition of open government is being seriously eroded. We have a regime in Washington under President George Bush that strives to control information beyond anything in my experience. Of course, all governments hide their mistakes and misdeeds and spin news to their advantage. Of course. But I am talking about control of information that threatens evaluation – and perhaps democracy. In the wake of the September 11 terrorist attacks, the US administration instigated neo-fundamentalist policies, a blend of religious fundamentalism and neoconservatism (House, 2003). In evaluation the new policy I call ‘methodological fundamentalism’. Some government agencies demand that all evaluations must be randomized experiments. Other ways of producing evidence are not scientific and not acceptable. There is one method for discovering the truth and one method only – the randomized experiment. If we employ randomized experiments, they will lead us to a Golden Age in education and social services. Such is the revelatory vision. There are sacred figures from the past and sacred texts. Core believers are absolutely certain they are right, that randomized experiments reveal the truth. They do not listen to critics, and they do not comprehend the arguments against the method. A mandate for randomized trials has been written into legislation without discussion with professional communities. Avoiding contrary ideas is part of the orientation, and, of course, the policy is enforced by government edict. Now the problem here is not randomized experimentation as such. It’s the belief that there is one source of truth and one only. The error would be equally Evaluation Copyright

Deliberative Democratic Evaluation in Practice

Many evaluators already implement the principles we explicate here without any urging from us. They have developed their own approaches, their own intuitions, and their own robust senses of justice. Nonetheless, such principles are too important to leave to chance or intuition all the time. It may help to have a justification and checklist to remind evaluators caught in the complexities of difficult evaluations what evaluation in democratic societies should aim for: deliberative democracy.

Blowback: Consequences of Evaluation for Evaluation

Drug studies are often cited as the best exemplars of evaluation design. However, many of these studies are seriously biased in favor of positive findings for the drugs evaluated, even to the point where dangerous effects are hidden. In spite of using randomized designs and double blinding, drug companies have found ways of producing the results they want, including manipulation of treatment, selection of sample, control of data, and calculated publication. Regulatory agencies have been neutralized. We have entered an era when evaluations are controlled by sponsors to produce the findings they want. Evaluations have become too important to be left to the evaluators. Such deceptive practices threaten the integrity of the evaluation field, perhaps its existence. There is no doubt these practices will spread into educational and social evaluation.Drug studies are often cited as the best exemplars of evaluation design. However, many of these studies are seriously biased in favor of positive findings for the drugs evaluated, even to the point where dangerous effects are hidden. In spite of using randomized designs and double blinding, drug companies have found ways of producing the results they want, including manipulation of treatment, selection of sample, control of data, and calculated publication. Regulatory agencies have been neutralized. We have entered an era when evaluations are controlled by sponsors to produce the findings they want. Evaluations have become too important to be left to the evaluators. Such deceptive practices threaten the integrity of the evaluation field, perhaps its existence. There is no doubt these practices will spread into educational and social evaluation.

Coherence and Credibility: The Aesthetics of Evaluation.

Consider two different images of the drinking driver. One may imagine the ordinary social drinker who happens to overindulge, and who, missing a stop sign, is detained by the police, thereby getting into trouble. Or, imagine a driver who is habitually drunk, a reeling, stumbling, insensate hazard to everyone on the road, including himself. The image that one constructs of the driver who drinks has much to do with the recommendations for action that one may embrace as a means of curtailing drinking drivers. Until the past decade, the drinking driver had been perceived as the social drinker, a civil problem susceptible to correction by legal actions such as imposed penalties and fines. Over the past 10 years or so, however, a series of studies has been influential in changing the dominant image of the drinking driver so that he is now perceived as more of a habitual, pathological drunk. The resulting recommendations for remediation are medical and are directed at a small subset of offenders rather than at all drivers who drink. The sociologist Gusfield (1976) analyzed the rhetoric of these studies-how the data were presented to persuade the reader of the conclusions. What he found is provoc-

Research news and Comment: Trends in Evaluation

W hen I began a career in evaluation more than 20 years ago, I tossed all the papers I could find about the topic into a small cardboard box in the corner of my office and read them in one month. Since that time, program evaluation has been transformed from a small, sideline activity conducted by part-time academics into a professionalized minor industry, replete with its own journals, awards, conventions, organizations, and standards. Evaluation even has its own entry in the Dictionary of Modern Thought (Bullock, Stallybrass, & Trombley, 1988). During this time, both the structural basis and conceptual underpinnings of the field have changed dramatically. Structurally, evaluation has become more integrated into organizational operations, and conceptually, evaluation has moved from monolithic to pluralist notions, to multiple methods, criteria, and interests. Evaluation is usually defined as the determination of the worth or value of something, in this case of educational programs, policies, and personnel, judged according to appropriate criteria, with those criteria explicated and justified. At its best, the evaluation of educational and social programs aspires to be an institution for democratizing public decisions by making programs and policies more open to public scrutiny and deliberation. As such, it should serve the interests not only of the sponsor but of the larger society and of diverse groups within society. Of course, evaluation has not always lived up to its own noble aspirations.

The issue of advocacy in evaluations

From time to time, articles are published in AJE that evoke comments from readers. In Response is reserved for this dialogue. Contributions should be to the point, concise, and easy for readers to track to targeted articles. Comments may be positive or negative, but tf the latter, then keep them at least relatively nice! Personal attacks and offensive, degrading criticisms will not be published. Please keep the length of comments to the minimum essential