Adrianne Bank

From program evaluation to intstructional information system

Abstract This article describes the emergence of a new and exciting possibility for upgrading teaching and learning in American schools. We call this new phenomenon, instructional information systems (IIS). Instructional information systems are computerized data banks that allow a variety of users to ask important evaluative questions about student learning, classroom function, school management, and district policies. They built on current practice in testing and program evaluation but go way beyond it.

Explanation of the Evaluation Design Project

The Evaluation Design Project, located at the Center for the Study of Evaluation (CSE), has completed the first year of its three-year (1979-1982) effort. The project represents one aspect of CSE’s long-term plan to conduct research into the ways testing and evaluation activities may be designed and conducted to impact beneficially on instruction and student learning. Our intentions remain as stated in our original plan, &dquo;to examine existing district policies, procedures and programs that appear to contribute to systematic use of evaluation and testing for educational improvement and make recommendations based on that examination for district guidance.&dquo; The primary purpose of the Evaluation Design Project, then, is 10 describe and analyze the management of testing and evaluation activities in those districts that have indicated that it is their intention to link testing and evaluation with instructional improvement. We begin with the following assumption: If tests and evaluations that do not originate with teachers are to influence teachers’ thinking and change institutional activities in classrooms in any directed or sustained fashion, then the district office’s system for supporting the use of test scores and of evaluation findings is a vital factor. We are therefore examining the management subsystems within districts that link testing and evaluation with instructional improvement. We are also examining the effect these subsystems have on teachers’ thinking and classroom activities.

Program Evaluation and Clinical Sociology

Chapters with straightforward titles such as this often start out with straightforward definitions of terms. I rarely find such definitions helpful. They are either common sense and therefore unnecessary or very technical and therefore counterintuitive.

A case study: The review of the Los Angeles Bureau of Jewish Education

About six months after my evaluation and consulting involvements with the Los Angeles Bureau of Jewish Education had ended, I was invited to speak about my experiences to a group of Directors of Bureaus of Jewish Education from around the country. I made my presentation to a rather silent group seated in high-backed chairs around a conference table in a wood paneled Iibrary. I concluded my talk by saying that what I saw as a successful evaluation-the Review of the L.A. Bureau of Jewish Education-was possible only because of the high energy, passionate commitment and good ideas of all the people who had been involved. I concluded with the same statement that ends this article. “I believe that the utilization-focused evaluation we did paid off.” I then asked several of these people from the Los Angeles Bureau, whom I had asked to be panehsts with me, to make their own comments on the Review. The newly appointed Director of the L.A. Bureau said that, from his perspective, the best thing about the evaluative Review that I had done was that the Bureau now had a mandate to carry out a defined, circumscribed set of tasks and did not any longer need to be excessively responsive to the pleading of special interest groups. The worst thing about the evaluation, he said, was the emotional “aftertaste” it left: The organization’s inner workings had been exposed to public view. The second panelist, a BJE staff member, said the best thing about the evaluation was that it was cathartic for himself and for others on the staff. It had given them the opportunity to be heard, to react to past practice and suggest future activities; but, he said, the evaiuation Iasted too long, and work was interrupted for an entire year because the spotlight was turned up so high on everyone’s activities. The third panelist, a member of the oversight group which had initiated the study, said that simply getting the evaluation done gave everyone in his group hope that something constructive in Jewish education could really happen; but, on the other hand, he asked, how come there are not very many large-sized, visible changes yet? The audience of Bureau Directors listened carefully and asked thoughtful questions. Was Jewish education in Los Angeles or was the BJE as an organization really better off after the evaluation than before? Why were recommendations based on empirical evaluations done by an independent outsider better than recommendations based on the judgments of knowIedgeabie persons such as Jewish educational experts or lay policy makers? Was evaluation worth the money in the long run? Maybe the money should have been better spent on program or on staff? It seemed to me that many of the questioners were imagining themselves being evaluated and not liking the fantasy. Several were as skeptical about the value of the evaluation after my talk as they had been before it began. The evahiation of an evaluation clearly depends, as we say in California, on who you are, where you’re coming from and where you’re going.

Evaluation Design Project

to all schools receiving grants so that they know (a) that an evaluation is required, and (b) to ensure uniform reporting procedures. (3) Staff development is a major aspect of an integration plan. The ultimate success of staff training is the degree of implementation in the classroom. Procedures to monitor this area need to be built into the program. (4) Bilingual instruction in Spanish was provided for teachers at schools affected by NES/ LES students. The time it takes for adults to learn a second language varies from person to person. Thus, some provision for continued training of individuals who fail to master the language in a set period of time, but still wish to continue, should be made. (5) The coordination and management of all components requires that a dialogue be maintained to ensure that duplication of effort in collecting data does not occur and, as a consequence, that school personnel are not deluged at the end of the year with requests for similar information.

Book Reviews: New Technique for Evaluation. By Nick Smith (Ed.). Sage Publications, Beverly Hills, CA

Campbell, D. T., & Erlebacher, A. E. How regression artifacts in quasi-experimental evaluations can mistakenly make compensatory education look harmful. In J. Hellmuth (Ed.), Compensatory education: A national debate (The Disadvantaged Child, Vol. 3). New York: Brunner/Mazel, 1970. Cook, T. D. The potential and limitations of secondary evaluations. In M. W. Apple, H. S. Subkoviak, & J. R. Lufler (Eds.), Educational evaluation: Analysis and responsibility, pp. 155-234. Berkeley, CA: McCutchan, 1974. Cook, T. D. et al. Sesame Street revisited. New York: Russell Sage Foundation, 1975. Elashoff, J. D., & Snow, R. E. (Eds.). Pygmalion reconsidered. Worthington, OH: Charles A. Jones, 1971. Glass, G. V Primary, secondary, and meta-analysis of research. Educational Researcher, 1976, 5(10), 3-8. Hopkins, K. D.Theunit of analysis: Group means versus individual observations. Anteri-