Walt Haney

The Fractured Marketplace for Standardized Testing

List of Figures and Tables. Preface. 1. Introduction. 2. The Testing Industry. 3. The Extent of the Marketplace for Tests. 4. Social Investment in Educational Testing. 5. Forces behind the Testing Marketplace. 6. Spin-Offs from the Testing Industry. 7. Test Quality and the Fractured Marketplace for Testing. 8. Meding the Fractured Marketplace. References. Appendices. Index.

The Evolution of Ethical and Technical Standards for Testing

Efforts to develop standards for tests and testing practices have a long history. The purpose of this chapter is to identify these various efforts to develop test standards as well as to document the nature of the changes in these standards over the years. Of special interest in this chapter are the three major revisions of the APA’s Ethical Standards of Psychologists, and APA, AERA, and NCME’s Standards for Educational and Psychological Tests. In a final section of the chapter, several suggestions are offered about the issues which the two sets of standards, ethical and technical, do and do not treat.

Cheating on Tests: Prevalence, Detection, and Implications for Online Testing

Publisher Summary This chapter discusses the detection and prevention of cheating on tests. Studies dating back to the 1960s indicate that cheating on tests is fairly common, being admitted to by some 15–45 % of college students surveyed. As numerous forms of cheating on tests have been documented, numerous observers have noted that the problem of cheating on tests is addressed far less commonly than it occurs. This is unfortunate because the existence of statistical methods for detecting copying appears to be little known among instructors using multiple choice tests and examinations. It is found that students adapt to unproctored online testing over time by collaborating. While various forms of collaboration in online tests are used, the easiest and the most common way is for students to take the test together. In addition, it is helpful to employ a variety of indices available for monitoring answer concordance in tests. Time-based correlations, such as plots of unusual answer similarity versus an index of unusual temporal similarity, may be useful in detecting collaboration in online testing and can potentially detect collaboration when students have few incorrect responses.

Practice, Participatory Research and Creative Research Designs: The Evolution of Ethical Guidelines for Research

In this chapter we discuss dilemmas we have experienced as researchers within communities of action. We summarize briefly the guidelines and ethical standards for research with human subjects,1 standards that formed the core of our socialization into the ethics of behavioral research. Until recently, these ethical standards and guidelines have evolved among various occupational groups with relatively little attention to the intersection of research methodology and ethical considerations. We describe how federal guidelines on research with human subjects have evolved and some of the effects these guidelines have had on research within the professions. Our personal experiences with ethical issues in research beyond university walls provide examples for discussing some of the inevitable dilemmas encountered in participatory and action-oriented research for change. Specifically, we explore the meanings of informed consent when one is engaged with communities in struggles for justice and/or in challenges to injustice within dominant institutions on behalf of those who are not being well-served by these institutions. Our experiences suggest limitations in applying abstract ethical guidelines and standards to such cases. We elaborate several dilemmas raised by creative research designs characteristic of much outreach scholarship for ethical standards and guidelines of selected occupational groups. Also, in an appendix we provide a summary of additional resources, both in print and on the World Wide Web, regarding ethical issues on social research.

Making Sense of School Testing

Making sense of school testing programs nowadays is difficult for a variety of reasons. One aspect of testing in the nation’s schools nevertheless seems clear—the sheer volume of standardized testing is increasing. As Chris Popho wrote in 1985: Testing is on the increase! Business for commercial test publishers is up and new companies are eyeing the market, ready to jump into a business that looks like it is firmly standing on the up escalator. Nearly every large education reform effort of the past few years has either mandated a new form of testing or expanded uses of existing testing. (Pipho 1985,19)

The Role of Testing in Evaluations

Testing is closely tied to evaluation. Tests of some sort play a role in virtually all educational program evaluations; indeed, too often an “evaluation” is no more than a hasty analysis of whether test scores rose. Supervising or conducting evaluations requires an understanding of basic concepts and central issues in testing; such an understanding helps ensure that tests will not be misused as an overly simplistic “bottom line.” When tests are chosen carefully and interpreted appropriately in evaluations, test results can help answer the question, “Is this project making a difference?” Because testing is a complex technology, it is easy for program sponsors to assign concerns about how tests work or how they are constructed to the experts. But just as a wise patient would never undergo surgery without asking questions, those who intend to make use of test results need to pose pertinent questions about the costs, alternatives, and consequences of the testing decisions made in evaluations. This chapter will introduce some of the aspects of test use that ought to be looked into by those who commission and employ program evaluations. The chapter opens with an explanation of what a test is. Subsequently, two types of testing will be considered with respect to evaluation. Traditional forms of testing, such as the multiple choice test, will be discussed first. Other forms of testing, the “alternative” forms of assessment which have received much recent

The Follow Through Experiment: Summary of an Analysis of Major Evaluation Reports

The decision to use the very limited funds available for Follow Through... to initiate a program which will permit examination in depth of the consequence of different program approaches holds promise of inaugurating what could be literally a new era in governmental supportfor educational and social ventures, i.e., an era in which the knowledge and technical expertise of the educational specialist, the systems engineer, and the behavioral scientist are brought into harmony with the pluralistic value structure of our society.1-JOHN HUGHES

The Testing Industry

A definitive analysis of the testing industry in the United States describing how standardized testing grew from a fairly obscure academic enterprise in the late 19th century into a major commercial endeavor in this century has yet to be written. Over the last fifty years there has been a variety of treatments of various segments of the testing marketplace. A few analysts have provided relatively broad accounts of the industry (e.g. Holmen & Docter, 1972; Kohn, 1977; Fremer, 1989). Several people have written sketchy accounts of the history of standardized testing in the United States in general (Cronbach, 1975; Haney, 1981, 1984), in education (Resnick 1981; Madaus & Kellaghan, 1992), in psychology (Sokal, 1987), and in employment (Hale, 1982). A number of histories have been written on particular testing programs (e.g. Ruger’s 1975 history of the American College Testing Program; Peterson’s 1983 history of the Iowa Testing Programs; Hazlett’s 1974 and Greenbaum, Solomon & Garet’s 1977 accounts of the early history of the National Assessment of Educational Progress), or on particular testing organizations (e.g. Nairn & Associates’ 1980 history and critique of ETS; Sokal’s 1981 account of the founding of the Psychological Corporation; Valentine’s 1987 history of the College Board). Additionally, there have been a number of biographies of people important in the development of testing in the United States, such as Edward Lee Thorndike (Clifford, 1968), Lewis Terman (Chapman, 1980), G. Stanley Hall (Ross, 1972), James McKeen Cattell (Sokal, 1987b), and Walter Dill Scott (Mayrhauser, 1989).

Spin-Offs from the Testing Industry

Like the automotive industry, the testing industry has spawned a number of spin-off companies and endeavors that market products or services related to test preparation, construction, interpretation, administration, and scoring. Some of these endeavors accommodate examinees directly, others target test users in education, business, and clinical and counseling psychology, and still others are directed at people and institutions that are affected by test results. Spin-off products and services from the testing marketplace are highly diverse, ranging from publishers who provide books on how to make, take and interpret tests, to organizations aimed at promoting, and challenging the role of testing in American society. In this chapter, we examine three spin-offs from the testing industry, namely: The computer connection; Test preparation and coaching; and Honesty or integrity testing. We examine these three spin-offs for two reasons. First, each of these topics has garnered a fair amount of attention in its own right in recent years, and hence is worth examining. But a second and broader reason is that while each is an offshoot of the testing industry, each also suggests something about likely future influences on the role of testing in the United States.

Social Investment in Educational Testing

In this chapter we present ideas on analyzing the costs and benefits of standardized educational testing at the state and local level. The total value of resources devoted to educational testing is much larger than the direct, observable costs of developing, purchasing, and scoring standardized tests. There are significant indirect resource costs that should be included in any attempt to evaluate the social investment in local and state testing programs. In order to make such an analysis fruitful, planners and policy makers need to reorient the way they think about testing in two important ways. First, they should pay closer attention to the fundamental economic concept of marginal value. Second, testing and the time devoted to it should be considered as “inputs” into education, instead of merely as measures of educational “output”.