Blaine R. Worthen

Relative Effectiveness of Personalized and Form Covering Letters in Initial and Follow-up Mail Surveys

Nonresponse is a serious threat to validity in mail surveys, prompting the need for research on techniques for increasing response rates. The relative effectiveness of personalized versus form covering letters in obtaining responses in a mail questionnaire survey was investigated for both initial and follow-up mailings. A questionnaire eliciting opinions about content in college courses in measurement was sent to 1000 Utah school teachers, randomly divided into two groups, with 500 receiving personalized letters and 500 receiving form letters. Nonrespondents in each group were randomly divided, with half receiving personalized follow-up letters and half receiving form follow-up letters. Using nondirectional hypothesis tests, no statistically significant differences were found in the relative effectiveness of personalized or form letters in either initial or follow-up mailings. These findings do not support the widely held belief that personalized covering letters improve response rates.

Critical Challenges Confronting Certification of Evaluators

There are many clear-cut reasons why it would be desirable to have a certification process for evaluators, both to help improve the practice of evaluation and to further the field’s maturation into a full-fledged profession. But there are also many strong arguments against attempting to launch an evaluator certification system, at least at this time. The arguments for and against AEA attempting to set up an evaluator certification system are discussed, including two contextual changes that have led to a significant shift in my thinking on this topic over the last three decades. Also presented are four major challenges that would need to be overcome in any effort to develop a viable evaluator certification system. In light of these challenges, I argue that the development and implementation of a certification process can succeed only if pursued patiently and incrementally, with adequate time for AEA to test the feasibility of each of the definitional and procedural processes that would need to undergird such a system. Finally, an agenda of steps is suggested that AEA might take to determine if our field and our collective wisdom have matured sufficiently for us to succeed in such an important undertaking.

The impact of integrated learning system implementation on student outcomes: Implications for research and evaluation

Abstract Recently, studies have been initiated to evaluate computer-based integrated learning systems (ILSs). Among those characterized by good scientific design, few have found the ILS to have positive effects on student achievement or attitudes. These findings might lead one to believe that the ILS has little impact on valued student outcomes. However, in a large national study partially reported in this chapter, it was shown that there may be another explanation for such findings, namely that in most schools ILSs are not being implemented effectively. Analyses of effects of level of implementation on outcome measures revealed that with weak implementation there is little if any positive effect of the ILS and sometimes even a negative effect, whereas with strong implementation, the ILS had a positive effect on student outcomes. The authors argue that evaluation of an ILS before its implementation fully meets the vendors implementation standards is premature at best and may seriously underestimate the potential effectiveness of the ILS. The authors suggest why school implementation of ILSs may not be optimal, and outline implications of these findings for future ILS research and evaluation.

The changing face of educational evaluation

In 1965, educational evaluation was often viewed as merely a nuisance imposed on schools receiving federal funds for special projects. During the ensuing quarter century, it has permeated the field of education, becoming an instrument of public policy

Class Level as a Possible Mediator of the Relationship Between Grades and Student Ratings of Teaching.

Abstract Although the research literature investigating the relationship between grade awarded to students and students’ evaluations of teaching performance is voluminous, very few studies have examined the grade‐rating relationship according to level of student. The present study examined correlations between mean instructor rating and mean class grade for all course evaluations (N = 625 classes) at Utah State University during an academic quarter. In lower‐division (courses 100–299) and upper‐division (courses 300–599) undergraduate classes, correlations between grades and ratings of faculty were of an expected direction and magnitude (0.29 and 0.28, respectively); however, the grade‐rating correlation for graduate classes (courses 600 +) was — 0.20. It is speculated that graduate students are both better students and more critical evaluators of instruction, but replication and extension with different samples are needed before this tentative explanation can be accepted with confidence.

AEA members’ opinions concerning evaluator certification

Professional certification is sometimes advocated as a means of assuring consumers that they are getting someone who is skilled and knowledgeable within their trade. Certification is also sometimes viewed as advantageous for enhancing professions’ prestige, promoting professionalism, improving academic programs, and helping to define a profession. Without the acceptance by an organization’s members, however, any efforts to implement a certification process are likely instead to be divisive and dysfunctional. This article presents the results of a survey carried out by a recent AEA Task Force on certification.

Do Personalization and Postage Make a Difference on Response Rates to Surveys of Professional Populations

The purpose of this study was to test the effectiveness of two techniques for increasing response rates to mailed questionnaires, namely, type of covering letter salutation (handwritten vs typed) and return postage (metered vs nonmetered). Surveys were sent to 300 randomly selected members of the National Council for Measurement in Education. A 2 × 2 factorial analysis of variance showed typed salutations were more effective than handwritten salutations in eliciting responses from this professional population. No differences in response rates were found between metered and nonmetered return envelopes.

Teachers' Perceptions of the Importance of Educational Measurement.

AbstractThe adequacy of undergraduate educational measurement courses in preparing classroom teachers is difficult to ascertain due to limited and contradictory research on this topic. The present study was aimed at determining teachers’ perceptions of the importance of various educational measurement topics taught in undergraduate courses. A questionnaire was sent to 1,000 Utah school teachers; respondents were divided into subgroups based on length of teaching experience and grade level taught. The teachers rated the importance of 27 educational measurement topics. Results indicate that teachers perceived most of the 27 topics as at least moderately important. Similar ratings for most topics by the different subgroups suggest that a single undergraduate course could be developed to meet the needs of most teacher education students. A few significant subgroup differences were found.

Adversary Evaluation and the School Psychologist.

Abstract This article describes adversary evaluation and explains its use in evaluation of school-based programs by school psychologists. The authors discuss the history of adversary evaluation, the various models which have been used, and major research findings. Three diverse examples of adversary evaluation are presented, along with criteria for evaluating such models. In addition the article raises some potential concerns regarding the use of adversary evaluation, and suggests potential roles that school psychologists might play within this paradigm.

How can we call Evaluation a Profession if there are no Qualifications for Practice

Anyone making a living as an evaluator has probably found it difficult on occasion to explain to others just what the occupation of evaluation entails. Indeed, some people I meet seem a bit bewildered how persons other than Ralph Nader could earn their income merely by “evaluating things” and reporting their findings. Yet questions about how one can earn a living simply by evaluating things are not nearly as taxing as the more insightful and penetrating questions of those who can easily imagine one doing evaluations for a living (doubtlessly conjuring up referent images of tax auditors, quality control experts, meat inspectors, theater critics and the like), but probe deeper by asking, “Are you required to have to have a state license to practice evaluation?,” “Are there university programs to train evaluators?,” “Are evaluators accredited?,” “What qualifications do members of your professional association have to meet?” And when all these are answered in the negative, the obvious question becomes: “Then is evaluation really a profession?” It is these and related questions that I explore here, along with a few comments about challenges we confront and options we might pursue if we desire evaluation ever to become a mature, widely recognized, and understood profession.2