John C. Ory

Assessing Faculty Work: Enhancing Individual and Institutional Performance

EXPANDING THE PURPOSES AND GOALS OF FACULTY ASSESSMENT. The Current Status of Faculty Assessment. A New Perspective on Faculty Assessment and Development. SETTING EXPECTATIONS. Defining Faculty Work. Discussing Expectations. COLLECTING AND ORGANIZING EVIDENCE. Gathering Acceptable and Trustworthy Evidence. Establishing the Credibility of the Evidence. Constructing a Full Portrayal of Faculty Work. USING EVIDENCE IN FACULTY ASSESSMENT. Clarifying Appropriate Uses of Assessment Evidence. Enhancing Assessments Value to Individual Faculty Members. Enhancing Institutional Uses. METHODS OF COLLECTING EVIDENCE. Written Appraisals. Rating Scales and Checklists. Interviews. Observations and Videotaping. Indicators of Eminence, Quality, and Impact. Achievement and Outcome Measures. Records and Portfolios.

Faculty Perceptions of the Quality and Usefulness of Three Types of Evaluative Information.

The purpose of the study was to investigate faculty perceptions of student evaluative information collected by three methods: objective questionnaire items, open-ended questions, and group interviews. Faculty rated three simulated evaluation reports on their potential for accuracy, trustworthiness, usefulness, comprehensiveness, believability, interpretability, and value as information used for self-improvement and promotion purposes. Faculty, in general, regarded the evaluative information to be more credible, useful, and accurate for their own self-improvement than for promotion purposes. Faculty also desired more than one type of evaluative information regardless of the purpose of evaluation.

Evaluating Instructional Technology Implementation in a Higher Education Environment

For decades colleges and universities have experimented with various technological innovations to deliver instruction. Although not new in higher education, use of learning technology has greatly increased in recent years (Burnaska, 1998). This increased use brings an increased need for understanding the methodologies and approaches best suited to the evaluation of learning technologies in higher education. Here we first review the literature and describe the methods used in a myriad of evaluation studies in this area. Our primary purpose, however, is to describe our experiences evaluating a campus-wide learning technology effort (the SCALE Project) at the University of Illinois at Urbana-Champaign. This evaluation spanned three years and used multiple methods. More importantly, it provided some challenges from which others may learn and benefit.

Changes in evaluating teaching in higher education

We evaluate teaching in higher education more today than ever before. In the last 30 years teaching evaluation has evolved from its primary reliance on a chairs informal assessment to a formal, comprehensive, and systematic approach. Colleges and universities are using evaluation systems that incorporate data collected from multiple sources using multiple methods to satisfy multiple institutional purposes. Additionally, the growing history of teaching evaluation research has greatly improved the quality of the instruments and methods used to collect evaluative information.

Assessment Activities at Large, Research Universities.

Assessment and outcomes measurement is receiving considerable attention in higher education nationwide. But what are colleges and universities doing and calling assessment? To answer these and other questions, a telephone survey was conducted with a sample of large, research universities. Responses to the survey are presented and discussed in this paper. Results indicated limited university involvement in activities commonly associated with the current assessment “movement,” i.e., measures of added value or student progress and growth. Instead, much of the reported activities are “traditional” testing and evaluation activities such as placement testing and student evaluation of instruction. In sum, assessmentis evaluation at large, research universities.

Meta-assessment: Evaluating assessment activities

Given the breadth and depth of assessment activity in higher education today there is a need for a set of standards to follow in its conduct. This paper briefly describes theStandards of the Joint Committee (1981) developed for educational evaluation and demonstrates their application in conducting assessment and meta-assessment activities.

Faculty Development and Achievement: A Faculty's View

A conceptual framework of career development based on professional rank was examined for its usefulness in describing three qualitatively different stages of career development of faculty. Sixty faculty at a large university were interviewed to obtain information about their aspirations, motivation, satisfaction and external expectations. Faculty interviewed generally progressed through a series of stages but the diversity of their professional life style increased with each rank. Implications of using rank to interpret faculty career paths were discussed.

End-of-course and long-term retention outcomes for mastery and nonmastery learning paradigms

Using both end-of-course achievement outcomes and long-term cognitive retention as criteria, the present study provides comparative data on the effectiveness of a mastery and nonmastery approach to instruction. Differential effects across taxonomic levels were assessed for both criteria. The results indicated that mastery students performed significantly higher than non-mastery students for end-of-course outcomes at the highest taxonomic level and equally well for knowledge, comprehension, and application level outcomes. Retention differences were found for knowledge level outcomes only, with mastery students demonstrating significantly greater retention performance. The feasibility and desirability of implementing a learning for mastery paradigm in a single course at the college level are discussed relative to the magnitude of the present results.

Student use of and attitudes about on-campus ALN

In March of 1995, the Sloan Center for Asynchronous Learning Environments (SCALE) was established at the University of Illinois at Urbana-Champaign (UIUC). SCALE is charged with organizing and directing a three year project to promote the use of asynchronous learning networks (ALN) at UIUC. Faculty involved in the Sloan Center are restructuring their undergraduate courses to incorporate various computer techniques associated with ALN. These ALN techniques include network-based access both to learning materials (e.g., multimedia tutorials, information on the World Wide Web) and to people (via conferencing software such as FirstClass or PacerForum). Computer conferencing was the primary ALN method used by faculty in the first semester. A comprehensive evaluation is being conducted of all SCALE activities. One aspect of the evaluation has been a survey of student use of and attitudes about ALN activities. In Fall 1995 the survey administered in SCALE courses focused on the most often used ALN application at the time-computer conferencing. In subsequent semesters two different surveys were used, one for courses using computer conferencing, and a second survey for courses incorporating Web-based activities. The purpose of this paper is to present results of the student surveys for the first three semesters of project.

Response-Mode Variation Effects On Affective Measures

The study investigated the effect of response mode variation (the use or nonuse of separate answer sheets) on subject responses (N = 1017) to affective instruments with respect to both instrument and subject characteristics. Instrument characteristics examined included attitudinal versus personality scale composition, scale items written in first- versus third-person and variations in response actions (circling, writing or darkening response). Subject characteristics investigated were individual differences in cognitive style (field dependence-independence) and school grade level (junior high, high school or college). Results indicated significant score differences across scales with and without answer sheets.