Donald P. Hoyt

The evaluation of faculty development programs

This article reviews literature pertinent to the evaluation of faculty development programs and presents data from several studies conducted at two institutions. These data were consistent with those previously reported in that faculty participants consistently expressed satisfaction with development services. In addition, one study found that most faculty members voluntarily took some action to improve their instructional effectiveness, though only a minority pursued these efforts in depth. Volunteers who worked intensively with a faculty development consultant improved more on objective measures of effectiveness than did those who were only superficially involved in improvement efforts; those who received no consultative assistance failed to improve significantly. Evidence from a final study provided a control for faculty motivation and led to the conclusion that improvement was contingent both on faculty desire to improve and on the availability of professional assistance.

Faculty research involvement and instructional outcomes

Faculty members in the natural-mathematical sciences (N=93) and in the social-behavioral sciences (N=90) were rated on “research involvement” by department heads and on “teaching outcomes” by students. Differences in teaching outcomes for members of the two disciplines confirmed several common stereotypes. Those heavily involved in research appeared to establish higher academic standards for students than did the relatively uninvolved, but the two faculty groups generally did not differ on student progress ratings. However, for four of ten criterion measures, a significant interaction occurred. Student ratings were positively related to the degree of research involvement for faculty members in natural-mathematical sciences but negatively related to research involvement on the part of social-behavioral scientists.

Interrelationships Among Instructional Effectiveness, Publication Record, and Monetary Reward.

Measures of teaching effectiveness, scholarly publication, and average salary adjustments were interrelated for a sample of 222 experienced college teachers. A moderate relationship was found between scholarly productivity and salary increases, and a more modest relationship existed between teaching effectiveness and salary. Scholarly publication and teaching effectiveness were independent, however.

Measurement of instructional effectiveness

AbstractThe problem of measuring instructional effectiveness was examined, and a rationale was offered for employing “student progress on relevant objectives” for this purpose. To assess such progress, it was suggested that instructor ratings of the importance of objectives be combined with student ratings of progress on these objectives.On the basis of this suggestion, data were collected from 708 undergraduate classes at Kansas State University. An analysis of these data resulted in the following conclusions:1.Faculty members appeared to make reliable judgments of the relative importance of these objectives.2.Student progress ratings were made with acceptable reliability when there were 20–25 raters. Reliability of the overall progress measure was satisfactory when only 10 raters were used.3.Students used some discrimination in rating progress on various objectives, but their ratings were also noticeably subject to the halo effect.4.An indirect test of the validity of class progress ratings yielded positive results. The proposed method of evaluating instruction appears generally feasible and useful. Its application would provide a practical approach to judging teaching success. Such an approach is essential before investigations can be undertaken of how teaching might be improved.

Academic Grades as a Predictor of Occupational Success among Engineering Graduates

The assumption that there is a direct relationship between academic achievement and occupational success was tested with a sample of bachelor degree-level engineering graduates who had 5 to 10 year...

The measurement of administrative effectiveness of the academic department head

A total of 103 academic department heads in four universities rated a set of 15 administrative activities as to their importance. Faculty members in these departments (totalN=1,333) used the same set of activities to rate both the importance they should be given by the department head and the effectiveness with which the head performed each set during the previous 12 months. Tests of reliability revealed that faculty ratings of both importance and performance were made with reasonable internal consistency. Three tests of construct validity showed that each of the three types of ratings were made with at least minimal validity. A principal components analysis of faculty ratings of performance suggested that the department head has three major types of responsibility: personnel management; departmental planning and development; and building the departments reputation.

Identifying and Adjusting for Biases in Student Evaluations of Instruction: Implications for Validity.

Student evaluations of teaching effectiveness are widely used for improvement of instruction and for administrative decision-making. Several kinds of contaminating variables may influence these ratings. Criteria for identifying such contaminants are set forth. Results of a search for extraneous, biasing variables at the high school level are reported. Of the several potential extraneous influences studied, student motivation/preference and class size meet the criteria. It is concluded that student rating forms should make provisions for adjusting for these and similar unfair influences.

USING DISCRIMINANT ANALYSIS TO PREDICT FACULTY RANK

Past attempts to predict the salary of faculty have been flawed by the use of rank, a potential source of bias, but removing rank from the equation seriously reduces prediction. This research studies the degree of bias in rank. Discriminant analysis produced a solution that correctly predicted rank 69% of the time for the 1083 faculty members used in the study. Among teaching and research faculty, there was a significant relationship between the direction of prediction error and gender. For women, the number of times they were predicted to have a rank higher than their actual rank exceeded the opposite condition by 2 1/2 times; for men, there was a slight tendency for the opposite. Methods were suggested for using the information administratively to correct problems. Further, recommendations are made to use the results to study salary-bias problems.

A Comparison of High School Student Ratings of Teaching Effectiveness with Teacher Self-Ratings: Factor Analytic and Multitrait-Multimethod Analyses

Two classes of each of four hundred high school teachers were rated for teaching effectiveness by the teachers and by their students using the same instrument. Separate factor analyses yielded highly similar underlying dimensions. Convergent validity coefficients were significant for all factors. These findings provide support for the validity of high school student ratings of instruction.

Identifying "Comparable" Institutions.

To be most useful in the resource acquisition and resource allocation processes, cost studies require the collection of detailed information from “comparable” institutions. This investigation reports the rationale, procedures, and outcomes of an attempt to identify institutions that were comparable to specific universities operating under the Kansas Board of Regents. The three-phase study involved (1) selecting states that were similar to Kansas on several key variables, (2) using information about breadth of academic programs and locale to identify public universities in those states that resembled a given Kansas institution on these features, and (3) using detailed information about enrollment, expenditure pattern, and academic emphases to measure the similarity of universities in each pool to a given Regents university. The system produced results that proved to be credible by three different tests and responded effectively to some issues involving the Kansas formula funding process. It is sufficiently flexible that it could be adapted to other states or universities whose rationale or assumptions differed from those employed in Kansas.