Kathleen A. Scanlan

Radiology faculty evaluation: Bi-institutional implementation of an experimental appraisal instrument

RATIONALE AND OBJECTIVES The authors evaluated the reliability and validity of an experimental radiology faculty appraisal instrument. MATERIALS AND METHODS Residents from the University of Wisconsin (UW) and Indiana University (IU) were asked to use a previously developed behaviorally based, 53-item experimental faculty appraisal instrument. Twenty UW residents evaluated 29 UW faculty members and 37 IU residents evaluated 31 IU faculty members by using the experimental instrument. Residents also evaluated faculty by using their institutions existing appraisal instrument. RESULTS Correlations between existing and experimental forms were .69 and .87 for UW and IU, respectively. Existing form reliabilities were .89 and .94 and experimental form reliabilities were .98 and .98 for UW and IU, respectively. Experimental form length was reduced to 30 items by eliminating the questions that correlated the least with section scores. Reliabilities of scores on the shortened form were .97 and .98 and correlated .65 and .88 with scores on the longer form for UW and IU, respectively. CONCLUSION Ratings obtained with the existing forms correlated substantially with the experimental form, attesting to the experimental forms validity. Shortening the experimental form had a minimal effect on the reliability and validity of the data obtained. The behavior-based form was used to rate behaviors that residents believed discriminated between effective and ineffective instructors, enabling an objective and relevant assessment to be made.

Development of a radiology faculty appraisal instrument by using critical incident interviewing

RATIONALE AND OBJECTIVES To develop a valid and reliable radiology faculty appraisal instrument based on scientific methods. MATERIALS AND METHODS Fifteen radiology residents participated in critical incident interviewing. During a 1-hour interview, a resident was asked to describe five incidents each of effective and ineffective faculty behavior. Two investigators independently listened to the tape-recorded interviews, and two different investigators sorted the incidents into broad categories. A faculty appraisal instrument was developed by listing similar incidents under broad categories. A five-point rating scale was applied to each item. Content validity was assessed by resident and faculty critique of the appraisal instrument. RESULTS A total of 168 incidents of faculty behavior were generated. The frequency with which similar incidents were reported was recorded. The most common behaviors reported were related to staff expertise and teaching. Interjudge reliability was good, as determined by computing K indices of agreement (overall K = 0.59). There was good agreement regarding instrument content validity among residents but not among faculty. CONCLUSION Residents supported the use of the new appraisal instrument, but further tests of validity and reliability and faculty acceptance of the instrument will determine its usefulness as a tool for monitoring faculty teaching performance and making decisions regarding faculty promotion.