Michael D. Prislin

A pilot randomized, controlled trial of a longitudinal residents-as-teachers curriculum.

Purpose To determine whether a longitudinal residents-as-teachers curriculum improves generalist residents teaching skills. Method From May 2001 to February 2002, 23 second-year generalist residents in four residencies affiliated with the University of California, Irvine, College of Medicine, completed a randomized, controlled trial of a longitudinal residents-as-teachers program. Thirteen intervention residents underwent a 13-hour curriculum during one-hour noon conferences twice monthly for six months, practicing teaching skills and receiving checklist-guided feedback. In a 3.5-hour, eight-station objective structured teaching examination (OSTE) enacted and rated by 15 senior medical students before and after the curriculum, two trained, blinded raters independently assessed each station with detailed, case-specific rating scales (rating scale reliability = 0.96, inter-rater reliability = 0.78). Results The intervention and control groups were similar in academic performance, specialty distribution, and gender (χ2 = 0.434, p = .81). On a five-point Likert scale (5 = best teaching skills), intervention and control residents showed similar mean pretest OSTE scores (2.83 vs. 2.88, p = .736). The intervention group improved their mean overall OSTE scores 22.3% (more than two standard deviations) from 2.83 (pretest) to 3.46 (post-test; p < .0005; 95% CI 0.53 to 0.72). Intervention residents also improved significantly on six of eight OSTE stations. Within the control group, no pretest-to-post-test change achieved statistical significance. Mann–Whitney and Wilcoxon signed-rank tests confirmed these results. Conclusions Generalist residents randomly assigned to receive a 13-hour longitudinal residents-as-teachers curriculum consistently showed improved OSTE scores. Future research should clarify which aspects of residents-as-teachers curricula most effectively improve educational outcomes.

Reliability and validity of an objective structured teaching examination for generalist resident teachers.

For more than a decade, medical educators have employed standardized students and objective structured teaching examinations (OSTEs) to evaluate the clinical teaching skills of medical faculty. Recent studies have set more rigorous standards of validity and reliability for these performance-based assessments. Some have begun using OSTEs for resident physicians, whom the Liaison Committee for Medical Education (LCME) and others increasingly recognize as critically important teachers for medical students and peers. OSTEs hold great promise for rapid and rigorous evaluation of clinical teaching skills and of new approaches to teacher training. For resident teachers and clinician–educators, it may require years to accumulate sufficient numbers of ‘‘real life’’ teaching evaluations for reliable teaching assessments. OSTEs can truncate this timeline to produce meaningful, prompt teaching assessments for important decisions such as resident evaluations or faculty promotions. OSTEs also facilitate outcomes-based educational research, as well as program evaluation of novel initiatives to improve teaching skills. A major challenge in OSTE practice lies in developing accurate rating scale or checklist instruments appropriate for carefully assessing teaching performance on OSTE stations. Although earlier research has delineated characteristics of exemplary clinical teachers, it remains a challenge to translate this body of knowledge into sensitive and specific assessment instruments. Educational researchers have developed and studied numerous instruments, some tailored to evaluating residents’ teaching skills. The SFDP-26, a 26-item rating scale based on the seven teaching constructs of the Stanford Faculty Development Program (SFDP), is one of the best-validated rating scales available to evaluate clinical teachers. The emerging OSTE literature has yet to address definitively the issue of selecting between dichotomously scored checklists and multi-point rating scales for best assessment of teaching performance. Research offers clearer support for using standardized students portrayed by senior medical students, who in non-OSTE studies have shown themselves to be capable evaluators of teaching. The related literature on objective structured clinical examinations (OSCEs) sheds light on some of these issues. Senior medical students who act as standardized patient examiners for learners may benefit by improving their own communication skills, suggesting that standardized students may improve their own teaching skills. The OSCE literature manifests more controversy over the choice between checklists and rating scales, with a minority of OSCEs featuring multi-point rating scales, although both formats can be used successfully. The purpose of our study was to develop and assess reliability and validity for an eight-station OSTE with case-specific, behaviorally-anchored rating scales, all developed specifically for resident teachers. This OSTE is the primary outcome measure for Bringing Education & Service Together (BEST), an ongoing randomized, controlled trial of a longitudinal residents-as-teachers curriculum at the University of California, Irvine (UCI). We hypothesized that our OSTE would demonstrate acceptable reliability and validity when used to evaluate generalist residents’ clinical teaching skills before and after a pilot administration of the BEST curriculum. Method

Can There Be a Single System for Peer Assessment of Professionalism among Medical Students? A Multi-Institutional Study

Purpose Peer assessment is a valuable source of information about medical students’ professionalism. How best to facilitate peer assessment of students’ professional behavior remains to be answered, however. This report extends previous research through a multi-institutional study of students’ perspectives about system characteristics for peer assessment of professionalism. It examines whether students from different schools and year levels prefer different characteristics of peer assessment to assess each other candidly, or whether a single system can be designed. It then identifies the characteristics of the resulting preferred system(s). Method At the beginning of academic year 2004–2005, students (1,661 of 2,115; 78%) in years one through four at four schools replied to a survey about which peer assessment characteristics—related to, for example, who receives the assessment, its anonymity, and timing—would prevent or encourage their participation. Multivariate analysis of variance was used to detect differences among institutions and students from each year level. Results Students across year levels and schools generally agreed about the characteristics of peer assessment. They prefer a system that is 100% anonymous, provides immediate feedback, focuses on both unprofessional and professional behaviors, and uses peer assessment formatively while rewarding exemplary behavior and addressing serious repetitive professional lapses. The system, they emphasize, must be embedded in a supportive environment. Conclusions Students’ agreement about peer-assessment characteristics suggests that one system can be created to meet the majority of students’ preferences. Once implemented, the system should be monitored for student acceptability to maximize participation and to determine the formative and summative value of the process.

Using standardized patients to assess medical students' professionalism.

STANDARDIZED PATIENTS—WILL THE QUESTIONS NEVER END? Moderator: Craig Scott, PhD Using Standardized Patients to Assess Medical Students’ Professionalism ´ LIE, JOHANNA SHAPIRO, JOHN BOKER, and STEPHEN RADECKI MICHAEL D. PRISLIN, DESIREE The subject of professionalism is currently engendering great inter- est within the medical education community. Concern exists that conditions within the health care delivery environment threaten established standards of professional behavior, and, perhaps more insidiously, that the medical education experience itself may be negatively influencing the development of physicians’ profession- alism. 1–3 As a consequence, much energy has recently been directed toward defining competencies that reflect professionalism and in creating corresponding curricula that will foster learning in this domain. 4–6 However, having instruments that can accurately measure the attainment of professionalism remains an elusive goal. 7–9 This study examines the utility of standardized patient-based assessments of professional characteristics. Comparisons are made with other mea- sures of professionalism, such as faculty evaluation, performance on a written self-reflective exercise, and student-reported participation in community service activities. Method This study was conducted at the University of California, Irvine (UCI), College of Medicine. Participants were students completing the year two patient–doctor course during the 1999–00 academic year. This course represents the second segment of a vertically in- tegrated four-year course sequence in professional skill develop- ment. The year two segment focuses on patient–physician com- munications, physical diagnosis, and the development of basic clinical reasoning skills. Eight core clinical modules are linked to topics concurrently being taught in the year-two pathology, path- ophysiology, and pharmacology courses. Each module begins with a standardized patient interaction, followed by generation of learn- ing issues within small tutorial groups. Mid-module activities in- clude topical didactic presentations and physical diagnosis instruc- tion. Each module concludes with a wrap-up session in which the diverse learning activities are tied together through small-group dis- cussion of the original learning issues. These discussions typically feature a heavy emphasis on patient–physician communication and professional behavior. Assessments of students occurring during the course consist of a written final examination, structured written evaluations completed by the faculty group leaders, and an appraisal of clinical skills. The clinical skills appraisal for 1999–00 consisted of a three-station standardized-patient–based examination. The cases were a patient presenting with fatigue, a patient presenting with upper gastroin- testinal and chest discomfort, and a patient presenting with tran- sient neurologic deficits. The first two cases each entailed 25 minutes and the third case entailed 35 minutes of patient contact. Each station required students to perform a history and physical examination. In addition, students performed a rapid computer- based literature search following the initial encounter with the neu- rology case, written and oral clinical presentations following the fatigue case, and a written reflective essay, pertaining to students’ reactions to a poem describing a 39-year-old man experiencing an acute myocardial infarction and sudden death, following the upper gastrointestinal and chest pain case. Each standardized patient en- counter included assessments of history and physical exam perfor- mance based on a checklist and assessments of communication S90 skills and professionalism using a rating scale. The rating scale for communication skills used in this study was a modification of the Communications Skills Form developed at East Tennessee State University by Forrest Lang, MD, to assess patient-centered com- munications as evaluated by standardized patients. It is based upon an instrument developed by the American Board of Internal Med- icine to assess patients’ satisfaction. The rating scale includes six items relating to communication that are reported here as the cu- mulative communication score; a single item relating to overall professional competence; and a single item relating to overall stan- dardized patients’ satisfaction. The professionalism scale used for this study was constructed based upon the work of Arnold and colleagues, 9 and consisted of three items: one that allowed stan- dardized patients to rate students’ knowledge and competence, one that rated students’ integrity, and one that rated students’ altruism. Taken together, these three items are reported as the cumulative professionalism score. The specific rating scale items for commu- nication and professionalism are presented in List 1. Both the communication and the professionalism rating instru- ments used five-point Likert scales with the following specific an- chors: 5—outstanding; 4—very good; 3—good; 2—needs improve- ment; 1—marginal. Therefore, the maximum achievable scores were: cumulative communications—30 points, cumulative profes- sionalism—15 points, professional competence—5 points, and overall satisfaction—5 points. Standardized patients received de- tailed verbal and written instructions on how to complete the com- munication scale, including descriptive anchors for performance at varying levels of competence, and were observed rating perfor- mances using practice tapes before participating in the examina- tion. In terms of the specific professionalism items, the standardized patients were instructed to respond based upon their own personal perceptions of the students. Fourteen standardized patients were used during the course of the examination: seven for the fatigue case, three for the chest-discomfort case, and four for the neurology case. Faculty evaluations of students’ performances during the patient– doctor II course were based on an 11-item rating scale in which one item assessed whether the student ‘‘demonstrates professional behavior.’’ This evaluation also used a five-point Likert scale in which five represented outstanding, four represented above ex- pected, three represented at expected, two represented below ex- pected, and one represented problematic performance. Hence the maximum possible score for faculty professionalism ratings was five points. Evaluating faculty received verbal instructions regarding evaluating students’ performances during faculty development ses- sions. Evaluation of the professionalism item focused on students’ citizenship and academic honesty, team participation, and inter- actions with standardized patients during the interview sessions. The essay was scored by one of the study’s authors for emotional content and problem-solving capacity using a modification of a method described by Pennebaker and colleagues. 10 Subscale scores relating to empathy and positive coping attitudes were used as mea- sures reflecting students’ expressions of professional attributes. The scores students received represented a sum of these two subscales. Students’ descriptions of their participation in community ser- vice activities were elicited by means of a written survey distributed at the conclusion of the skills-appraisal exercise. Participation was scored as ‘‘did’’ or ‘‘did not’’ participate. A CADEMIC M EDICINE , V OL . 76, N O . 10 / O CTOBER S UPPLEMENT 2001

The generalist disciplines in American medicine one hundred years following the Flexner Report: a case study of unintended consequences and some proposals for post-Flexnerian reform.

Abraham Flexners analysis of U.S. medical education at the turn of the 20th century transformed the processes of student selection and instruction, the roles and responsibilities of faculty members, and the provision of resources to support medical education. Flexners report also led to the nearly universal adoption of the academic medical center as the focal point of medical school teaching, research, and clinical activities. In this article, the authors describe the effects of the dissemination of this model and how the subsequent introduction of public funding for research and patient care transformed academic health centers and altered the composition of the physician workforce, resulting in the proliferation of specialties. They also describe how these workforce changes, along with the evolution of health care financing during the late 20th century, have led to a system that affords the most scientifically advanced and potentially efficacious care in the world, yet so profoundly fails to ensure affordability and equitable access and quality, that the system is no longer sustainable. The authors propose that both health care system reform and medical education reform are needed now to restore economic viability and moral integrity, and that a key element of this process will be to rebalance the generalist and specialist composition of the physician workforce. They conclude by suggesting that post-Flexnerian reform of medical education should include broadening the scope of criteria used to select medical students and reshaping the curriculum to address the evolving needs of patient care during the 21st century.

What do medical students learn from early clinical experiences (ECE)

What are the common learning themes perceived by medical students during ECE with varying practice settings and patient profiles? Retrospective qualitative and quantitative analyses of structured descriptive reports completed by one class (nu2009=u200992) for 895 observed patient encounters identified common learning themes. Identified themes were examined by practice setting and patient characteristics. Student response rates were 85 to 94% across settings. Fifty-five percent of ECE were in outpatient settings. Chief complaints were predominantly medical (67%); only 20% represented psychosocial and 8% preventive care, respectively (5% were ambiguous). The five most common learning themes (out of 13 themes coded) were communication (>50%), procedures/time management, cross-cultural issues, feeling useful as a student, and presenting medical problems. Cross-cultural issues were addressed mainly in community settings. Negative learning occurred only rarely (<3%). Student observations from ECE can be used by course managers to design effective early clinical exposures to address specific course learning objectives.

On-site Colposcopy Services in a Family Practice Residency Clinic: Impact on Physician Test-Ordering Behavior, Patient Compliance, and Practice Revenue Generation

Background: Using colposcopy as a model, we examined the impact of introducing a new diagnostic. technology into the ambulatory primary care setting. Methods: Records of patients with abnormal findings on Papanicolaou smears were reviewed from three study periods: 1 year before, 1 year after, and 5 years after initiation of on-site colposcopy services. Data analyzed include physician management decisions, site of colposcopic service, and patient compliance. Practice revenue estimates were based upon patterns of physician management and patient compliance found during each study period. Results: Management of low-grade squamous intraepithelial lesions varied during each study period. By period 3, however, most patients were undergoing colposcopy (P = 0.03). High-grade squamous intraepithelial lesions were uniformly managed with colposcopy during all study periods (P < 0.001). Introduction of on-site colposcopic services resulted in a rapid shift to the on-site location for evaluation of low-grade squamous intraepithelial lesions and a more gradual shift to the on-site location for evaluation of high-grade squamous intraepithelial lesions. Patient compliance was not affected by the introduction of on-site services. On-site colposcopy resulted in a nearly 100 percent transfer of revenue to the practice, but the economic benefit was quite modest. Conclusions: Although offering on-site colposcopy services might have had some impact on physician management of low-grade squamous intraepithelial lesions, the lack of benefit regarding patient compliance, the relatively small patient volume for this procedure, and its modest impact on practice revenue cause us to question the value of including colposcopy in everyday practice.

Literary narratives examining control, loss of control and illness: Perspectives of patient, family and physician

This article examines how three literary narratives portraying the psychological effects of illness deal with issues of control in the doctor-patient-family relationship. Specifically explored are how patient, physician, and family members cope with the feelings of vulnerability and loss of control brought on by illness; and how each seeks to gain and maintain a sense of control in this coping process. Suggestions are offered for uses of these narratives in medical education. By exploring themes of control in the doctor-patient relationship, physicians develop greater insight and empathy that in turn can lead to improved outcomes for both patient and doctor.

Educational correlates of students' perceptions of learning in longitudinal ambulatory primary care clerkships.

No abstract available.

Do Simulation-Based Skill Exercises and Post-Encounter Notes Add Additional Value to a Standardized Patient-Based Clinical Skills Examination?

Background. Standardized patient (SP) clinical assessments have limited utility in assessing higher-level clinical competencies. This study explores the value of including simulation exercises and postencounter notes in an SP clinical skills examination. Methods. Two exercises involving cardiac auscultation and ophthalmic funduscopy simulations along with written post encounter notes were added to an SP-based performance examination. Descriptive analyses of students performance and correlations with SP-based performance measures were obtained. Results. Students abilities to detect abnormalities on physical exam were highly variable. There were no correlations between SP-based and simulation-derived measures of physical examination competency. Limited correlations were found between students abilities to perform and document physical examinations and their formulation of appropriate differential diagnoses. Conclusions. Clinical simulation exercises add depth to SP-based assessments of performance. Evaluating the content of post encounter notes offers some insight into students integrative abilities, and this appears to be improved by the addition of simulation-based post encounter skill exercises. However, further refinement of this methodology is needed.