Louis N. Pangaro

How accurate are faculty evaluations of clinical competence

AbstractObjective:To determine the degree and sources of variability in faculty evaluations of residents for the American Board of Internal Medicine (ABIM) Clinical Evaluation Exercise (CEX). Design:Videotaped simulated CEX containing programmed resident strengths and weaknesses shown to faculty evaluators, with responses elicited using the openended form recommended by the ABIM followed by detailed questionnaires.Setting:University hospital.Participants:Thirty-two full-time faculty internists.Intervention:After the open-ended form was completed and collected, faculty members rated the resident’s performance on a five-point scale and rated the importance of various aspects of the history and physical examination for the patient shown.Measurements and Main Results:Very few of the resident’s strengths and weaknesses were mentioned on the openended form, although responses to specific questions revealed that faculty members actually had observed many errors and some strengths that they had failed to document. Faculty members also displayed wide variance in the global assessment of the resident: 50% rated him marginal, 25% failed him, and 25% rated him satisfactory. Only for performance areas not directly related to the patient’s problems could substantial variability be explained by disagreement on standards.Conclusions:Faculty internists vary markedly in their observations of a resident and document little. To be useful for resident feedback and evaluation, exercises such as the CEX may need to use more specific and detailed forms to document strengths and weaknesses, and faculty evaluators probably need to be trained as observers.

Faculty Development Seminars Based on the One‐Minute Preceptor Improve Feedback in the Ambulatory Setting

OBJECTIVE: While several models of medical student instruction in the ambulatory setting exist, few have been formally studied. We wished to assess the impact of a faculty development workshop based on the One-Minute Preceptor model on the amount and quality of feedback in the outpatient setting.DESIGN: Ambulatory teaching behaviors were studied during consecutive outpatient precepting sessions before and after 3 faculty development workshops. Student-teacher interactions were assessed using audiotapes of teaching encounters coded through qualitative techniques, and surveys of teacher, learner, and patient satisfaction.SETTING: Ambulatory internal medicine clinic in a tertiary care medical center.PATIENTS/PARTICIPANTS: Nine board-certified internist faculty preceptors and 44 third-year medical students.INTERVENTIONS: Three 90-minute faculty development seminars based on the One-Minute Preceptor teaching model.MEASUREMENTS AND MAIN RESULTS: Ninety-four encounters with 18,577 utterances were recorded, half before and half after the seminars. After the workshops, the proportion of utterances that contained feedback increased from 17% to 22% (P=.09) and was more likely to be specific (9% vs 15%; P=.02). After the workshops, teachers reported that the learning encounters were more successful (P=.03) and that they were better at letting the students reach their own conclusions (P=.001), at evaluating the learners (P=.03), and at creating plans for post-encounter learning (P=.02). The workshops had no effect on the duration of the student-teacher encounter or on student or patient satisfaction with the encounters.CONCLUSIONS: Brief, interactive, faculty development workshops based on the One-Minute Preceptor model of clinical teaching resulted in modest improvements in the quality of feedback delivered in the ambulatory setting.

Context and clinical reasoning: understanding the perspective of the expert's voice.

Medical Education 2011: 45: 927–938

Perspective: Redefining Context in the Clinical Encounter: Implications for Research and Training in Medical Education

Physician training and practice occur in complex environments. These complex environments, or contexts, raise important challenges and opportunities for research and training in medical education. The authors explore how studies from fields outside medicine can assist medical educators with their approach to the notion of context in the clinical encounter. First, they discuss the use of the term context in the clinical encounter as it relates to medical education. They then detail the meaning and use of the term in diverse fields outside medicine, such as mathematics, physics, and psychology, all of which suggest a nonlinear approach to the notion of context. Next, the authors highlight two inclusive theories, situated cognition and ecological psychology, that propose factors that relate to context and that suggest some potential next steps for research and practice. By redefining context as it relates to the clinical encounter (by linking it to theory and research from several diverse fields), the authors hope to move the field forward by providing guidance for the theory, research, and practice of medical education.

Medical Student Documentation in Electronic Health Records: A Collaborative Statement From the Alliance for Clinical Education

Purpose: The electronic health record (EHR) is an important advancement in health care. It facilitates improvement of health care delivery and coordination of care, but it creates special challenges for student education. This article represents a collaborative effort of the Alliance for Clinical Education (ACE), a multidisciplinary group formed in 1992. ACE recognizes the importance of medical student participation in patient care including the ability of documentation. This article proposes guidelines that can be used by educators to establish expectations on medical student documentation in EHRs. Summary: To provide the best education for medical students in the electronic era, ACE proposes to use the following as practice guidelines for medical student documentation in the EHR: (a) Students must document in the patients chart and their notes should be reviewed for content and format, (b) students must have the opportunity to practice order entry in an EHR—in actual or simulated patient cases—prior to graduation, (c) students should be exposed to the utilization of the decision aids that typically accompany EHRs, and (d) schools must develop a set of medical student competencies related to charting in the EHR and state how they would evaluate it. This should include specific competencies to be documented at each stage, and by time of graduation. In addition, ACE recommends that accreditation bodies such as the Liaison Committee for Medical Education utilize stronger language in their educational directives standards to ensure compliance with educational principles. This will guarantee that the necessary training and resources are available to ensure that medical students have the fundamental skills for lifelong clinical practice. Conclusions: ACE recommends that medical schools develop a clear set of competencies related to student in the EHR which medical students must achieve prior to graduation in order to ensure they are ready for clinical practice.

Expectations of and for Clerkship Directors: A Collaborative Statement from the Alliance for Clinical Education

Purpose: The clerkship director (CD) is an essential leader in the education of medical students on clinical rotations. This article represents a collaborative effort of the national clerkship organizations that comprise the Alliance for Clinical Education (ACE), a multidisciplinary group formed in 1992. ACE suggests that selection of a CD be regarded as an implied contract between the CD and the department chair that each will take the steps to ensure the success of the clerkship and of the CD. This article sets standards for what should be expected of a CD and provides guidelines for the resources and support to be provided to the person selected for leadership of the clerkship. Summary: In their roles as CDs, educators engage in three principal activities: administration, teaching, and scholarly activity, such as educational research. This article describes (a) the work products that are the primary responsibility of the CD; (b) the qualifications to be considered in selection of a CD; (c) the support structure, resources, and personnel that are necessary for the CD to accomplish his or her responsibilities; (d) incentives and career development for the CD; and (e) the dedicated time that should be provided for the clerkship and the CD to succeed. Studies by several CD organizations conclude that 25% should be considered a minimum estimate of time for the administrative aspects of running a clerkship. With the added teaching and scholarly activities undertaken by a CD, a minimum of 50% of an full-time equivalent has been recognized as appropriate. The complexity and the need for timeliness in the cyclic and often repetitive tasks of the clerkship require that a full-time administrative assistant be part of the structure dedicated to running the clerkship. Conclusion: ACE recommends that institutions have clear standards for what is expected of the director of a clinical clerkship and have correspondingly clear guidelines as to what should be expected for CDs in their career development and in the support they are given.

A controlled trial of a seminar to improve medical student attitudes toward, knowledge about, and use of the medical literature.

Objective: To determine whether an interactive seminar could affect medical student knowledge of research design, basic critical appraisal skills, and attitudes toward and clinical use of the medical literature.Design: Controlled, nonrandomized clinical trialParticipants: Third-year clinical clerks (n=146) during their core medicine clerkship.Interventions: Two 90 minute interactive seminars.Measurements and main results: Pre- and postquestionnaires were used to assess knowledge and attitudes regarding the use of the medical literature among 65 study and 81 control students. Blinded review of write-ups assessed actual use of the medical literature. Overall, 80% of the students subscribed to one or more journals and reported reading three or more journal articles per month. After the intervention, the study students were more likely than the control students to consider: 1) study design important in article selection and 2) use of medical literature critical to patient care decisions. Knowledge scores were significantly improved in the study group (p=0.0001). The intervention yielded no increase in the actual use of medical literature in patient write-ups over that encouraged by usual clerkship goals. 51% of the study and 48% of the control students cited literature at baseline, and 53% of all the students did so after the intervention. Of these citations, 50% were for journal articles and the remainder were for textbooks. The students infrequently mentioned the quality of the cited literature.Conclusions: An interactive seminar designed to introduce medical students to critical appraisal improved student knowledge and attitudes but did not increase the actual use of literature in patient writeups.

Opportunities and Challenges in Integrating Electronic Health Records Into Undergraduate Medical Education: A National Survey of Clerkship Directors

Background: Few studies have reported on the utilization and the effect of electronic health records on the education of medical students. Purpose: The purpose of this study was to describe the current use of electronic health records by medical students in the United States and explore the opportunities and challenges of integrating electronic health records into daily teaching of medical students. Methods: A survey with 24 questions regarding the use of electronic health records by medical students was developed by the Alliance for Clinical Educators and sent to clerkship directors across the United States. Both quantitative and qualitative responses were collected and analyzed to determine current access to and use of electronic health records by medical students. Results: This study found that an estimated 64% of programs currently allow student use of electronic health records, of which only two thirds allowed students to write notes within the electronic record. Overall, clerkship directors’ opinions on the effects of electronic health records on medical student education were neutral, and despite acknowledging many advantages to electronic health records, there were many concerns raised regarding their use in education. Conclusions: Medical students are using electronic health records at higher rates than physicians in practice. Although this is overall reassuring, educators have to be cautious about the limitations being placed on students documentation in electronic health records as this can potentially have consequences on their training, and they need to explore ways to maximize the benefits of electronic health records in medical education.

The Impact of the Stanford Faculty Development Program on Ambulatory Teaching Behavior

CONTEXT: Faculty development has received considerable investment of resources from medical institutions, though the impact of these efforts has been infrequently studied.OBJECTIVE: To measure the impact of the Stanford Faculty Development Program in Clinical Teaching on ambulatory teaching behavior.DESIGN: Pre-post.SETTING AND PARTICIPANTS: Eight internal medicine faculty participating in local faculty development.INTERVENTION: Participants received 7 2-hour sessions of faculty development. Each session included didactic, role-play, and videotaped performance evaluation.MAIN OUTCOME MEASURE: Before and after the intervention, faculty were video-taped during a case presentation from a standardized learner, who had been trained to portray 3 levels of learners: a third-year medical student, an intern, and a senior medical resident. Teacher and learner utterances (i.e. phrases) were blindly and randomly coded, using the Teacher Learner Interaction Analysis System, into categories that capture both the nature and intent of the utterances. We measured change in teaching behavior as detected through analysis of the coded utterances.RESULTS: Among the 48 videotaped encounters, there were a total of 7,119 utterances, with 3,203 (45%) by the teacher. Examining only the teacher, the total number of questions asked declined (714 vs 426, P=.02) with an increase in the proportion of higher-level, analytic questions (44% vs 55%, P<.0001). The quality of feedback also improved, with less “minimal” feedback (87% vs 76%, P<.0005) and more specific feedback (13% vs 22%) provided.CONCLUSIONS: Teaching behaviors improved after participation in this faculty development program, specifically in the quality of questions asked and feedback provided.

Research in assessment: Consensus statement and recommendations from the Ottawa 2010 conference

Medical education research in general is a young scientific discipline which is still finding its own position in the scientific range. It is rooted in both the biomedical sciences and the social sciences, each with their own scientific language. A more unique feature of medical education (and assessment) research is that it has to be both locally and internationally relevant. This is not always easy and sometimes leads to purely ideographic descriptions of an assessment procedure with insufficient general lessons or generalised scientific knowledge being generated or vice versa. For medical educational research, a plethora of methodologies is available to cater to many different research questions. This article contains consensus positions and suggestions on various elements of medical education (assessment) research. Overarching is the position that without a good theoretical underpinning and good knowledge of the existing literature, good research and sound conclusions are impossible to produce, and that there is no inherently superior methodology, but that the best methodology is the one most suited to answer the research question unambiguously. Although the positions should not be perceived as dogmas, they should be taken as very serious recommendations. Topics covered are: types of research, theoretical frameworks, designs and methodologies, instrument properties or psychometrics, costs/acceptability, ethics, infrastructure and support.