David R. Lambert

Relationship Between Peer Assessment During Medical School, Dean’s Letter Rankings, and Ratings by Internship Directors

BackgroundIt is not known to what extent the dean’s letter (medical student performance evaluation [MSPE]) reflects peer-assessed work habits (WH) skills and/or interpersonal attributes (IA) of students.ObjectiveTo compare peer ratings of WH and IA of second- and third-year medical students with later MSPE rankings and ratings by internship program directors.Design and ParticipantsParticipants were 281 medical students from the classes of 2004, 2005, and 2006 at a private medical school in the northeastern United States, who had participated in peer assessment exercises in the second and third years of medical school. For students from the class of 2004, we also compared peer assessment data against later evaluations obtained from internship program directors.ResultsPeer-assessed WH were predictive of later MSPE groups in both the second (F = 44.90, P < .001) and third years (F = 29.54, P < .001) of medical school. Interpersonal attributes were not related to MSPE rankings in either year. MSPE rankings for a majority of students were predictable from peer-assessed WH scores. Internship directors’ ratings were significantly related to second- and third-year peer-assessed WH scores (r = .32 [P = .15] and r = .43 [P = .004]), respectively, but not to peer-assessed IA.ConclusionsPeer assessment of WH, as early as the second year of medical school, can predict later MSPE rankings and internship performance. Although peer-assessed IA can be measured reliably, they are unrelated to either outcome.

Relationship Between Dean's Letter Rankings and Later Evaluations by Residency Program Directors

Background: It is not known how well deans letter rankings predict later performance in residency. Purpose: To assess the accuracy of deans letter rankings to predict clinical performance in internship. Method: Participants were medical students who graduated from the University of Rochester School of Medicine and Dentistry in the classes of 2003 and 2004. In their Deans Letter, each student was ranked as either “Outstanding” (upper quartile), “Excellent” (second quartile), “Very good” (lower 2 quartiles), or “Good” (lowest few percentile). We compared these deans letter rankings against results of questionnaires sent to program directors 9 months after graduation. Results: Response rate to the questionnaire was 58.9% (109 of 185 eligible graduates). There were no differences in response rate across the four deans letter ranking categories. Program directors rated students in the top two categories of deans letter rankings significantly higher than those in the very good group. Students in all three groups were rated significantly higher than those in the good group, F (3, 105) = 13.37, p < .001. Students in the very good group were most variable in their ratings by program directors, with many receiving similarly high ratings as students in the upper 2 groups. There were no differences by gender or specialty. Conclusion: Deans letter rankings are a significant predictor of later performance in internship among graduates of our medical school. Students in the bottom half of the class are most likely either to underperform or overperform in internship.

Internal Medicine Clerkship Characteristics Associated With Enhanced Student Examination Performance

Purpose To determine which internal medicine (IM) clerkship characteristics are associated with better student examination performance. Method The authors collected data from 17 U.S. medical schools (1,817 students) regarding characteristics of their IM clerkships, including structural characteristics, pedagogical approaches, patient contact, and clinical teacher characteristics. Outcomes of interest were postclerkship National Board of Medical Examiners (NBME) subject examination score, United States Medical Licensing Examination (USMLE) 2 score, and change in score from USMLE 1 to 2. To examine how associations of various clerkship characteristics and examination performance may differ for students of different prior achievement, the authors categorized students into those who scored in the top ¼ of the cohort on USMLE 1 and the bottom ¼. The authors conducted analyses at both the school and the individual student levels. Results In school-level analyses (using a reduced four-variable model), independent variables associated with higher NBME subject examination score were more small-group hours/week and use of community-based preceptors. Greater score increase from USMLE 1 to 2 was associated with students caring for more patients/day. Several variables were associated with enhanced student examination performance at the student level. The most consistent finding was that more patients cared for per day was associated with higher examination performance. More structured learning activities were associated with higher examination scores for students with lower baseline USMLE 1 achievement. Conclusion Certain clerkship characteristics are associated with better student examination performance, the most salient being caring for more patients per day.

Standardizing and Personalizing Science in Medical Education

In the century since the initial publication of the Flexner Report, medical education has emphasized a broad knowledge of science and a fundamental understanding of the scientific method, which medical educators believe are essential to the practice of medicine. The enormous growth of scientific knowledge that underlies clinical practice has challenged medical schools to accommodate this new information within the curricula. Although innovative educational modalities and new curricula have partly addressed this growth, the authors argue for a systematic restructuring of the content and structure of science education from the premedical setting through clinical practice. The overarching goal of science education is to provide students with a broad, solid foundation applicable to medicine, a deep understanding of the scientific method, and the attitudes and skills needed to apply new knowledge to patient care throughout their careers. The authors believe that to accomplish this successfully, the following changes must occur across the three major stages of medical education: (1) a reshaping of the scientific preparation that all students complete before medical school, (2) an increase in individualized science education during medical school, and (3) an emphasis on knowledge acquisition skills throughout graduate medical education and beyond to assure lifelong scientific learning. As students progress through the educational continuum, the balance of standardized and personalized scientific knowledge will shift toward personalization. Greater personalization demands that physicians possess well-refined skills in information acquisition, interpretation, and application for optimal lifelong learning and effective clinical practice.

A new measure of the cognitive, metacognitive, and experiential aspects of residents' learning.

Purpose Psychometric data are presented for the Cognitive Behavior Survey: Residency Level (rCBS), a survey that profiles cognitive, metacognitive, and experiential aspects of residents’ learning. Method The authors asked 963 residents from seven medicine residencies of large academic medical centers to participate in their study and gathered data from the respondents during a three-year period, 2000–2002. A factor analysis cross-validation design guided the development of rCBS’s seven scales: memorization, conceptualization, reflection, independent learning, critical thinking, meaningful learning experience, and attitude toward educational experience. Interscale correlations and MANOVA provided preliminary evidence of scale construct validity. Results A total of 424 residents (44%) responded. With several minor exceptions, items for each scale loaded .40 or higher. Memorization did not correlate with any other scale, except correlating negatively with critical thinking. Higher-order thinking scales (conceptualization, reflection, independent learning, critical thinking) correlated with one another and with meaningful learning experience and attitude toward educational experience. The one exception: conceptualization did not correlate with critical thinking. MANOVA results reveal that residents who scored in the top 20% on the reflection scale conceptualized, learned independently, and thought critically more than did the bottom 20%. Conclusions Results provide preliminary support for scale reliability and construct validity. As residencies seek to meet expectations of the Accreditation Council for Graduate Medical Education’s Outcome Project, rCBS could prove useful in program evaluation, residents’ self-assessment, and assessment by serving as a means to explore how residents learn, how residency programs affect learning behavior, and how clinically strong and weak residents differ in learning behaviors.

Maturation of medical student musculoskeletal medicine knowledge and clinical confidence

Purpose : Despite the prevalence of musculoskeletal (MSK) disorders, the degree to which medical schools are providing students the knowledge and confidence to treat these problems is unclear. This study evaluated MSK knowledge in second and fourth year medical students using a newly developed written assessment tool and examined the maturation of clinical confidence in treating core MSK disorders. Methods : Over a 3-year period, the National Board of Medical Examiners (NBME) MSK subject examination consisting of 75 items was administered to 568 second and fourth year students at a single institution. Students were also asked to rate their confidence in treating a selection of medicine/pediatric and MSK clinical scenarios on a 5-point Likert scale. Results : Participation rate was 98%. The NBME MSK assessment score was 59.2±10.6 for all second year medical students and 69.7±9.6 for all fourth year medical students. There was a significant increase in NBME scores between the second and fourth years (p<0.0001). Students were more confident in treating internal medicine/pediatric conditions than MSK medicine conditions (p=0.001). Confidence in treating MSK medicine conditions did not improve between the second and fourth years (p=0.41). Conclusions : To our knowledge, this is the first study to report increased MSK medicine knowledge as measured by a standardized examination after completing medical school core clinical rotations. Despite increased MSK knowledge, low levels of MSK clinical confidence among graduating students were noted. Further research is needed to determine the factors that influence MSK knowledge and clinical confidence in medical students.

The Medical Education Pathway: description and early outcomes of a student-as-teacher program.

Problem Although senior medical students at the University of Rochester School of Medicine and Dentistry (URSMD) have a long history of teaching junior peers, no formal educational training existed for students until 2007. The Medical Education Pathway (MEP) at the URSMD is a longitudinal student-as-teacher program that addresses both the local precedent of medical student teaching and the ongoing need to prepare students for teaching in residency and beyond. Approach In 2007, administrative faculty spearheaded efforts to create the MEP Committee, whose members then designed and implemented an elective curriculum. The curriculum balances didactics and experiential learning. A rigorous two-step application process precedes acceptance into the MEP. Participating students receive mentoring, assessment, and formative feedback on lecture delivery and leadership of various small-group formats. Outcomes Since 2007, 89 students have enrolled in the MEP: 49 have successfully completed it, and 40 are currently enrolled. MEP students teach in basic science and clinical courses, and they regularly make novel contributions to the medical school curriculum. Student learner peers demonstrate an ability to give constructive feedback to MEP students. Exit survey comments demonstrate that the MEP influences participating students’ career plans. Lessons learned from implementing the MEP include the importance of institutional support, dedicated faculty who value student teaching, and flexibility in scheduling. Next Steps Future improvements to the MEP include enhancing the assessment process and tracking the careers of graduates as outcome data. The MEP serves as a model for a successful student-as-teacher program in other institutions and settings.

Medical Student Attitudes Toward Patients in Diverse Care Settings: The Impact of a Patient Evaluation Course

Background: First-year medical students typically have limited exposure to patients in diverse care settings, such as rehabilitation facilities and nursing homes. Purpose: It is unknown whether students bring predetermined attitudes toward these patients, or whether attitudes are influenced by early exposure. We studied this in a new course that provides opportunities for students to interact with patients of various ages and disabilities. Methods: We conducted surveys of 1st-year medical students at the University of Rochester in the year prior to the new course and during its initial year. We used factor analysis to derive underlying dimensions of students’ responses. We also investigated the impact that the course had on their perceptions. Results: In both years, we found that students conceptualize patient care along 2 affective dimensions (comfort and pleasure) and 2 attitudinal dimensions (bias and pessimism), rather than by type of disability. Conclusions: This 10-week course improved their affect toward these patient groups but had little effect on their general attitudes toward the value of caring for them.

Development of an Audit Method to Assess the Prevalence of the ACGME’s General Competencies in an Undergraduate Medical Education Curriculum

Background: Despite the use of competency-based frameworks to evaluate physicians, the role of competency-based objectives in undergraduate medical education remains uncertain. Purpose: By use of an audit methodology, we sought to determine how the six Accreditation Council for Graduate Medical Education (ACGME) competencies, conceptualized as educational domains, would map onto an undergraduate medical curriculum. Methods: Standardized audit forms listing required activities were provided to course directors, who were then asked to indicate which of the domains were represented in each activity. Descriptive statistics were calculated. Results: Of 1,500 activities, there was a mean of 2.13 domains per activity. Medical Knowledge was the most prevalent (44%), followed by Patient Care (20%), Interpersonal and Communication Skills (12%), Professionalism (9%), Systems-Based Practice (8%), and Practice-Based Learning and Improvement (7%). There was considerable variation by year and course. Conclusions: The domains provide a useful framework for organizing didactic components. Faculty can also consider activities in light of the domains, providing a vocabulary for instituting curricular change and innovation.

Process of discovery: A fourth-year translational science course

Background The Liaison Committee on Medical Education notes the importance of educating medical students on clinical and translational research principles. Purpose To describe a fourth-year course, “Process of discovery,” which addresses teaching these principles, and to discuss students’ perceptions of the course. Methods Core components and pedagogical methods of this course are presented. Course assessment was performed with specific pre- and post-course assessments. Results During academic years 2004 to 2009, 562 students were enrolled, with assessment response rate of 94% pre-course and 85% post-course. The students’ self-assessment of their current understanding of clinical and translation research significantly increased, as well as their understanding of how clinical advances will take place over the next decade. Conclusions A fourth-year course teaching clinical and translational research is successful, is seen as a positive experience and can meet the requirements for including clinical and translational research in the medical school curriculum.