Jorie M. Colbert-Getz

Assessing Medical Students' and Residents' Perceptions of the Learning Environment: Exploring Validity Evidence for the Interpretation of Scores From Existing Tools

Purpose Although most agree that supportive learning environments (LEs) are essential for effective medical education, an accurate assessment of LE quality has been challenging for educators and administrators. Two previous reviews assessed LE tools used in the health professions; however, both have shortcomings. The primary goal of this systematic review was to explore the validity evidence for the interpretation of scores from LE tools. Method The authors searched ERIC, PsycINFO, and PubMed for peer-reviewed studies that provided quantitative data on medical students’ and/or residents’ perceptions of the LE published through 2012 in the United States and internationally. They also searched SCOPUS and the reference lists of included studies for subsequent publications that assessed the LE tools. From each study, the authors extracted descriptive, sample, and validity evidence (content, response process, internal structure, relationship to other variables) information. They calculated a total validity evidence score for each tool. Results The authors identified 15 tools that assessed the LE in medical school and 13 that did so in residency. The majority of studies (17; 61%) provided some form of content validity evidence. Studies were less likely to provide evidence of internal structure, response process, and relationship to other variables. Conclusions Given the limited validity evidence for scores from existing LE tools, new tools may be needed to assess medical students’ and residents’ perceptions of the LE. Any new tools would need robust validity evidence testing and sampling across multiple institutions with trainees at multiple levels to establish their utility.

Warm-up on a simulator improves residents' performance in laparoscopic surgery: a randomized trial.

Introduction and hypothesisOur aim was to assess the impact of immediate preoperative laparoscopic warm-up using a simulator on intraoperative laparoscopic performance by gynecologic residents.MethodsEligible laparoscopic cases performed for benign, gynecologic indications were randomized to be performed with or without immediate preoperative warm-up. Residents randomized to warm-up performed a brief set of standardized exercises on a laparoscopic trainer immediately before surgery. Intraoperative performance was scored using previously validated global rating scales. Assessment was made immediately after surgery by attending faculty who were blinded to the warm-up randomization.ResultsWe randomized 237 residents to 47 minor laparoscopic cases (adnexal/ tubal surgery) and 44 to major laparoscopic cases (hysterectomy). Overall, attendings rated upper-level resident performances (postgraduate year [PGY-3, 4]) significantly higher on global rating scales than lower-level resident performances (PGY-1, 2). Residents who performed warm-up exercises prior to surgery were rated significantly higher on all subscales within each global rating scale, irrespective of the difficulty of the surgery. Most residents felt that performing warm-up exercises helped their intraoperative performances.ConclusionPerforming a brief warm-up exercise before a major or minor laparoscopic procedure significantly improved the intraoperative performance of residents irrespective of the difficulty of the case.

Measuring the impact of the flipped anatomy classroom: The importance of categorizing an assessment by Bloom's taxonomy

The flipped classroom (FC) model has emerged as an innovative solution to improve student‐centered learning. However, studies measuring student performance of material in the FC relative to the lecture classroom (LC) have shown mixed results. An aim of this study was to determine if the disparity in results of prior research is due to level of cognition (low or high) needed to perform well on the outcome, or course assessment. This study tested the hypothesis that (1) students in a FC would perform better than students in a LC on an assessment requiring higher cognition and (2) there would be no difference in performance for an assessment requiring lower cognition. To test this hypothesis the performance of 28 multiple choice anatomy items that were part of a final examination were compared between two classes of first year medical students at the University of Utah School of Medicine. Items were categorized as requiring knowledge (low cognition), application, or analysis (high cognition). Thirty hours of anatomy content was delivered in LC format to 101 students in 2013 and in FC format to 104 students in 2014. Mann Whitney tests indicated FC students performed better than LC students on analysis items, U = 4243.00, P = 0.030, r = 0.19, but there were no differences in performance between FC and LC students for knowledge, U = 5002.00, P = 0.720 or application, U = 4990.00, P = 0.700, items. The FC may benefit retention when students are expected to analyze material. Anat Sci Educ 10: 170–175.

Gauging Events That Influence Students' Perceptions of the Medical School Learning Environment: Findings From One Institution

Purpose The medical school learning environment (LE), encompassing the physical, social, and psychological context for learning, holds significant influence on students’ professional development. Among these myriad experiences, the authors sought to gauge what students judge as influencing their perceptions of the LE. Method Fourth-year medical students at Johns Hopkins University participated in this cohort survey study before their 2010 graduation. A list of 55 events was iteratively revised and pilot-tested before being administered online. Responses assessed whether students experienced each event and, if so, the degree of impact on perceptions of the LE. A calculated mean impact score (MIS) provided a means to compare the relative impact of events. Results Of 119 students, 84 (71%) completed the survey. Students rated the overall LE as exceptional (29/84; 35%), good (36/84; 43%), fair (17/84; 20%), or poor (2/84; 2%). Eighty percent of students experienced at least 41 of the 55 events. MIS values ranged from 2.00 to 3.76 (highest possible: 4.00). Students rated positive events as having the highest impact. Few significant differences were found across gender, age, or surgical/nonsurgical specialty choice. MIS distributions differed between those perceiving the LE as exceptional or fair to poor for 22 (40%) of 55 events. Conclusions This study attempted to identify the discrete events that medical students perceive as most affecting their sense of the LE. Knowing the phenomena that most strongly influence student perceptions can inform how settings, relationships, and interactions can be shaped for meaningful learning and professional formation.

How do gender and anxiety affect students' self-assessment and actual performance on a high-stakes clinical skills examination?

Purpose Research suggests that medical students are not accurate in self-assessment, but it is not clear whether students over- or underestimate their skills or how certain characteristics correlate with accuracy in self-assessment. The goal of this study was to determine the effect of gender and anxiety on accuracy of students’ self-assessment and on actual performance in the context of a high-stakes assessment. Method Prior to their fourth year of medical school, two classes of medical students at Johns Hopkins University School of Medicine completed a required clinical skills exam in fall 2010 and 2011, respectively. Two hundred two students rated their anxiety in anticipation of the exam and predicted their overall scores in the history taking and physical examination performance domains. A self-assessment deviation score was calculated by subtracting each student’s predicted score from his or her score as rated by standardized patients. Results When students self-assessed their data gathering performance, there was a weak negative correlation between their predicted scores and their actual scores on the examination. Additionally, there was an interaction effect of anxiety and gender on both self-assessment deviation scores and actual performance. Specifically, females with high anxiety were more accurate in self-assessment and achieved higher actual scores compared with males with high anxiety. No differences by gender emerged for students with moderate or low anxiety. Conclusions Educators should take into account not only gender but also the role of emotion, in this case anxiety, when planning interventions to help improve accuracy of students’ self-assessment.

The Johns Hopkins Learning Environment Scale: Measuring Medical Students' Perceptions of the Processes Supporting Professional Formation

Purpose To construct a new measure to assess students’ perceptions of the medical school learning environment (LE). Method In 2012, students at Johns Hopkins University School of Medicine completed a survey containing 32 LE items. Additional questions asked about overall perception of the LE, personal growth, and recommending the school to a friend. Validity evidence for content, response process, internal structure, and relation to other variables was collected for interpretation of scores. Results Of 465 students surveyed, 377 (81%) completed all LE items. Exploratory factor analysis yielded the 28-item Johns Hopkins Learning Environment Scale (JHLES) with seven factors/subscales: community of peers, faculty relationships, academic climate, meaningful engagement, mentoring, inclusion and safety, and physical space. Students’ overall JHLES scores ranged from 51 to 139, of a possible 28 to 140, with a mean (SD) of 107 (15). Overall scores and most subscale scores did not differ significantly by gender or racial/ethnic background, but did differ significantly by overall perception of the LE (P ⩽ .001) and increased incrementally as overall perception improved. Overall JHLES scores were significantly higher for students with higher personal growth scores and students who would recommend the school (both P < .001). Subscale scores for all seven factors increased with improved overall perception of the LE (all P ⩽ .005). Conclusions The JHLES is a new measure to assess students’ perceptions of the medical school LE, with supporting validity evidence and content describing the social, relational, and academic processes of medical school that support students’ professional formation.

Revisiting effects of contextual strength on the subordinate bias effect: Evidence from eye movements

In this study, we examined two issues regarding the role of context in ambiguity resolution: whether access to the contextually appropriate meaning is exhaustive or selective, and whether the contextually inappropriate meaning is inhibited. Participants read texts in which a biased ambiguous word was encountered twice while their eye movements were measured. The context preceding the first encounter varied in the extent to which the subordinate meaning was supported; the context preceding the second encounter always supported the dominant meaning. The findings suggest that lexical access is exhaustive but can be influenced by context, and that the subsequent accessibility of the contextually inappropriate meaning is unaffected by previous selection processes. The results were interpreted in terms of the assumptions of the reordered-access model and activation mechanisms that operate during reading.

Number-of-Features Effects During Reading: Evidence From Eye Movements

Researchers have demonstrated that words with high numbers of features (NOF) are recognized more quickly than words with low NOF. One difficulty in testing theories of word recognition with paradigms that present words in isolation, however, is that these paradigms can produce task demands not present in naturalistic reading situations. Extending previous research on this topic, we embedded high and low NOF words in neutral sentences that participants read as their eye movements were monitored. We investigated how NOF and four other variables known to affect word recognition (frequency, age of acquisition, familiarity, and length) influenced word reading time. This is the first experiment to demonstrate NOF effects on eye fixations during reading.

The Two-Stage Examination: A Method to Assess Individual Competence and Collaborative Problem Solving in Medical Students.

Problem Effectively solving problems as a team under stressful conditions is central to medical practice; however, because summative examinations in medical education must test individual competence, they are typically solitary assessments. Approach Using two-stage examinations, in which students first answer questions individually (Stage 1) and then discuss them in teams prior to resubmitting their answers (Stage 2), is one method for rectifying this discordance. On the basis of principles of social constructivism, the authors hypothesized that two-stage examinations would lead to better retention of, specifically, items answered incorrectly at Stage 1. In fall 2014, they divided 104 first-year medical students into two groups of 52 students. Groups alternated each week between taking one- and two-stage examinations such that each student completed 6 one-stage and 6 two-stage examinations. The authors reassessed 61 concepts on a final examination and, using the Wilcoxon signed ranked tests, compared performance for all concepts and for just those students initially missed, between Stages 1 and 2. Outcomes Final examination performance on all previously assessed concepts was not significantly different between the one-and two-stage conditions (P = .77); however, performance on only concepts that students initially answered incorrectly on a prior examination improved by 12% for the two-stage condition relative to the one-stage condition (P = .02, r = 0.17). Next Steps Team assessment may be most useful for assessing concepts students find difficult, as opposed to all content. More research is needed to determine whether these results apply to all medical school topics and student cohorts.

Evaluation of a simulation-based pediatric clinical skills curriculum for medical students.

Introduction Simulation-based education is expensive and requires greater resources than traditional methods, yet there is limited evidence to justify such expenditures for medical student education. Methods We describe the implementation and evaluation of a simulation-based curriculum delivered to medical students during a pediatric clerkship. This prospective mixed-methods study evaluated a 5-day long simulation-based clinical skills curriculum (PRE-Clerkship EDucational Exercises [PRECEDE]) at the Johns Hopkins University School of Medicine. Two hundred medical students participated in PRECEDE during a 2-year period and were compared with 236 students who had not. Outcomes were assessed across 3 levels of Kirkpatrick’s framework for evaluation. The 4-level model consists of reaction, learning, behavior, and results criteria. Secondary outcomes measured changes in assessment scores across 16 student performance domains during clerkship, changes in performance on the National Board of Medical Examiners subject examination in pediatrics, and student assessments of the curriculum. Results Improvements were noted across 3 levels of the Kirkpatrick’s model. Student performance evaluations were significantly higher across all 16 evaluation components, with effect sizes ranging from small to medium (Cohen’s d, 0.23–0.44). Students scored significantly higher on the National Board of Medical Examiners pediatric shelf examination (80 vs. 77, P < 0.001). Ninety-seven percent of the medical students agreed that their skills increased and that the time lost to real clinical experiences was a worthwhile trade-off for this curriculum Conclusions The implementation of a simulation-based curriculum within a pediatrics clerkship resulted in higher knowledge scores and led to improvements in medical student clinical performance during the clerkship.