Julie Woodhouse

Reliability and Validity of a Simulation-based Acute Care Skills Assessment for Medical Students and Residents

Background Medical students and residents are expected to be able to manage a variety of critical events after training, but many of these individuals have limited clinical experiences in the diagnosis and treatment of these conditions. Life-sized mannequins that model critical events can be used to evaluate the skills required to manage and treat acute medical conditions. The purpose of this study was to develop and test simulation exercises and associated scoring methods that could be used to evaluate the acute care skills of final-year medical students and first-year residents. Methods The authors developed and tested 10 simulated acute care situations that clinical faculty at a major medical school expects graduating physicians to be able to recognize and treat at the conclusion of training. Forty medical students and residents participated in the evaluation of the exercises. Four faculty members scored the students/residents. Results The reliability of the simulation scores was moderate and was most strongly influenced by the choice and number of simulated encounters. The validity of the simulation scores was supported through comparisons of students’/residents’ performances in relation to their clinical backgrounds and experience. Conclusion Acute care skills can be validly and reliably measured using a simulation technology. However, multiple simulated encounters, covering a broad domain, are needed to effectively and accurately estimate student/resident abilities in acute care settings.

Performance of residents and anesthesiologists in a simulation-based skill assessment

Background:Anesthesiologists and anesthesia residents are expected to acquire and maintain skills to manage a wide range of acute intraoperative anesthetic events. The purpose of this study was to determine whether an inventory of simulated intraoperative scenarios provided a reliable and valid measure of anesthesia residents’ and anesthesiologists’ skill. Methods:Twelve simulated acute intraoperative scenarios were designed to assess the performance of 64 residents and 35 anesthesiologists. The participants were divided into four groups based on their training and experience. There were 31 new CA-1, 12 advanced CA-1, and 22 CA-2/CA-3 residents as well as a group of 35 experienced anesthesiologists who participated in the assessment. Each participant managed a set of simulated events. The advanced CA-1 residents, CA-2/CA-3 residents, and 35 anesthesiologists managed 8 of 12 intraoperative simulation exercises. The 31 CA-1 residents each managed 3 intraoperative scenarios. Results:The new CA-1 residents received lower scores on the simulated intraoperative events than the other groups of participants. The advanced CA-1 residents, CA-2/CA-3 residents, and anesthesiologists performed similarly on the overall assessment. There was a wide range of scores obtained by individuals in each group. A number of the exercises were difficult for the majority of participants to recognize and treat, but most events effectively discriminated among participants who achieved higher and lower overall scores. Conclusion:This simulation-based assessment provided a valid method to distinguish the skills of more experienced anesthesia residents and anesthesiologists from residents in early training. The overall score provided a reliable measure of a participant’s ability to recognize and manage simulated acute intraoperative events. Additional studies are needed to determine whether these simulation-based assessments are valid measures of clinical performance.

Acute care skills in anesthesia practice: a simulation-based resident performance assessment.

Background:A recurring initiative in graduate education is to find more effective methods to assess specialists’ skills. Life-sized simulators could be used to assess the more complex skills expected in specialty practice if a curriculum of relevant exercises were developed that could be simply and reliably scored. The purpose of this study was to develop simulation exercises and associated scoring methods and determine whether these scenarios could be used to evaluate acute anesthesia care skills. Methods:Twenty-eight residents (12 junior and 16 senior) managed three intraoperative and three postoperative simulation exercises. Trainees were required to make a diagnosis and intervention in a simulation encounter designed to recreate an acute perioperative complication. The videotaped performances were scored by six raters. Three raters used a checklist scoring system. Three faculty raters measured when trainees performed three key diagnostic or therapeutic actions during each 5-min scenario. These faculty also provided a global score using a 10-cm line with scores from 0 (unsatisfactory) to 10 (outstanding). The scenarios included (1) intraoperative myocardial ischemia, (2) postoperative anaphylaxis, (3) intraoperative pneumothorax, (4) postoperative cerebral hemorrhage with intracranial hypertension, (5) intraoperative ventricular tachycardia, and (6) postoperative respiratory failure. Results:The high correlation among all of the scoring systems and small variance among raters’ scores indicated that all of the scoring systems measured similar performance domains. Scenarios varied in their overall difficulty. Even though trainees who performed well on one exercise were likely to perform well in subsequent scenarios, the authors found that there were considerable differences in case difficulty. Conclusion:This study suggests that simulation can be used to measure more complex skills expected in specialty training. Similar to other studies that assess a broad content domain, multiple encounters are needed to estimate skill effectively and accurately.

An acute care skills evaluation for graduating medical students: a pilot study using clinical simulation

Purpose  This investigation aimed to explore the measurement properties of scores from a patient simulator exercise.

Setting performance standards for mannequin-based acute-care scenarios: an examinee-centered approach.

Background: In medicine, standard setting methodologies have been developed for both selected-response and performance-based assessments. For simulation-based tasks, research efforts have been directed primarily at assessments that incorporate standardized patients. Mannequin-based evaluations often demand complex, time-sensitive, hierarchically ordered, sequential actions that are difficult to evaluate and score. Moreover, collecting reliable proficiency judgments, necessary to estimate meaningful cut points, can be challenging. The purpose of this investigation was to explore whether expert judgments obtained using an examinee-centered standard setting method that was previously validated for standardized patient-based assessments could be used to set defensible standards for acute-care, mannequin-based scenarios. Methods: Nineteen physicians were recruited to serve as panelists. For each of 12 simulation scenarios, between 8 and 10 performance samples (audio-video recordings), covering the expected ability continuum, were chosen for review. The performance samples were selected from a previously administered evaluation of postgraduate trainees. Based on a consensus definition of readiness to enter unsupervised practice, the panelists made independent judgments of each performance. For each scenario, the association between the panelists’ judgments and the assessment scores was summarized and used to estimate a scenario-specific cut score. Results: For 9 of the scenarios, there was at least a moderately strong relationship between the aggregate panelists’ rating and the performance scores, thus allowing for estimation of meaningful numeric standards. For the other 3 scenarios, the aggregate decision rules used by the panelists did not correspond with the achievement measures. For scenarios independently rated by split panels, the estimated cut scores were similar. Conclusions: An examinee-centered approach, using aggregate expert judgments of audio-video performances, was suitable for setting standards on most acute-care, mannequin-based scenarios. It is necessary, however, to have valid scores for the chosen scenarios and to sample performances across the ability spectrum.

Video Review Using a Reliable Evaluation Metric Improves Team Function in High-Fidelity Simulated Trauma Resuscitation

OBJECTIVE To demonstrate that instruction of proper team function can occur using high-fidelity simulated trauma resuscitation with video-assisted debriefing and that this process can be integrated rapidly into a standard general surgery curriculum. DESIGN The rater reliability of our team metric was assessed by having physicians and nonphysicians rate the same video-recorded trauma simulations at intervals in time. To assess the effectiveness of video debriefing, subjects participated in a 3-week trauma team training course that consisted of 2 video-recorded simulation sessions, each approximately 2 hours in length separated by a 90-minute debriefing session. To assess the impact of the debriefing session, video recordings of participants performing resuscitations before and after the debriefing were reviewed by a panel of blinded traumatologists and graded using our team evaluation instrument. SETTING The study took place at the high-fidelity simulation center at a large, urban academic training hospital. PARTICIPANTS All 11 PGY-2 general surgery and combined general surgery and plastic surgery residents at our institution. RESULTS Our instrument was found to have high interrater correlation (interclass correlation coefficient [ICC], 0.926; 95% confidence interval, 0.893-0.953). Initially, residents were either unsure as to their competency to serve as team leader (70%) or felt they were not competent to serve as team leader (30%). Ninety percent of residents found the video debriefing very to extremely helpful in improving team function and clinical competency. All participants felt more competent as both team leaders and team members because of the video debriefing. The mean team function score improved significantly after video debriefing (4.39 [±0.3] vs 5.45 [±0.4] prevideo vs postvideo review, p < 0.05). CONCLUSIONS Video review with debriefing is an effective means of teaching team competencies and improving team function in simulated trauma resuscitation. This strategy can be integrated readily into the surgical curriculum analogous to other applications of simulation technology.

Do preclinical background and clerkship experiences impact skills performance in an accelerated internship preparation course for senior medical students

BACKGROUND Dedicated skills courses may help to prepare 4th-year medical students for surgical internships. The purpose of this study was to analyze the factors that influence the preparedness of 4th-year medical students planning a surgical career, and the role that our skills course plays in that preparedness. METHODS A comprehensive skills course for senior medical students matching in a surgical specialty was conducted each spring from 2006 through 2009. Students were surveyed for background skills, clerkship experience, and skills confidence levels (1-5 Likert scale). Assessment included 5 suturing and knot-tying tasks pre- and postcourse and a written examination. Data are presented as mean values ± standard deviations; statistical analyses were by 2-tailed t test, linear regression, and analysis of variance. RESULTS Sixty-five 4th-year students were enrolled; most common specialties were general surgery (n = 22) and orthopedics (n = 16). Thirty-five students were elite musicians (n = 16) or athletes (n = 19) and 8 regular videogamers. Suturing task times improved significantly from pre- to postcourse for all 5 tasks (total task times pre, 805 ± 202 versus post, 627 ± 168 seconds [P < .0001]) as did confidence levels for 8 skills categories, including management of on-call problems (P < .05). Written final examination proficiency (score ≥70%) was achieved by 81% of students. Total night call experience 3rd year was 23.3 ± 10.7 nights (7.3 ± 4.3 surgical call) and 4th year 10.5 ± 7.4 nights (7.2 ± 6.8 surgical call). Precourse background variables significantly associated with outcome measures were athletics with precourse suturing and 1-handed knot tying (P < .05); general surgery specialty and instrument tying (P = .012); suturing confidence levels and precourse suturing and total task times (P = .024); and number of nonsurgical call nights with confidence in managing acute on-call problems (P = .028). No significant correlation was found between these variables and postcourse performance. CONCLUSION Completion of an accelerated skills course results in comparable levels of student performance postcourse across a variety of preclinical backgrounds and clerkship experiences.

Team Behavior During Trauma Resuscitation: A Simulation-Based Performance Assessment

INTRODUCTION Trauma resuscitations require a coordinated response from a diverse group of health care providers. Currently, there are no widely accepted methods of assessing team effectiveness in this setting. Simulation affords a method to assess team effectiveness. The purpose of this study was to use a simulation setting to develop a specialized assessment instrument for team response in trauma resuscitation. METHODS We developed our assessment instrument using clinical simulation. Four teams of 3 postgraduate year-2 surgical trainees in conjunction with scripted confederates were videotaped enacting 6 separate trauma resuscitation scenarios that mirrored clinical conditions encountered at our level 1 trauma center. Ten of the resulting videotaped scenarios represented a spectrum of team behavior (ineffective to effective) and were scored by 8 experienced clinicians using the Mayo High Performance Teamwork Scale. RESULTS Based in part on the Mayo High Performance Teamwork Scale, we created a prototype trauma team assessment tool consisting of 7 attributes that we scored in binary fashion (present/absent). We validated this prototype by assigning a normalized ranking score to each of the 10 scenarios based on the score supplied by each rater. The presence/absence of the 7 attributes varied significantly among scenarios (52.5% to 93.8%; P < .001). Median scores differed significantly comparing the 5 lowest-ranking scenarios with the 5 highest-ranking scenarios (P < .001). CONCLUSION Our prototype instrument may be effective at assessing team effectiveness during trauma resuscitations. This instrument may prove useful for assessing team competency skills, providing timely feedback to teams, and examining the relationship between effective team function and clinically important outcomes. Further, it may be applicable to other high-acuity, time-sensitive clinical situations.

Decision making in trauma settings: simulation to improve diagnostic skills.

Introduction In the setting of acute injury, a wrong, missed, or delayed diagnosis can impact survival. Clinicians rely on pattern recognition and heuristics to rapidly assess injuries, but an overreliance on these approaches can result in a diagnostic error. Simulation has been advocated as a method for practitioners to learn how to recognize the limitations of heuristics and develop better diagnostic skills. The objective of this study was to determine whether simulation could be used to provide teams the experiences in managing scenarios that require the use of heuristic as well as analytic diagnostic skills to effectively recognize and treat potentially life-threatening injuries. Methods Ten scenarios were developed to assess the ability of trauma teams to provide initial care to a severely injured patient. Seven standard scenarios simulated severe injuries that once diagnosed could be effectively treated using standard Advanced Trauma Life Support algorithms. Because diagnostic error occurs more commonly in complex clinical settings, 3 complex scenarios required teams to use more advanced diagnostic skills to uncover a coexisting condition and treat the patient. Teams composed of 3 to 5 practitioners were evaluated in the performance of 7 (of 10) randomly selected scenarios (5 standard, 2 complex). Expert rates scored teams using standardized checklists and global scores. Results Eighty-three surgery, emergency medicine, and anesthesia residents constituted 21 teams. Expert raters were able to reliably score the scenarios. Teams accomplished fewer checklist actions and received lower global scores on the 3 analytic scenarios (73.8% [12.3%] and 5.9 [1.6], respectively) compared with the 7 heuristic scenarios (83.2% [11.7%] and 6.6 [1.3], respectively; P < 0.05 for both). Teams led by more junior residents received higher global scores on the analytic scenarios (6.4 [1.3]) than the more senior team leaders (5.3 [1.7]). Conclusions This preliminary study indicates that teams led by more senior residents received higher scores when managing heuristic scenarios but were less effective when managing the scenarios that require a more analytic approach. Simulation can be used to provide teams with decision-making experiences in trauma settings and could be used to improve diagnostic skills as well as study the decision-making process.

Outcomes of a proficiency-based skills curriculum at the beginning of the fourth year for senior medical students entering surgery

INTRODUCTION We hypothesized that a proficiency-based curriculum administered early in the fourth year to senior medical students (MS4) would achieve outcomes comparable to a similar program administered during surgical internship. METHODS MS4 (n = 18) entering any surgical specialty enrolled in a proficiency-based skills curriculum at the beginning of the fourth year that included suturing/knot-tying, on-call problems, laparoscopic, and other skills (urinary catheter, sterile prep/drape, IV placement, informed consent, electrosurgical use). Assessment was at 4-12 weeks after training by a modified Objective Structured Assessment of Technical Skills (OSATS). Suturing and knot tying tasks were assessed by time and OSATS technical proficiency (TP) scores (1 [novice], 3 [proficient], 5 [expert]). Outcomes were compared with PGY-1 residents who received similar training at the beginning of internship and assessment 4-12 weeks later. Data are presented as mean values ± standard deviation; statistical significance was assessed by Students t test. RESULTS Fifteen of 18 MS4 (83%) reached proficiency on all 15 tasks, and 2 others were proficient on all but 1 laparoscopic task. Compared with PGY-1s, MS4 were significantly faster for 3 of 5 suturing and tying tasks and total task time (547 ± 63 vs 637 ± 127 s; P < .05). Mean TP scores were similar for both groups (MS4, 3.4 ± 0.5 vs PGY-1, 3.1 ± .57; P = NS). MS4 OSATS scores were higher for IV placement, informed consent, and urinary catheter placement, but lower for prep and drape and for management of on-call problems. CONCLUSION MS4 who participate in a proficiency-based curriculum taught early in the fourth year are able to meet proficiency targets in a high percentage of cases. This approach should better prepare MS4 for surgical internship.