Stanley J. Hamstra

Technology-Enhanced Simulation for Health Professions Education: A Systematic Review and Meta-analysis

CONTEXT Although technology-enhanced simulation has widespread appeal, its effectiveness remains uncertain. A comprehensive synthesis of evidence may inform the use of simulation in health professions education. OBJECTIVE To summarize the outcomes of technology-enhanced simulation training for health professions learners in comparison with no intervention. DATA SOURCE Systematic search of MEDLINE, EMBASE, CINAHL, ERIC, PsychINFO, Scopus, key journals, and previous review bibliographies through May 2011. STUDY SELECTION Original research in any language evaluating simulation compared with no intervention for training practicing and student physicians, nurses, dentists, and other health care professionals. DATA EXTRACTION Reviewers working in duplicate evaluated quality and abstracted information on learners, instructional design (curricular integration, distributing training over multiple days, feedback, mastery learning, and repetitive practice), and outcomes. We coded skills (performance in a test setting) separately for time, process, and product measures, and similarly classified patient care behaviors. DATA SYNTHESIS From a pool of 10,903 articles, we identified 609 eligible studies enrolling 35,226 trainees. Of these, 137 were randomized studies, 67 were nonrandomized studies with 2 or more groups, and 405 used a single-group pretest-posttest design. We pooled effect sizes using random effects. Heterogeneity was large (I(2)>50%) in all main analyses. In comparison with no intervention, pooled effect sizes were 1.20 (95% CI, 1.04-1.35) for knowledge outcomes (n = 118 studies), 1.14 (95% CI, 1.03-1.25) for time skills (n = 210), 1.09 (95% CI, 1.03-1.16) for process skills (n = 426), 1.18 (95% CI, 0.98-1.37) for product skills (n = 54), 0.79 (95% CI, 0.47-1.10) for time behaviors (n = 20), 0.81 (95% CI, 0.66-0.96) for other behaviors (n = 50), and 0.50 (95% CI, 0.34-0.66) for direct effects on patients (n = 32). Subgroup analyses revealed no consistent statistically significant interactions between simulation training and instructional design features or study quality. CONCLUSION In comparison with no intervention, technology-enhanced simulation training in health professions education is consistently associated with large effects for outcomes of knowledge, skills, and behaviors and moderate effects for patient-related outcomes.

The educational impact of bench model fidelity on the acquisition of technical skill: The use of clinically relevant outcome measures

Objective:To evaluate the impact of bench model fidelity on the acquisition of technical skill using clinically relevant outcome measures. Methods:Fifty junior surgery residents participated in a 1-day microsurgical training course. Participants were randomized to 1 of 3 groups: 1) high-fidelity model training (live rat vas deferens; n = 21); 2) low-fidelity model training (silicone tubing; n = 19); or 3) didactic training alone (n = 10). Following training, all participants were assessed on the high- and low-fidelity bench models. Immediate outcome measures included procedure times, blinded, expert assessment of videotaped performance using checklists and global rating scales, anastomotic patency, suture placement precision, and final product ratings. Delayed outcome measures (obtained from the live rat vas deferens 30 days following training) included anastomotic patency, presence of a sperm granuloma, and the presence of sperm on microscopy. Results:Following training, checklist (P < 0.001) and global rating scores (P < 0.001) on the bench model simulators were higher among subjects who received hands-on training, irrespective of model fidelity. Immediate anastomotic patency rates of the rat vas deferens were higher with increasing model fidelity training (P = 0.048). Delayed anastomotic patency rates were higher among subjects who received bench model training, irrespective of model fidelity (P = 0.02). Rates of sperm presence on microscopy were higher among subjects who received high-fidelity model training compared with subjects who received didactic training (P = 0.039) but did not differ among subjects in the high- and low-fidelity groups. Conclusions:Surgical skills training on low-fidelity bench models appears to be as effective as high-fidelity model training for the acquisition of technical skill among novice surgeons.

Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis

Background: Although technology-enhanced simulation is increasingly used in health professions education, features of effective simulation-based instructional design remain uncertain. Aims: Evaluate the effectiveness of instructional design features through a systematic review of studies comparing different simulation-based interventions. Methods: We systematically searched MEDLINE, EMBASE, CINAHL, ERIC, PsycINFO, Scopus, key journals, and previous review bibliographies through May 2011. We included original research studies that compared one simulation intervention with another and involved health professions learners. Working in duplicate, we evaluated study quality and abstracted information on learners, outcomes, and instructional design features. We pooled results using random effects meta-analysis. Results: From a pool of 10 903 articles we identified 289 eligible studies enrolling 18 971 trainees, including 208 randomized trials. Inconsistency was usually large (I 2 > 50%). For skills outcomes, pooled effect sizes (positive numbers favoring the instructional design feature) were 0.68 for range of difficulty (20 studies; p < 0.001), 0.68 for repetitive practice (7 studies; p = 0.06), 0.66 for distributed practice (6 studies; p = 0.03), 0.65 for interactivity (89 studies; p < 0.001), 0.62 for multiple learning strategies (70 studies; p < 0.001), 0.52 for individualized learning (59 studies; p < 0.001), 0.45 for mastery learning (3 studies; p = 0.57), 0.44 for feedback (80 studies; p < 0.001), 0.34 for longer time (23 studies; p = 0.005), 0.20 for clinical variation (16 studies; p = 0.24), and −0.22 for group training (8 studies; p = 0.09). Conclusions: These results confirm quantitatively the effectiveness of several instructional design features in simulation-based education.

Nontechnical Skills in Anesthesia Crisis Management with Repeated Exposure to Simulation-based Education

Background:Critical incident reporting and observational studies have identified nontechnical skills that are vital to successful anesthesia crisis management. Examples of such skills include task management, team working, situation awareness, and decision making. These skills are not necessarily acquired through clinical experience and may need to be specifically taught. This study uses a high-fidelity patient simulator to assess the effect of repeated exposure to simulated anesthesia crises on the nontechnical skills of anesthesia residents. Methods:After institutional research board approval and informed consent, 20 anesthesia residents were recruited. Each resident was randomized to participate as the primary anesthesiologist in the management of three different simulated anesthesia crises using a high-fidelity patient simulator. After each session, videotaped footage was used to facilitate debriefing of their nontechnical skills. The videotapes were later reviewed by two expert blinded independent assessors who rated each residents nontechnical skills by using a previously validated and reliable marking system. Results:A significant improvement in the nontechnical skills of residents was demonstrated from their first to second session and from their first to third session (both P < 0.005). However from their second to third session, no significant improvement was observed. Interrater reliability between assessors was modest (single rater intraclass correlation = 0.53). Conclusion:A single exposure to anesthesia crises using a high-fidelity patient simulator can improve the nontechnical skills of anesthesia residents. However, an additional simulation session may confer little or no additional benefit.

Effect of visual-spatial ability on learning of spatially-complex surgical skills.

Visual-spatial ability is thought to be important in competency in specific surgical procedures. To test this hypothesis, 37 surgical residents completed six tests of visual-spatial ability, ranging from low-level to high-level visual processing. Using previously validated and objective instruments, we then assessed their ability to complete and learn a spatially-complex surgical procedure. Residents with higher visual-spatial scores in the form-board test and the mental-rotations test did significantly better in the procedure than did those with lower scores. After practice and feedback, residents with lower scores achieved a comparable level of competency. Our results suggest that visual-spatial ability is related to competency and quality of results in complex surgery, and could potentially be used in resident selection, career counselling, and training.

Fiberoptic orotracheal intubation on anesthetized patients: Do manipulation skills learned on a simple model transfer into the operating room?

Background With increasing pressure to use operating room time efficiently, opportunities for residents to learn fiberoptic orotracheal intubation in the operating room have declined. The purpose of this study was to determine whether fiberoptic orotracheal intubation skills learned outside the operating room on a simple model could be transferred into the clinical setting. Methods First-year anesthesiology residents and first- and second-year internal medicine residents were recruited. Subjects were randomized to a didactic-teaching-only group (n = 12) or a model-training group (n = 12). The didactic-teaching group received a detailed lecture from an expert bronchoscopist. The model-training group was guided, by experts, through tasks performed on a simple model designed to refine fiberoptic manipulation skills. After the training session, subjects performed a fiberoptic orotracheal intubation on healthy, consenting, anesthetized, paralyzed female patients undergoing elective surgery with predicted “easy” laryngoscopic intubations. Two blinded anesthesiologists evaluated each subject. Results After the training session, the model group significantly outperformed the didactic group in the operating room when evaluated with a global rating scale (P < 0.01) and checklist (P < 0.05). Model-trained subjects completed the fiberoptic orotracheal intubation significantly faster than didactic-trained subjects (P < 0.01). Model-trained subjects were also more successful at achieving tracheal intubation than the didactic group (P < 0.005). Conclusion Fiberoptic orotracheal intubation skills training on a simple model is more effective than conventional didactic instruction for transfer to the clinical setting. Incorporating an extraoperative model into the training of fiberoptic orotracheal intubation may greatly reduce the time and pressures that accompany teaching this skill in the operating room.

Gap detection thresholds as a function of tonal duration for younger and older listeners

Twenty normal hearing younger and twenty older adults in the early stages of presbycusis, but with relatively normal hearing at 2 kHz, were asked to discriminate between the presence versus absence of a gap between two equal-duration tonal markers. The duration of each marker was constant within a block of trials but varied between 0.83 and 500 ms across blocks. Notched-noise, centered at 2 kHz, was used to mask on- and off-transients. Gap detection thresholds of older adults were markedly higher than those of younger adults for marker durations of less than 250 ms but converged on those of younger adults at 500 ms. For both age groups, gap detection thresholds were independent of audiometric thresholds. These results indicate that older adults have more difficulty detecting a gap than younger adults when short marker durations (i.e., durations characteristic of speech sounds) are employed. It is shown that these results cannot be explained by linear models of temporal processing but are consistent with differential adaptation effects in younger and older adults.

Visual-spatial ability correlates with efficiency of hand motion and successful surgical performance

BACKGROUND This study examines the influence of visual-spatial ability and manual dexterity on surgical performance across 3 levels of expertise. METHODS Dental students, surgical residents, and staff surgeons completed standardized tests of manual dexterity and visual-spatial ability and were assessed objectively while performing the rigid fixation of an anterior mandible on bench model simulations. Outcome variables included expert assessment of technical performance and efficiency of hand motion during the procedure (recorded using electromagnetic sensors). RESULTS Visual-spatial scores correlated significantly with surgical performance scores within the group of dental students (r=.40 to.73), but this was not the case for residents or staff surgeons. For all groups, manual dexterity did not correlate with hand motion parameters. There were no differences between groups in visual-spatial ability or manual dexterity, but highly significant differences were seen in surgical performance scores (P<.001), in that surgeons outperformed residents, who in turn outperformed students. CONCLUSIONS Among novices, visual-spatial ability is associated with skilled performance on a spatially complex surgical procedure. However, advanced trainees and experts do not score any higher on carefully selected visual-spatial tests, suggesting that practice and surgical experience may supplant the influence of visual-spatial ability over time. Thus, the use of these tests for the selection of residents is not currently recommended; they may be of more use in identifying those novice trainees (ie, those with lower test scores) who might benefit most from brief supplementary instruction on specific technical tasks.

Reconsidering fidelity in simulation-based training.

In simulation-based health professions education, the concept of simulator fidelity is usually understood as the degree to which a simulator looks, feels, and acts like a human patient. Although this can be a useful guide in designing simulators, this definition emphasizes technological advances and physical resemblance over principles of educational effectiveness. In fact, several empirical studies have shown that the degree of fidelity appears to be independent of educational effectiveness. The authors confronted these issues while conducting a recent systematic review of simulation-based health professions education, and in this Perspective they use their experience in conducting that review to examine key concepts and assumptions surrounding the topic of fidelity in simulation.Several concepts typically associated with fidelity are more useful in explaining educational effectiveness, such as transfer of learning, learner engagement, and suspension of disbelief. Given that these concepts more directly influence properties of the learning experience, the authors make the following recommendations: (1) abandon the term fidelity in simulation-based health professions education and replace it with terms reflecting the underlying primary concepts of physical resemblance and functional task alignment; (2) make a shift away from the current emphasis on physical resemblance to a focus on functional correspondence between the simulator and the applied context; and (3) focus on methods to enhance educational effectiveness using principles of transfer of learning, learner engagement, and suspension of disbelief. These recommendations clarify underlying concepts for researchers in simulation-based health professions education and will help advance this burgeoning field.

Cost: The missing outcome in simulation-based medical education research: A systematic review

BACKGROUND The costs involved with technology-enhanced simulation remain unknown. Appraising the value of simulation-based medical education (SBME) requires complete accounting and reporting of cost. We sought to summarize the quantity and quality of studies that contain an economic analysis of SBME for the training of health professions learners. METHODS We performed a systematic search of MEDLINE, EMBASE, CINAHL, ERIC, PsychINFO, Scopus, key journals, and previous review bibliographies through May 2011. Articles reporting original research in any language evaluating the cost of simulation, in comparison with nonstimulation instruction or another simulation intervention, for training practicing and student physicians, nurses, and other health professionals were selected. Reviewers working in duplicate evaluated study quality and abstracted information on learners, instructional design, cost elements, and outcomes. RESULTS From a pool of 10,903 articles we identified 967 comparative studies. Of these, 59 studies (6.1%) reported any cost elements and 15 (1.6%) provided information on cost compared with another instructional approach. We identified 11 cost components reported, most often the cost of the simulator (n = 42 studies; 71%) and training materials (n = 21; 36%). Ten potential cost components were never reported. The median number of cost components reported per study was 2 (range, 1-9). Only 12 studies (20%) reported cost in the Results section; most reported it in the Discussion (n = 34; 58%). CONCLUSION Cost reporting in SBME research is infrequent and incomplete. We propose a comprehensive model for accounting and reporting costs in SBME.