Patorn Piromchai

The Construct Validity and Reliability of an Assessment Tool for Competency in Cochlear Implant Surgery

Introduction. We introduce a rating tool that objectively evaluates the skills of surgical trainees performing cochlear implant surgery. Methods. Seven residents and seven experts performed cochlear implant surgery sessions from mastoidectomy to cochleostomy on a standardized virtual reality temporal bone. A total of twenty-eight assessment videos were recorded and two consultant otolaryngologists evaluated the performance of each participant using these videos. Results. Interrater reliability was calculated using the intraclass correlation coefficient for both the global and checklist components of the assessment instrument. The overall agreement was high. The construct validity of this instrument was strongly supported by the significantly higher scores in the expert group for both components. Conclusion. Our results indicate that the proposed assessment tool for cochlear implant surgery is reliable, accurate, and easy to use. This instrument can thus be used to provide objective feedback on overall and task-specific competency in cochlear implantation.

Developing effective automated feedback in temporal bone surgery simulation.

Objective We aim to test the effectiveness, accuracy, and usefulness of an automated feedback system in facilitating skill acquisition in virtual reality surgery. Study Design We evaluate the performance of the feedback system through a randomized controlled trial of 24 students allocated to feedback and nonfeedback groups. Setting The feedback system was based on the Melbourne University temporal bone surgery simulator. The study was conducted at the simulation laboratory of the Royal Victorian Eye and Ear Hospital, Melbourne. Subjects and Methods The study participants were medical students from the University of Melbourne, who were asked to perform virtual cortical mastoidectomy on the simulator. The extent to which the drilling behavior of the feedback and nonfeedback groups differed was used to evaluate the effectiveness of the system. Its accuracy was determined through a postexperiment observational assessment of recordings made during the experiment by an expert surgeon. Its usability was evaluated using students’ self-reports of their impressions of the system. Results A Friedman’s test showed that there was a significant improvement in the drilling performance of the feedback group, χ2(1) = 14.450, P < .001. The postexperiment assessment demonstrated that the system provided timely feedback (when trainee behavior was detected) 88.6% of the time and appropriate feedback (accurate advice) 84.2% of the time. Participants’ opinions about the usefulness of the system were highly positive. Conclusion The automated feedback system was observed to be effective in improving surgical technique, and the provided feedback was found to be accurate and useful.

Supporting skill acquisition in cochlear implant surgery through virtual reality simulation

Objectives: To evaluate the effectiveness of a virtual reality (VR) temporal bone simulator in training cochlear implant surgery. Methods: We compared the performance of 12 otolaryngology registrars conducting simulated cochlear implant surgery before (pre-test) and after (post-tests) receiving training on a VR temporal bone surgery simulator with automated performance feedback. The post-test tasks were two temporal bones, one that was a mirror image of the temporal bone used as a pre-test and the other, a novel temporal bone. Participant performances were assessed by an otologist with a validated cochlear implant competency assessment tool. Structural damage was derived from an automatically generated simulator metric and compared between time points. Results: Wilcoxon signed-rank test showed that there was a significant improvement with a large effect size in the total performance scores between the pre-test (PT) and both the first and second post-tests (PT1, PT2) (PT-PT1: P = 0.007, r = 0.78, PT-PT2: P = 0.005, r = 0.82). Conclusion: The results of the study indicate that VR simulation with automated guidance can effectively be used to train surgeons in training complex temporal bone surgeries such as cochlear implantation.

Effects of anatomical variation on trainee performance in a virtual reality temporal bone surgery simulator.

OBJECTIVE To investigate the importance of anatomical variation in acquiring skills in virtual reality cochlear implant surgery. METHODS Eleven otolaryngology residents participated in this study. They were randomly allocated to practice cochlear implant surgery on the same specimen or on different specimens for four weeks. They were then tested on two new specimens, one standard and one challenging. Videos of their performance were de-identified and reviewed independently, by two blinded consultant otolaryngologists, using a validated assessment scale. The scores were compared between groups. RESULTS On the standard specimen, the round window preparation score was 2.7 ± 0.4 for the experimental group and 1.7 ± 0.6 for the control group (p = 0.01). On the challenging specimen, instrument handling and facial nerve preservation scores of the experimental group were 3.0 ± 0.4 and 3.5 ± 0.7 respectively, while the control group received scores of 2.1 ± 0.8 and 2.4 ± 0.9 respectively (p < 0.05). CONCLUSION Training on temporal bones with differing anatomies is beneficial in the development of expertise.

Correlations of External Landmarks With Internal Structures of the Temporal Bone.

Hypothesis The internal anatomy of a temporal bone could be inferred from external landmarks. Background Mastoid surgery is an important skill that ENT surgeons need to acquire. Surgeons commonly use CT scans as a guide to understanding anatomical variations before surgery. Conversely, in cases where CT scans are not available, or in the temporal bone laboratory where residents are usually not provided with CT scans, it would be beneficial if the internal anatomy of a temporal bone could be inferred from external landmarks. Methods We explored correlations between internal anatomical variations and metrics established to quantify the position of external landmarks that are commonly exposed in the operating room, or the temporal bone laboratory, before commencement of drilling. Mathematical models were developed to predict internal anatomy based on external structures. Results From an operating room view, the distances between the following external landmarks were observed to have statistically significant correlations with the internal anatomy of a temporal bone: temporal line, external auditory canal, mastoid tip, occipitomastoid suture, and Henle’s spine. These structures can be used to infer a low lying dura mater (p = 0.002), an anteriorly located sigmoid sinus (p = 0.006), and a more lateral course of the facial nerve (p < 0.001).  In the temporal bone laboratory view, the mastoid tegmen and sigmoid sinus were also regarded as external landmarks. The distances between these two landmarks and the operating view external structures were able to further infer the laterality of the facial nerve (p < 0.001) and a sclerotic mastoid (p < 0.001).  Two nonlinear models were developed that predicted the distances between the following internal structures with a high level of accuracy: the distance from the sigmoid sinus to the posterior external auditory canal (p < 0.001) and the diameter of the round window niche (p < 0.001). Conclusion The prospect of encountering some of the more technically challenging anatomical variants encountered in temporal bone dissection can be inferred from the distance between external landmarks found on the temporal bone. These relationships could be used as a guideline to predict challenges during drilling and choosing appropriate temporal bones for dissection.

Comparison of Experts and Residents Performing a Complex Procedure in a Temporal Bone Surgery Simulator

OBJECTIVE To investigate the use of automated metrics from a virtual reality (VR) temporal bone surgery simulator to determine how the performance of experts and trainees differs when performing a complex otological procedure (mastoidectomy with posterior tympanotomy and cochleostomy). STUDY DESIGN Cohort study. METHODS Using the University of Melbourne VR temporal bone surgery simulator, seven ENT consultants and seven ENT residents performed two trials of the surgical approach to cochlear implantation on a virtual temporal bone. Simulator recordings were used to calculate a range of automated metrics for each stage of the procedure, capturing efficiency, technique characteristics, drilled bone regions, and damage to vital anatomical structures. RESULTS Results confirm that experts drilled more efficiently than residents. Experts generally used larger burrs and applied higher forces, resulting in faster material removal. However, they exercised more caution when drilling close to anatomical structures. Residents opened the temporal bone more widely, but neglected important steps in obtaining a clear view toward the round window, such as thinning the external ear canal wall and skeletonizing the medial aspect of the facial nerve. Residents used higher magnification and reoriented the temporal bone more often than experts. CONCLUSION VR simulation provides metrics that allow the objective analysis of surgical technique, and identification of differences between the performance of surgical residents and their senior colleagues. The performance of residents could be improved with more guidance regarding how much force they should apply, what burr size they should use, how they should orient the bone, and for cochlear implant surgery guidance regarding anatomical regions requiring particular attention, to visualize the round window.

Region-Specific Automated Feedback in Temporal Bone Surgery Simulation

The use of virtual reality simulators for surgical training has gained popularity in recent years, with an ever increasing body of evidence supporting the benefits and validity of simulation-based training. However, a crucial component of effective skill acquisition has not been adequately addressed, namely the provision of timely performance feedback. The utility of a surgical simulator is limited if it still requires the presence of experts to guide trainees. Automated feedback that emulates the advise provided by experts is necessary to facilitate independent learning. We propose an automated system that provides region-specific feedback on surgical technique within a temporal bone surgery simulator. The design of this system allows easy transfer of feedback models to multiple temporal bone specimens in the simulator. The system was validated by an expert otologist and was found to provide highly accurate and timely feedback.

Providing Automated Real-Time Technical Feedback for Virtual Reality Based Surgical Training: Is the Simpler the Better?

In surgery, where mistakes have the potential for dire consequences, proper training plays a crucial role. Surgical training has traditionally relied upon experienced surgeons mentoring trainees through cadaveric dissection and operating theatre practice. However, with the growing demand for more surgeons and more efficient training programs, it has become necessary to employ supplementary forms of training such as virtual reality simulation. However, the use of such simulations as autonomous training platforms is limited by the extent to which they can provide automated performance feedback. Recent work has focused on overcoming this issue by developing algorithms to provide feedback that emulates the advice of human experts. These algorithms can mainly be categorized into rule-based and machine learning based methods, and they have typically been validated through user studies against controls that received no feedback. To our knowledge, no investigations into the performance of the two types of feedback generation methods in comparison to each other have so far been conducted. To this end, we introduce a rule-based method of providing technical feedback in virtual reality simulation-based temporal bone surgery, implement a machine learning based method that has been proven to outperform other similar methods, and compare their performance in teaching surgical skills in practice through a user study. We show that simpler rule-based methods can be equally or more effective in teaching surgical skills when compared to more complex methods of feedback generation.