Elif Bilgic

Multicenter proficiency benchmarks for advanced laparoscopic suturing tasks

BACKGROUND Advanced laparoscopic suturing (LS) tasks were developed based on a needs assessment. Initial validity evidence has been shown. The purpose of this multicenter study was to determine expert proficiency benchmarks for these tasks. METHODS 6 tasks were included: needle handling (NH), offset-camera forehand suturing (OF), offset-camera backhand suturing (OB), confined space suturing (CF), suturing under tension (UT), and continuous suturing (CS). Minimally invasive surgeons experienced in LS completed the tasks twice. Mean time and median accuracy scores were used to establish the benchmarks. RESULTS Seventeen MIS surgeons enrolled, from 7 academic centers. Mean (95% CI) time in seconds to complete each task was: NH 169 (149-189), OF 158 (134-181), OB 189 (154-224), CF 181 (156-205), UT 379 (334-423), and CS 416 (354-477). Very few errors in accuracy were made by experts in each of the tasks. CONCLUSIONS Time- and accuracy-based proficiency benchmarks for 6 advanced LS tasks were established. These benchmarks will be included in an advanced laparoscopic surgery curriculum currently under development.

Reliable assessment of operative performance.

BACKGROUND There is no consensus regarding the number of intraoperative assessments required to reliably measure trainee performance. This study used generalizability theory (GT) to describe factors contributing to score variance and to estimate the number of assessments needed to achieve high standards of reliability. METHODS While performing laparoscopic procedures, trainees were assessed by the attending surgeon using Global Operative Assessment of Laparoscopic Skills (GOALS). Data were collected prospectively (2-month intervals), assessing each trainee multiple times. Reliability coefficient was calculated using trainees, cases, and raters as factors. RESULTS Eighteen trainees were included for a total of 65 assessments. Total variance in scores was accounted for as follows: 66.1% by trainees, 31.6% by the interaction between trainees and cases, and 2.3% by raters. At least 3 cases are required for reliable scores using GOALS. CONCLUSIONS Trainees accounted for most of the variance in GOALS scores with a minimum of 3 cases required to improve the reliability of the scores obtained. These data may guide the implementation of performance assessments in surgical training programs.

Reliable Assessment of Performance in Surgery: A Practical Approach to Generalizability Theory.

When we assess residents performing surgery using various assessment instruments, how do we know that we are indeed measuring what we think we are measuring? In recent years, surgical training programs have begun to use performance assessment tools as a means of providing formative feedback to trainees, assessing their progress, as well as for the purposes of summative feedback and certification. For any given purpose, the reliability of assessment tools should be demonstrated. The surgical environment is highly variable and trainee performance depends on many external factors such as the type of surgical procedure being performed, the level of case difficulty, and the individuals involved in a case. All these external factors could affect the operative performance and assessment scores of trainees. Classic methods for estimating reliability, such as interrater reliability (raters), internal consistency (items), and test-retest reliability (cases), allow for separate evaluation of the effect of a given variable on trainee scores. This limits the accurate interpretation of the data as classic reliability measurements do not take into account the effects of the different variables on one another. In addition, in order to calculate test-retest reliability, each occasion needs to be the same or similar, which is less applicable for operative performance assessment in the real world, as each case (occasion) is different and is affected by external factors. For this reason, when assessing trainee performance in a variable work environment, reliability needs to be established using methods that can account for simultaneously interacting factors. One of the ways to address this variability is by using Generalizability Theory (GT). Instead of providing a reliability coefficient from a single factor, as in classic reliability methodology, GT integrates multiple potential factors, such as raters, items, and cases, into a single reliability coefficient. This theory has been used for simulation studies but few studies have examined the application

Effectiveness of Telementoring in Surgery Compared With On-site Mentoring: A Systematic Review

Background. Mentorship is important but may not be feasible for distance learning. To bridge this gap, telementoring has emerged. The purpose of this systematic review was to evaluate the effectiveness of telementoring compared with on-site mentoring. Methods. A search was done up to March 2015. Studies were included if they used telementoring between surgeons during a clinical encounter and if they compared on-site mentoring and telementoring. Results. A total of 11 studies were included. All reported no difference in complication rates, and 9 (82%) reported similar operative times; 4 (36%) reported technical issues, which was 3% of the total number of cases in the 11 studies. No study reported on higher levels of evidence for effectiveness of telementoring as an educational intervention. Conclusion. Studies reported that telementoring is associated with similar complication rates and operative times compared with on-site mentoring. However, the level of evidence to support the effectiveness of telementoring as a training tool is limited. There is a need for studies that provide evidence for the equivalence of the effectiveness of telementoring as an educational intervention in comparison with on-site mentoring.

What are the Training Gaps for Acquiring Laparoscopic Suturing Skills

INTRODUCTION Advanced laparoscopic suturing is considered a challenging skill to acquire. The aim of this study was to investigate the learning process for advanced laparoscopic suturing in the operating room to understand the obstacles trainees face when trying to master the skill. METHODS A qualitative methodology using semistructured interviews and field observations was used. Data were analyzed using a Grounded Theory approach. Participants were general surgery residents and surgeons with advanced minimally invasive surgery (MIS) experience. RESULTS Ten MIS surgeons across different institutions and 15 local general surgery residents were interviewed. The semistructured interviews and field observations of 9 advanced MIS operations (27h) yielded the following 6 themes around the acquisition of laparoscopic suturing skills for residents: complexity, training misalignment, variability of opportunities, inconsistency of techniques, lack of feedback, and differing expectations. CONCLUSION There are several unmet training needs around laparoscopic suturing skills. Training for advanced laparoscopic skills requires more emphasis on coaching and the development of advanced models. This study heralded the need to incorporate advanced laparoscopic skills into the surgical simulation curriculum.

Development of a Model for the Acquisition and Assessment of Advanced Laparoscopic Suturing Skills Using an Automated Device

Background. Needs assessment identified a gap regarding laparoscopic suturing skills targeted in simulation. This study collected validity evidence for an advanced laparoscopic suturing task using an Endo StitchTM device. Methods. Experienced (ES) and novice surgeons (NS) performed continuous suturing after watching an instructional video. Scores were based on time and accuracy, and Global Operative Assessment of Laparoscopic Surgery. Data are shown as medians [25th-75th percentiles] (ES vs NS). Interrater reliability was calculated using intraclass correlation coefficients (confidence interval). Results. Seventeen participants were enrolled. Experienced surgeons had significantly greater task (980 [964-999] vs 666 [391-711], P = .0035) and Global Operative Assessment of Laparoscopic Surgery scores (25 [24-25] vs 14 [12-17], P = .0029). Interrater reliability for time and accuracy were 1.0 and 0.9 (0.74-0.96), respectively. All experienced surgeons agreed that the task was relevant to practice. Conclusion. This study provides validity evidence for the task as a measure of laparoscopic suturing skill using an automated suturing device. It could help trainees acquire the skills they need to better prepare for clinical learning.

Defining competencies for endoscopic submucosal dissection (ESD) for gastric neoplasms

BackgroundCurrent methods for teaching and assessing competencies for endoscopic submucosal dissection (ESD) are highly variable, non-systematic, and are inefficient for the learner to acquire adequate skills. The present study aims to define and establish expert consensus regarding competencies required to perform ESD for gastric neoplasms.MethodsFourteen ESD experts from 12 institutions in Japan were invited to complete an online survey to rate potential items for their importance in performing ESD proficiently. By using methodology based on the Delphi principles, the results of each round were analyzed and re-sent to the experts until consensus was established. Items were included if ranked 8 out of a 10-point Likert scale, by ≥ 80% of the respondents.ResultsA list of 29 potential items was generated through a review of the literature, textbooks, and experience of the steering group members. Ten new items were added through the survey. Consensus was reached after three rounds. Response rate ranged from 93 to 100%. Thirty-four items achieved consensus as important surrogates of competency in performing ESD.ConclusionsEssential competencies for performing ESD were identified through expert consensus. These competencies can serve as the foundation for structured training and for development of objective assessment tools to evaluate trainee performance in ESD.

A scoping review of assessment tools for laparoscopic suturing

BackgroundA needs assessment identified a gap in teaching and assessment of laparoscopic suturing (LS) skills. The purpose of this review is to identify assessment tools that were used to assess LS skills, to evaluate validity evidence available, and to provide guidance for selecting the right assessment tool for specific assessment conditions.MethodsBibliographic databases were searched till April 2017. Full-text articles were included if they reported on assessment tools used in the operating room/simulation to (1) assess procedures that require LS or (2) specifically assess LS skills.ResultsForty-two tools were identified, of which 26 were used for assessing LS skills specifically and 26 for procedures that require LS. Tools had the most evidence in internal structure and relationship to other variables, and least in consequences.ConclusionThrough identification and evaluation of assessment tools, the results of this review could be used as a guideline when implementing assessment tools into training programs.

New report preparation system for endoscopic procedures using speech recognition technology

Background and study aims  We developed a new reporting system based on structured data entry, which selectively extracts only endoscopic findings from endoscopists’ oral statements and automatically inputs them into appropriate columns in real time during endoscopic procedures. Methods  We compared the time for endoscopic procedures and report preparation (ER time) by using an esophagogastroduodenoscopy simulator in three groups: one preparing reports using a mouse after endoscopic procedures (CE group); a second group preparing reports by using voice alone during endoscopic procedures (SR group); and the final group preparing reports by operating the system with a foot switch and inputting findings using voice during endoscopic procedures (SR + FS group). For the SR and SR + FS groups, we identified the recognition rates of the speech recognition system. Results  Mean ER times for cases with three findings each were 162, 130 and 119 seconds in the CE, SR and SR + FS groups, respectively. The mean ER times for cases with six findings each were 220, 144 and 128 seconds, respectively. The times in the SR and SR + FS groups were significantly shorter than that in the CE group ( P  < 0.017). The recognition rate of the SR group for cases with three findings each was 98.4 %, and 97.6 % in the same group for cases with six findings each. The rates in the SR + FS group were 95.2 % and 98.4 %, respectively. Conclusion  Our reporting system was demonstrated to allow an endoscopist to efficiently complete the report in real time during endoscopic procedures.