Lisa Satterthwaite

Developing Criteria for Proficiency-Based Training of Surgical Technical Skills Using Simulation: Changes in Performances as a Function of Training Year

BACKGROUND Proficiency-based residency training programs can be more efficient than the current duration-based formats. For their successful implementation, appropriate proficiency criteria must be developed. The objective of this study was to investigate the relationship between technical skill performances assessed using computer- and expert-based methods and training year. An assumption was that asymptotes in performance as a function of training year can be used to set the proficiency level for a technical skill, so the value at which the asymptote occurs can be labeled as the proficiency criteria. STUDY DESIGN Thirty-eight general surgery residents performed one-handed knot tying on bench-top simulators at two levels of difficulty: superficial and deep. Motion-efficiency measures and expert-based measures were used to evaluate performance. Total number of operations (ie, surgical volume) that each trainee participated in during residency was also acquired. RESULTS On the superficial model, asymptotes were observed at year 1 for motion-efficiency and year 3 for expert-based measures. On the deep model, asymptotes were observed at year 2 for motion-efficiency and year 4 for expert-based measures. CONCLUSIONS The data demonstrate the challenges associated with defining technical skills proficiency criteria. Different asymptotes were observed for the two assessment methods and neither covaried substantially with surgical volume. These data suggest that this asymptote approach in defining proficiency criteria can be suitable for development of proficiency-based residency training programs. The sensitivity of this approach to the type of assessment method and to the functional difficulty of the simulators used for assessment must be considered.

Setting Up a Surgical Skills Center

With the increasing use of simulation in medicine, many departments of surgery are considering the development of a surgical skills center. This article focuses on practical issues that must be considered when setting up a surgical skills center. The importance of developing a mission statement and including relevant stakeholders is discussed. The types of curricula that can be developed as well as the appropriate equipment purchased to support different curricula are considerations. Space requirements, funding sources, and staffing are also covered. Setting up a surgical skills center requires institutional buy-in and planning from the outset. Various models of skills centers, depending on local politics, are discussed.

A novel approach to assessing technical competence of colorectal surgery residents: The development and evaluation of the colorectal objective structured assessment of technical skill (COSATS)

Objective: To develop and evaluate an objective method of technical skills assessment for graduating subspecialists in colorectal (CR) surgery—the Colorectal Objective Structured Assessment of Technical Skill (COSATS). Background: It may be reasonable for the public to assume that surgeons certified as competent have had their technical skills assessed. However, technical skill, despite being the hallmark of a surgeon, is not directly assessed at the time of certification by surgical boards. Methods: A procedure-based, multistation technical skills examination was developed to reflect a sample of the range of skills necessary for CR surgical practice. These consisted of bench, virtual reality, and cadaveric models. Reliability and construct validity were evaluated by comparing 10 graduating CR residents with 10 graduating general surgery (GS) residents from across North America. Expert CR surgeons, blinded to level of training, evaluated performance using a task-specific checklist and a global rating scale. The mean global rating score was used as the overall examination score and a passing score was set at “borderline competent for CR practice.” Results: The global rating scale demonstrated acceptable interstation reliability (0.69) for a homogeneous group of examinees. Both the overall checklist and global rating scores effectively discriminated between CR and GS residents (P < 0.01), with 27% of the variance attributed to level of training. Nine CR residents but only 3 GS residents were deemed competent. Conclusions: The Colorectal Objective Structured Assessment of Technical Skill effectively discriminated between CR and GS residents. With further validation, the Colorectal Objective Structured Assessment of Technical Skill could be incorporated into the colorectal board examination where it would be the first attempt of a surgical specialty to formally assess technical skill at the time of certification.

Implementing and Evaluating a National Certification Technical Skills Examination: The Colorectal Objective Structured Assessment of Technical Skill.

Objective: To implement the Colorectal Objective Structured Assessment of Technical skill (COSATS) into American Board of Colon and Rectal Surgery (ABCRS) certification and build evidence of validity for the interpretation of the scores of this high stakes assessment tool. Background Data: Currently, technical skill assessment is not a formal component of board certification. With the technical demands of surgical specialties, documenting competence in technical skill at the time of certification with a valid tool is ideal. Methods: In September 2014, the COSATS was a mandatory component of ABCRS certification. Seventy candidates took the examination, with their performance evaluated by expert colorectal surgeons using a task-specific checklist, global rating scale, and overall performance scale. Passing scores were set and compared using 2 standard setting methodologies, using a compensatory and conjunctive model. Inter-rater reliability and the reliability of the pass/fail decision were calculated using Cronbach alpha and Subkoviak methodology, respectively. Overall COSATS scores and pass/fail status were compared with results on the ABCRS oral examination. Results: The pass rate ranged from 85.7% to 90%. Inter-rater reliability (0.85) and reliability of the pass/fail decision (0.87 and 0.84) were high. A low positive correlation (r= 0.25) was seen between the COSATS and oral examination. All individuals who failed the COSATS passed the ABCRS oral examination. Conclusions: COSATS is the first technical skill examination used in national surgical board certification. This study suggests that the current certification process may be failing to identify individuals who have demonstrated technical deficiencies on this standardized assessment tool.

Addressing the severe shortage of health care providers in Ethiopia: bench model teaching of technical skills

Context  There is a severe shortage of health care workers in Ethiopia. This situation must be addressed by the efficient training of mass cohorts of students.

Surgical Simulation Lab: Feasibility and Management

The management and administration of a simulation centre is both complex and unique. Centres globally struggle in their capacity to deliver quality training within an economical framework. Challenges to educational skill training also include financial stability, faculty teaching support and inclusion of validated assessment tools, to name but a few. The University of Toronto Surgical Skills Centre at Mount Sinai Hospital has operated for two decades, and it supports a robust array of educational activities, including boot camps. This chapter can be of great use for simulation centres, large and small, as it offers insight, tips, and advice on how to establish and operate a successful skills lab.

Surgical Preparatory Camp (PREP Camp)

In 2013, the Department of Surgery at the University of Toronto launched its first department-wide boot camp, a program that is now offered to all incoming surgical residents. As an integral part of surgical training in Toronto, the course is mandatory for all first-year residents and serves over 50 trainees each July. In this chapter, we review the background and principles behind this ambitious course, and we offer the information needed to establish a similar skills course at other institutions.

Board 214 - Program Innovations Abstract Title: Surgical Preparatory Camp

Introduction/Background Recent developments within the medical education system are having a stifling effect on surgical training. Reduced work hours1-3 for surgical trainees, coupled with an increased focus on patient safety inevitably Results in fewer educational opportunities and decreased quality of learning experiences for new trainees. Today’s surgical educators are actively searching for learning opportunities that extend beyond the operating room. Simulation-based programs allow practice at an individual pace in a risk-free environment and present an important supplement to traditional clinical teaching.4-7 One such program is the Surgical Prep Camp (SPC), a new simulation-based course that aims to prepare all incoming surgical trainees at the University of Toronto for their residency. The Surgical Prep Camp was developed by experts in surgical education and skill acquisition and has evolved from the award-winning Toronto Orthopaedic Boot Camp program.8-9 Methods All 54 first year surgical trainees participated in SPC, which took place at the onset of their residency. The program was delivered at the University of Toronto Surgical Skills Centre at Mount Sinai Hospital. The Surgical Prep Camp program focused on core skills that Program Directors agreed were essential for all surgical residents and created a unique learning environment to accelerate the development of these skills. For two weeks, residents from all specialities practiced suturing, central line insertion, surgical airway and other fundamental skills. The program also included didactic sessions encompassing key concepts in surgery. These two weeks were followed by an additional training period (the length of which was determined by each surgical division) which focused on developing speciality-specific skills. Trainees’ progress was closely monitored. Throughout the course, the residents and instructors were encouraged to complete interim skills assessments using electronic progress logs. Upon completion of the SPC program, trainees completed a skills examination. A retention examination will be conducted in eight months time to determine how well the skills persist. Additionally, extensive feedback was collected to help improve future iterations of SPC. We present preliminary data from our early experiences with SPC. Skills examination performance Results were impressive with a normalized mean total checklist score of 0.85 (0.15) (out of a possible 1). Nonetheless, two tasks (chest tube insertion and tracheostomy) proved to be particularly challenging. We also present detailed feedback from our residents and staff which will offer insight into the program’s development and implementation. Results: Conclusion The ultimate goal of SPC is to enhance patient safety and produce more competent surgeons who are better prepared for clinical practice. This program provides new trainees with a sound foundation upon which they can build their technical and clinical skills. Beyond providing an advantage at the beginning of training, the program seeks to instill effective learning habits that allow residents to engage in meaningful deliberate practice. Focusing on technical skills at the beginning of residency aims to allow trainees to advance to more complex tasks earlier within residency, which provides a much richer educational experience than has previously been possible.10 This is achieved in a manner which does not produce significant additional load on faculty since the teaching is shared between staff surgeons, fellows, senior residents and members of the allied healthcare team. Both this program and its predecessor have generated much interest in the surgical education community. Early evidence suggests that simulation-based programs can have a profound positive impact on residency training across all specialities. We believe that more widespread adoption of such programs will follow, helping to ensure that our future surgeons are better prepared to face the challenges that lie ahead. References 1. Calman KC, Temple JG, Naysmith R, Cairncross RG and Bennett SJ: Reforming higher specialist training in the United Kingdom - a step along the continuum of medical education. Med Educ 1999; 33: 28–33. 2. Pickersgill T: The European working time directive for doctors in training. BMJ 2001; 323(7324):1266. 3. Irani JL et al.: Surgical residents’ perceptions of the effects of the ACGME duty hour requirements 1 year after implementation. Surgery 2005; 138(2):246-253. 4. Carter BN: The fruition of Halsted’s concept of surgical training. Surgery 1952; 32(3): 518–527. 5. Reznick RK, MacRae H: Medical education - Teaching surgical skills - Changes in the wind. N Engl J Med 2006; 355(25): 2664–2669. 6. Ericsson KA, Krampe RT, Tesch-romer C: The role of deliberate practice in the acquisition of expert performance. Psychological Review 1993; 100(3): 363–406. 7. Ahlberg G, Enochsson L, Gallagher AG, et al.: Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg 2007; 193(6):797–804. 8. Sonnadara RR, Van Vliet A, Safir O, et al.: Orthopedic boot camp: examining the effectiveness of an intensive surgical skills course. Surgery 2011; 149(6): 745–749. 9. Sonnadara RR, Garbedian S, Safir O, et al.: Orthopaedic Boot Camp II: examining the retention rates of an intensive surgical skills course. Surgery 2012; 151(6): 803–7. 10. Sonnadara RR, Garbedian S, Safir O, Mui C, Mironova P, Nousiainen M, Ferguson P, Kraemer W, Alman B and Reznick R: Orthopaedic Boot Camp III: Examining the efficacy of self-regulated learning during an intensive laboratory-based surgical skills course. Surgery (In Press). Disclosures None.