Reed G. Williams

The measured effect of delay in completing operative performance ratings on clarity and detail of ratings assigned

PURPOSEnOperative performance ratings (OPRs) need adequate clarity and detail to support self-directed learning and valid progress decisions. This study was designed to determine (1) the elapsed time between observing operative performances and completing performance ratings under field conditions and (2) the effect of increased elapsed time on rating clarity and detail.nnnMETHODSnOverall, 895 OPRs by 19 faculty members for 37 general surgery residents were the focus of this study. The elapsed time between observing the performance and completing the evaluation was recorded. No-delay comparison data included 45 additional ratings of 8 performances collected under controlled conditions immediately following the performance by 17 surgeons whose sole responsibility was to observe and rate the performances. Item-to-item OPR variation and the presence and nature of comments were indicators of evaluation clarity, detail, and quality.nnnRESULTSnElapsed time between observing and evaluating performances under field conditions were as follows: 1 day or less, 116 performances (13%); 2 to 3 days, 178 performances (20%); 4 to 14 days, 377 performances (42%); and more than 14 days, 224 performances (25%). Overall, 87% of performances rated more than 14 days after observation had no item-to-item ratings variation compared with 62% rated with a delay of 4 to 14 days, 41% rated with a delay of 2 to 3 days, 42% rated within 1 day, and 2% rated immediately. In addition, 70% of ratings completed more than 14 days after observation had no written comments, compared with 49% for those completed with a delay of 4 to 14 days, 45% for those completed in 2 to 3 days, and 46% for those completed within 1 day. Moreover, 47% of comments submitted after more than 14 days were exclusively global comments (less instructionally useful) compared with 7% for those completed with a delay of 4 to 14 days and 5% for those completed in 1 to 3 days.nnnCONCLUSIONSnThe elapsed time between observation and rating of operative performances should be recorded. Immediate ratings should be encouraged. Ratings completed more than 3 days after observation should be discouraged and discounted, as they lack clarity and detail about the performance.

Impact of a structured skills laboratory curriculum on surgery residents' intraoperative decision-making and technical skills.

Background This project sought to study the effectiveness of a curriculum to enhance the intraoperative clinical judgment and procedural skill of surgical residents. Method A multiinstitutional, prospective, randomized study was performed. A cognitive task analysis of laparoscopic cholecystectomy (LC) was conducted on which instructional activities and measurement instruments were designed. Residents were randomly assigned to a control or intervention group. Subjects took written pre- and posttests examining procedure-related judgment and knowledge. The intervention group participated in a three-session curriculum emphasizing LC critical decisions and error prevention. All subjects were evaluated performing the procedure on a cadaveric model. Scores from written and practical exams were compared using independent-sample and paired Student t tests. Results Written examination scores increased for both groups. The intervention group scored significantly higher (P < .05) on the written posttest than the control group. There were no differences between groups on the practical examination. Reliability coefficients for the written examination ranged from .65 to .75. Reliability coefficients for the oral exam, technical skill, and error items on the porcine practical exam were .83, .90, and .53. Conclusions The curriculum resulted in enhanced performance on a written exam designed to assess intraoperative judgment, but no differences in technical skills, showing important implications for future skills lab curriculum formats.

The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial

PURPOSEnIntraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs.nnnMETHODSnBetween September 1, 2015 and February 29, 2016, 15 U.S. general surgery residency programs were enrolled in an institutional review board-approved trial. SIMPL was made available after 70% of faculty and residents completed a 1-hour training session. Descriptive and univariate statistics analyzed multiple dimensions of feasibility, including training rates, volume of assessments, response rates/times, and dictation rates. The 20 most active residents and attendings were evaluated in greater detail.nnnRESULTSnA total of 90% of eligible users (1267/1412) completed training. Further, 13/15 programs began using SIMPL. Totally, 6024 assessments were completed by 254 categorical general surgery residents (n = 3555 assessments) and 259 attendings (n = 2469 assessments), and 3762 unique operations were assessed. There was significant heterogeneity in participation within and between programs. Mean percentage (range) of users who completed ≥1, 5, and 20 assessments were 62% (21%-96%), 34% (5%-75%), and 10% (0%-32%) across all programs, and 96%, 75%, and 32% in the most active program. Overall, response rate was 70%, dictation rate was 24%, and mean response time was 12 hours. Assessments increased from 357 (September 2015) to 1146 (February 2016). The 20 most active residents each received mean 46 assessments by 10 attendings for 20 different procedures.nnnCONCLUSIONSnSIMPL can be feasibly integrated into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. We believe SIMPL could help facilitate a national competency-based surgical training system, although local and systemic challenges still need to be addressed.

How much guidance is given in the operating room? Factors influencing faculty self-reports, resident perceptions, and faculty/resident agreement

BACKGROUNDnGuidance in the operating room impacts resident confidence and ability to function independently. The purpose of this study was to explore attending surgeon guidance practices in the operating room as reported by faculty members themselves and by junior and senior residents.nnnMETHODSnThis was an exploratory, cross-sectional survey research study involving 91 categorical residents and 82 clinical faculty members at two academic general surgery training programs. A series of analyses of variance along with descriptive statistics were performed to understand the impact of resident training year, program, and surgeon characteristics (sex and type of surgery performed routinely) on guidance practices.nnnRESULTSnResident level (junior versus senior) significantly impacted the amount of guidance given as reported by faculty and as perceived by residents. Within each program, junior residents perceived less guidance than faculty reported giving. For senior guidance practices, however, the differences between faculty and resident practices varied by program. In terms of the effects of surgeon practice type (mostly general versus mostly complex cases), residents at both institutions felt they were more supervised closely by the faculty who perform mostly complex cases.nnnCONCLUSIONnMore autonomy is given to senior than to junior residents. Additionally, faculty report a greater amount of change in their guidance practices over the training period than residents perceive. Faculty and resident agreement about the need for guidance and for autonomy are important for achieving the goals of residency training.

Do Residents Receive the Same OR Guidance as Surgeons Report? Difference Between Residents’ and Surgeons’ Perceptions of OR Guidance

PURPOSEnOperating room (OR) guidance is important for surgical residents performance and, ultimately, for the development of independence and autonomy. This study explores the differences in surgical residents and attending surgeons perceptions of OR guidance in prerecorded surgical cases.nnnMETHODSnA total of 9 attending surgeons and 8 surgical residents observed 8 prerecorded surgical cases and were asked to identify both the presence and the type of attending surgeons OR guidance. Each recorded case was observed by 2 attending surgeons and 1 resident. A previously developed taxonomy for types of OR guidance was applied to analyze the data to explore the difference. Agreement by both attending surgeons on the presence and the type of OR guidance served as the concordant guidance behaviors to which the responses of the residents were compared.nnnRESULTSnOverall, 116 OR guidance events were identified. Attending surgeons agreed on the presence of guidance in 80 of 116 (69.8%) events and consistently identified the type of OR guidance in 91.4% (73/80, Cohen κ = 0.874) of them. However, surgical residents only agreed with attending surgeons on the presence of guidance in 61.25% (49/80) of the events. In addition, there was significant disagreement (Cohen κ = 0.319) between surgical residents and attending surgeons in the type of OR guidance; the residents only identified 54.8% (40/73) of concordant guidance behaviors in the same guidance category as both the surgeons. Among the types of OR guidance, residents and attending surgeons were most likely to agree on the teaching guidance (66.67%) and least likely to agree on the assisting guidance (36.84%).nnnCONCLUSIONSnSurgical residents and attending surgeons have different perceptions of both the presence and the type of OR guidance. This difference in perception of OR guidance has important implications for the efficiency of training surgical residents in the OR, and, ultimately on residents development of independence and autonomy.

Readiness of US General Surgery Residents for Independent Practice

Objective: This study evaluates the current state of the General Surgery (GS) residency training model by investigating resident operative performance and autonomy. Background: The American Board of Surgery has designated 132 procedures as being “Core” to the practice of GS. GS residents are expected to be able to safely and independently perform those procedures by the time they graduate. There is growing concern that not all residents achieve that standard. Lack of operative autonomy may play a role. Methods: Attendings in 14 General Surgery programs were trained to use a) the 5-level System for Improving and Measuring Procedural Learning (SIMPL) Performance scale to assess resident readiness for independent practice and b) the 4-level Zwisch scale to assess the level of guidance (ie, autonomy) they provided to residents during specific procedures. Ratings were collected immediately after cases that involved a categorical GS resident. Data were analyzed using descriptive statistics and supplemented with Bayesian ordinal model-based estimation. Results: A total of 444 attending surgeons rated 536 categorical residents after 10,130 procedures. Performance: from the first to the last year of training, the proportion of Performance ratings for Core procedures (n = 6931) at “Practice Ready” or above increased from 12.3% to 77.1%. The predicted probability that a typical trainee would be rated as Competent after performing an average Core procedure on an average complexity patient during the last week of residency training is 90.5% (95% CI: 85.7%–94%). This falls to 84.6% for more complex patients and to less than 80% for more difficult Core procedures. Autonomy: for all procedures, the proportion of Zwisch ratings indicating meaningful autonomy (“Passive Help” or “Supervision Only”) increased from 15.1% to 65.7% from the first to the last year of training. For the Core procedures performed by residents in their final 6 months of training (cholecystectomy, inguinal/femoral hernia repair, appendectomy, ventral hernia repair, and partial colectomy), the proportion of Zwisch ratings (n = 357) indicating near-independence (“Supervision Only”) was 33.3%. Conclusions: US General Surgery residents are not universally ready to independently perform Core procedures by the time they complete residency training. Progressive resident autonomy is also limited. It is unknown if the amount of autonomy residents do achieve is sufficient to ensure readiness for the entire spectrum of independent practice.

Is a Single-Item Operative Performance Rating Sufficient?

OBJECTIVEnA valid measure of resident operative performance ability requires direct observation and accurate rating of multiple resident performances under the normal range of operating conditions. The challenge is to create an operative performance rating (OPR) system that: is easy to use, encourages completion of many ratings immediately after performances and minimally disrupts supervising surgeons work days. The purpose of this study was to determine whether a score based on a single-item overall OPR provides a valid and stable appraisal of resident operative performances.nnnDESIGNnA retrospective comparison of a single-item OPR with a gold-standard rating based on multiple procedure-specific and general OPR items.nnnSETTINGnData were collected in the general surgery residency program at Southern Illinois University from 2001 through 2012.nnnPARTICIPANTSnAssessments of 1033 operative performances (3 common procedures, 2 laparoscopic, and 1 open) by general surgery residents were collected. OPRs based on single-item overall performance scale scores were compared with gold-standard ratings for the same performances.nnnRESULTSnDifferences in performance scores using the 2 scales averaged 0.02 points (5-point scale). Correlations of the single-item and gold-standard scale scores averaged 0.95. Based on generalizability analyses of laparoscopic cholecystectomy ratings, each instrument required 5 observations to achieve reliabilities of 0.80 and 11 observations to achieve reliabilities of 0.90. Only 4.4% of single-item ratings misclassified the performance when compared with the gold-standard rating and all misclassifications were near misses. For 80% of misclassified ratings, single-item ratings were lower.nnnCONCLUSIONSnSingle-item operative performance measures produced ratings that were virtually identical to gold-standard scale ratings. Misclassifications occurred infrequently and were minor in magnitude. Ratings using the single-item scale: take less time to complete, should increase the sample of procedures rated, and encourage attending surgeons to complete ratings immediately after observing performances. Face-to-face and written comments and suggestions should continue to be used to provide the granular feedback residents need to improve subsequent performances.

A taxonomy of surgeons' guiding behaviors in the operating room

BACKGROUNDnThis study explores the nature and the intention of attending surgeons guiding behaviors performed in the operating room (OR) in order to build taxonomy of OR guiding behavior.nnnMETHODSnNine attending surgeons and 8 surgical residents were invited to observe 8 prerecorded surgical cases from 4 common procedures and completed semistructured interviews. All video-based observations were videotaped. Thematic analysis was applied to identify surgeons OR guiding behavior.nnnRESULTSnSeven hundred eighty minutes of video-based observations with interviews were conducted. Sixteen types of OR guiding behaviors in 3 intention-based categories were identified: 3 of the 16 was teaching (18.75%), 8 of the 16 was directing (50%), and 5 of the 16 was assisting (31.25%).nnnCONCLUSIONSnSurgeons OR guiding behaviors were grounded in 3 behavioral intentions: teaching, directing, and assisting. This taxonomy of OR guiding behavior can be used as a basis for developing OR guiding strategy to improve residents intraoperative competency, autonomy, and independence.

Practice Guidelines for Operative Performance Assessments.

Objective: To provide recommended practice guidelines for assessing single operative performances and for combining results of operative performance assessments into estimates of overall operative performance ability. Summary Background Data: Operative performance is one defining characteristic of surgeons. Assessment of operative performance is needed to provide feedback with learning benefits to surgical residents in training and to assist in making progress decisions for residents. Operative performance assessment has been a focus of investigation over the past 20 years. This review is designed to integrate findings of this research into a set of recommended operative performance practices. Methods: Literature from surgery and from other pertinent research areas (psychology, education, business) was reviewed looking for evidence to inform practice guideline development. Guidelines were created along with a conceptual and scientific foundation for each guideline. Results: Ten guidelines are provided for assessing individual operative performances and 10 are provided for combing data from individual operative performances into overall judgments of operative performance ability. Conclusions: The practice guidelines organize available information to be immediately useful to program directors, to support surgical training, and to provide a conceptual framework upon which to build as the base of pertinent knowledge expands through future research and development efforts.

Dissecting Attending Surgeons’ Operating Room Guidance: Factors That Affect Guidance Decision Making

PURPOSEnThe amount of guidance provided by the attending surgeon in the operating room (OR) is a key element in developing residents autonomy. The purpose of this study is to explore factors that affect attending surgeons decision making regarding OR guidance provided to the resident.nnnMETHODSnWe used video-stimulated recall interviews (VSRI) throughout this 2-phase study. In Phase 1, 3 attending surgeons were invited to review separately 30 to 45 minute video segments of their prerecorded surgical operations to explore factors that influenced their OR guidance decision making. In Phase 2, 3 attending surgeons were observed and documented in the OR (4 operations, 341min). Each operating surgeon reviewed their videotaped surgical performance within 5 days of the operation to reflect on factors that affected their decision making during the targeted guidance events. All VSRI were recorded. Thematic analysis and manual coding were used to synthesize and analyze data from VSRI transcripts, OR observation documents, and field notes.nnnRESULTSnA total of 255 minutes of VSRI involving 6 surgeons and 7 surgical operations from 5 different procedures were conducted. A total of 13 guidance decision-making influence factors from 4 categories were identified (Cohens κ = 0.674): Setting (case schedule and patient morbidity), content (procedure attributes and case progress), resident (current competency level, trustworthiness, self-confidence, and personal traits), and attending surgeon (level of experience, level of comfort, preferred surgical technique, OR training philosophy, and responsibility as surgeon). A total of 5 factors (case schedule, patient morbidity, procedure attributes, resident current competency level, and trustworthiness) influenced attending surgeons pre-OR guidance plans. OR training philosophy and responsibility as surgeon were anchor factors that affected attending surgeons OR guidance decision-making patterns.nnnCONCLUSIONSnSurgeons OR guidance decision making is a dynamic process that is influenced by 13 situational factors. These factors can be used by residency programs to tailor strategies designed to increase resident autonomy in the OR.