Katherine E. Law

The use of error analysis to assess resident performance

BACKGROUNDnThe aim of this study was to assess validity of a human factors error assessment method for evaluating resident performance during a simulated operative procedure.nnnMETHODSnSeven postgraduate year 4-5 residents had 30 minutes to complete a simulated laparoscopic ventral hernia (LVH) repair on day 1 of a national, advanced laparoscopic course. Faculty provided immediate feedback on operative errors and residents participated in a final product analysis of their repairs. Residents then received didactic and hands-on training regarding several advanced laparoscopic procedures during a lecture session and animate lab. On day 2, residents performed a nonequivalent LVH repair using a simulator. Three investigators reviewed and coded videos of the repairs using previously developed human error classification systems.nnnRESULTSnResidents committed 121 total errors on day 1 compared with 146 on day 2. One of 7 residents successfully completed the LVH repair on day 1 compared with all 7 residents on day 2 (P = .001). The majority of errors (85%) committed on day 2 were technical and occurred during the last 2 steps of the procedure. There were significant differences in error type (P ≤ .001) and level (P = .019) from day 1 to day 2. The proportion of omission errors decreased from day 1 (33%) to day 2 (14%). In addition, there were more technical and commission errors on day 2.nnnCONCLUSIONnThe error assessment tool was successful in categorizing performance errors, supporting known-groups validity evidence. Evaluating resident performance through error classification has great potential in facilitating our understanding of operative readiness.

Relationship Between Technical Errors and Decision-Making Skills in the Junior Resident

OBJECTIVEnThe purpose of this study is to coevaluate resident technical errors and decision-making capabilities during placement of a subclavian central venous catheter (CVC). We hypothesize that there would be significant correlations between scenario-based decision-making skills and technical proficiency in central line insertion. We also predict residents would face problems in anticipating common difficulties and generating solutions associated with line placement.nnnDESIGNnParticipants were asked to insert a subclavian central line on a simulator. After completion, residents were presented with a real-life patient photograph depicting CVC placement and asked to anticipate difficulties and generate solutions. Error rates were analyzed using chi-square tests and a 5% expected error rate. Correlations were sought by comparing technical errors and scenario-based decision-making skills.nnnSETTINGnThis study was performed at 7 tertiary care centers.nnnPARTICIPANTSnStudy participants (N = 46) largely consisted of first-year research residents who could be followed longitudinally. Second-year research and clinical residents were not excluded.nnnRESULTSnIn total, 6 checklist errors were committed more often than anticipated. Residents committed an average of 1.9 errors, significantly more than the 1 error, at most, per person expected (t(44) = 3.82, p < 0.001). The most common error was performance of the procedure steps in the wrong order (28.5%, p < 0.001). Some of the residents (24%) had no errors, 30% committed 1 error, and 46 % committed more than 1 error. The number of technical errors committed negatively correlated with the total number of commonly identified difficulties and generated solutions (r (33) = -0.429, p = 0.021, r (33) = -0.383, p = 0.044, respectively).nnnCONCLUSIONSnAlmost half of the surgical residents committed multiple errors while performing subclavian CVC placement. The correlation between technical errors and decision-making skills suggests a critical need to train residents in both technique and error management.

Advanced Engineering Technology for Measuring Performance.

The demand for competency-based assessments in surgical training is growing. Use of advanced engineering technology for clinical skills assessment allows for objective measures of hands-on performance. Clinical performance can be assessed in several ways via quantification of an assessees hand movements (motion tracking), direction of visual attention (eye tracking), levels of stress (physiologic marker measurements), and location and pressure of palpation (force measurements). Innovations in video recording technology and qualitative analysis tools allow for a combination of observer- and technology-based assessments. Overall the goal is to create better assessments of surgical performance with robust validity evidence.

A Comprehensive Methodology for Examining the Impact of Surgical Team Briefings and Debriefings on Teamwork

The adoptions of briefing and debriefing protocols have evolved from the Joint Commission’s initiative to improve communication and safety in the operating room. Briefing normally occurs prior to incision and is used to discuss and confirm critical information, while debriefing occurs during or after surgery. Debriefing provides a unique opportunity for individuals and teams to immediately reflect on their performance, allowing them to more easily identify errors and develop plans to improve their next performance. Studies have shown that using briefings and debriefings improve communication and teamwork. However, there is still much to learn about the value of both for surgical teams. This paper presents a robust methodology for examining and measuring the impacts of surgical team briefings and debriefings on teamwork. The methodology includes (1) audio/video recording the surgical care process, (2) prospective observations using a validated electronic data collection tool, (3) pre- and post-surgery surveys, and (4) individual surgical team member interviews. The current paper describes the methodology to obtain a robust and comprehensive data set for analyzing the impacts of briefing and debriefing on teamwork; the results of the surgeries recorded using this methodology will be presented in subsequent papers.

Exploring Senior Residents’ Intraoperative Error Management Strategies: A Potential Measure of Performance Improvement

OBJECTIVEnThe study aim was to determine whether residents error management strategies changed across 2 simulated laparoscopic ventral hernia (LVH) repair procedures after receiving feedback on their initial performance. We hypothesize that error detection and recovery strategies would improve during the second procedure without hands-on practice.nnnDESIGNnRetrospective review of participant procedural performances of simulated laparoscopic ventral herniorrhaphy. A total of 3 investigators reviewed procedure videos to identify surgical errors. Errors were deconstructed. Error management events were noted, including error identification and recovery.nnnSETTINGnResidents performed the simulated LVH procedures during a course on advanced laparoscopy. Participants had 30 minutes to complete a LVH procedure. After verbal and simulator feedback, residents returned 24 hours later to perform a different, more difficult simulated LVH repair.nnnPARTICIPANTSnSenior (N = 7; postgraduate year 4-5) residents in attendance at the course participated in this study.nnnRESULTSnIn the first LVH procedure, residents committed 121 errors (M = 17.14, standard deviation = 4.38). Although the number of errors increased to 146 (M = 20.86, standard deviation = 6.15) during the second procedure, residents progressed further in the second procedure. There was no significant difference in the number of errors committed for both procedures, but errors shifted to the late stage of the second procedure. Residents changed the error types that they attempted to recover (χ25=24.96, p<0.001). For the second procedure, recovery attempts increased for action and procedure errors, but decreased for strategy errors. Residents also recovered the most errors in the late stage of the second procedure (p < 0.001).nnnCONCLUSIONnResidents error management strategies changed between procedures following verbal feedback on their initial performance and feedback from the simulator. Errors and recovery attempts shifted to later steps during the second procedure. This may reflect residents error management success in the earlier stages, which allowed further progression in the second simulation. Incorporating error recognition and management opportunities into surgical training could help track residents learning curve and provide detailed, structured feedback on technical and decision-making skills.

Resident performance in complex simulated urinary catheter scenarios.

BACKGROUNDnUrinary catheter insertion is a common procedure performed in hospitals. Improper catheterization can lead to unnecessary catheter-associated urinary tract infections and urethral trauma, increasing patient morbidity. To prevent such complications, guidelines were created on how to insert and troubleshoot urinary catheters. As nurses have an increasing responsibility for catheter placement, resident responsibility has shifted to more complex scenarios. This study examines the clinical decision-making skills of surgical residents during simulated urinary catheter scenarios. We hypothesize that during urinary catheterization, residents will make inconsistent decisions relating to catheter choices and clinical presentations.nnnMETHODSnForty-five general surgery residents (postgraduate year 2-4) in Midwest training programs were presented with three of four urinary catheter scenarios of varying difficulty. Residents were allowed 15xa0min to complete the scenarios with five different urinary catheter choices. A chi-square test was performed to examine the relation between initial and subsequent catheter choices and to evaluate for consistency of decision-making for each scenario.nnnRESULTSnEighty-two percent of residents performed scenario A; 49% performed scenario B; 64% performed scenario C, and 82% performed scenario D. For initial attempt for scenario A-C, the 16 French Foley catheter was the most common choice (38%, 54%, 50%, Psxa0<xa00.001), whereas for scenario D, the 16 French Coude was the most common choice (37%, Pxa0<xa00.01). Residents were most likely to be successful in achieving urine output in the initial catheterization attempt (Pxa0<xa00.001). Chi-square analyses showed no relationship between residents first and subsequent catheter choices for each scenario (Psxa0>xa00.05).nnnCONCLUSIONSnEvaluation of clinical decision-making shows that initial catheter choice may have been deliberate based on patient background, as evidenced by the most popular choice in scenario D. Analyses of subsequent choices in each of the catheterization models reveal inconsistency. These findings suggest a possible lack of competence or training in clinical decision-making with regard to urinary catheter choices in residents.

Error tolerance: an evaluation of residents' repeated motor coordination errors

BACKGROUNDnThe study investigates the relationship between motor coordination errors and total errors using a human factors framework. We hypothesize motor coordination errors will correlate with total errors and provide validity evidence for error tolerance as a performance metric.nnnMETHODSnResidents laparoscopic skills were evaluated during a simulated laparoscopic ventral hernia repair for motor coordination errors when grasping for intra-abdominal mesh or suture. Tolerance was defined as repeated, failed attempts to correct an error and the time required to recover.nnnRESULTSnResidents (N = 20) committed an average of 15.45 (standard deviation [SD] = 4.61) errors and 1.70 (SD = 2.25) motor coordination errors during mesh placement. Total errors correlated with motor coordination errors (r[18] = .572, P = .008). On average, residents required 5.09 recovery attempts for 1 motor coordination error (SD = 3.15). Recovery approaches correlated to total error load (r[13] = .592, P = .02).nnnCONCLUSIONSnResidents motor coordination errors and recovery approaches predict total error load. Error tolerance proved to be a valid assessment metric relating to overall performance.

A Simulation-based, cognitive assessment of resident decision making during complex urinary catheterization scenarios

BACKGROUNDnThis study explores general surgery residents decision making skills in uncommon, complex urinary catheter scenarios.nnnMETHODSn40 residents were presented with two scenarios. Scenario A was a male with traumatic urethral injury and scenario B was a male with complete urinary blockage. Residents verbalized whether they would catheterize the patient and described the workup and management of suspected pathologies. Residents decision paths were documented and analyzed.nnnRESULTSnIn scenario A, 45% of participants chose to immediately consult Urology. 47.5% named five diagnostic tests to decide if catheterization was safe. In scenario B, 27% chose to catheterize with a 16 French Coude. When faced with catheterization failure, participants randomly upsized or downsized catheters. Chi-square analysis revealed no measurable consensus amongst participants.nnnCONCLUSIONSnResidents need more training in complex decision making for urinary catheterization. The decision trees generated in this study provide a useful blueprint of residents learning needs.nnnSUMMARYnExploration of general surgery residents decision making skills in uncommon, complex urinary catheter scenarios revealed major deficiencies. The resulting decision trees reveal residents learning needs.

Resident training in a teaching hospital: How do attendings teach in the real operative environment?

BACKGROUNDnThe study aim was to explore the nature of intraoperative education and its interaction with the environment where surgical education occurs.nnnMETHODSnVideo and audio recording captured teaching interactions between colorectal surgeons and general surgery residents during laparoscopic segmental colectomies. Cases and collected data were analyzed for teaching behaviors and workflow disruptions. Flow disruptions (FDs) are considered deviations from natural case progression.nnnRESULTSnAcross 10 cases (20.4 operative hours), attendings spent 11.2xa0hours (54.7%) teaching, using directing (Mxa0= 250.1), and confirming (Mxa0= 236.1) most. FDs occurred 410 times, accounting for 4.4xa0hours of case time (21.57%). Teaching occurred with FD events for 2.4xa0hours (22.2%), whereas 77.8% of teaching happened outside FD occurrence. Teaching methods shifted from active to passive during FD events to compensate for patient safety.nnnCONCLUSIONSnUnderstanding how FDs impact operative learning will inform faculty development in managing interruptions and improve its integration into resident education.

Hand smoothness in laparoscopic surgery correlates to psychomotor skills in virtual reality

The main purpose of this study is to find possible relationships between the smoothness of hand function during laparoscopic ventral hernia (LVH) repair and psychomotor skills in a defined virtual reality (VR) environment. Thirty four surgical residents N = 34 performed two scenarios. First, participants were asked to perform a simulated LVH repair during which their hand movement was tracked using electromagnetic sensors. Subsequently, the smoothness of hand function was calculated for each participants dominant and non-dominate hand. Then participants performed two modules in a defined VR environment, which assessed their force matching and target tracking capabilities. More smooth hand function during the LVH repair correlated positively with higher performance in VR modules. Also, translational smoothness of dominant hand is found as the most informative smoothness metric in the LVH repair scenario. Therefore, defined force matching and target tracking assessments in VR can potentially be used as an indirect assessment of fine motor skills in the LVH repair.