Kristy Pugh

Development and Validation of Trauma Surgical Skills Metrics: Preliminary Assessment of Performance after Training

BACKGROUND Maintaining trauma-specific surgical skills is an ongoing challenge for surgical training programs. An objective assessment of surgical skills is needed. We hypothesized that a validated surgical performance assessment tool could detect differences following a training intervention. METHODS We developed surgical performance assessment metrics based on discussion with expert trauma surgeons, video review of 10 experts and 10 novice surgeons performing three vascular exposure procedures and lower extremity fasciotomy on cadavers, and validated the metrics with interrater reliability testing by five reviewers blinded to level of expertise and a consensus conference. We tested these performance metrics in 12 surgical residents (Year 3–7) before and 2 weeks after vascular exposure skills training in the Advanced Surgical Skills for Exposure in Trauma (ASSET) course. Performance was assessed in three areas as follows: knowledge (anatomic, management), procedure steps, and technical skills. Time to completion of procedures was recorded, and these metrics were combined into a single performance score, the Trauma Readiness Index (TRI). Wilcoxon matched-pairs signed-ranks test compared pretraining/posttraining effects. RESULTS Mean time to complete procedures decreased by 4.3 minutes (from 13.4 minutes to 9.1 minutes). The performance component most improved by the 1-day skills training was procedure steps, completion of which increased by 21%. Technical skill scores improved by 12%. Overall knowledge improved by 3%, with 18% improvement in anatomic knowledge. TRI increased significantly from 50% to 64% with ASSET training. Interrater reliability of the surgical performance assessment metrics was validated with single intraclass correlation coefficient of 0.7 to 0.98. CONCLUSION A trauma-relevant surgical performance assessment detected improvements in specific procedure steps and anatomic knowledge taught during a 1-day course, quantified by the TRI. ASSET training reduced time to complete vascular control by one third. Future applications include assessing specific skills in a larger surgeon cohort, assessing military surgical readiness, and quantifying skill degradation with time since training.

Using an Individual Procedure Score Before and After the Advanced Surgical Skills Exposure for Trauma Course Training to Benchmark a Hemorrhage-Control Performance Metric

OBJECTIVE Test with an individual procedure score (IPS) to assess whether an unpreserved cadaver trauma training course, including upper and lower limb vascular exposure, improves correct identification of surgical landmarks, underlying anatomy, and shortens time to vascular control. DESIGN Prospective study of performance of 3 vascular exposure and control procedures (axillary, brachial, and femoral arteries) using IPS metrics by 2 colocated and trained evaluators before and after training with the Advanced Surgical Skills Exposure for Trauma (ASSET) course. IPS, including identification of anatomical landmarks, incisions, underlying structures, and time to completion of each procedure was compared before and after training using repeated measurement models. SETTING Audio-video instrumented cadaver laboratory at University of Maryland School of Medicine. PARTICIPANTS A total of 41 second to sixth year surgical residents from surgical programs throughout Mid-Atlantic States who had not previously taken the ASSET course were enrolled, 40 completed the pre- and post-ASSET performance evaluations. RESULTS After ASSET training, all components of IPS increased and time shortened for each of the 3 artery exposures. Procedure steps performed correctly increased 57%, anatomical knowledge increased 43% and skin incision to passage of a vessel loop twice around the correct vessel decreased by a mean of 2.5 minutes. An overall vascular trauma readiness index, a comprehensive IPS score for 3 procedures increased 28% with ASSET Training. CONCLUSIONS Improved knowledge of surface landmarks and underlying anatomy is associated with increased IPS, faster procedures, more accurate incision placement, and successful vascular control. Structural recognition during specific procedural steps and anatomical knowledge were key points learned during the ASSET course. Such training may accelerate acquisition of specific trauma surgery skills to compensate for shortened training hours, infrequent exposure to major vascular injuries, or when just-in-time training is necessary. IPS is a benchmark for competence in extremity vascular control.

Performance of Vascular Exposure and Fasciotomy Among Surgical Residents Before and After Training Compared With Experts

Importance Surgical patient outcomes are related to surgeon skills. Objective To measure resident surgeon technical and nontechnical skills for trauma core competencies before and after training and up to 18 months later and to compare resident performance with the performance of expert traumatologists. Design, Setting, and Participants This longitudinal study performed from May 1, 2013, through February 29, 2016, at Maryland State Anatomy Board cadaver laboratories included 40 surgical residents and 10 expert traumatologists. Interventions Performance was measured during extremity vascular exposures and lower extremity fasciotomy in fresh cadavers before and after taking the Advanced Surgical Skills for Exposure in Trauma (ASSET) course. Main Outcomes and Measures The primary outcome variable was individual procedure score (IPS), with secondary outcomes of IPSs on 5 components of technical and nontechnical skills, Global Rating Scale scores, errors, and time to complete the procedure. Two trained evaluators located in the same laboratory evaluated performance with a standardized script and mobile touch-screen data collection. Results Thirty-eight (95%) of 40 surgical residents (mean [SD] age, 31 [2.9] years) who were evaluated before and within 4 weeks of ASSET training completed follow-up evaluations 12 to 18 months later (mean [SD], 14 [2.7] months). The experts (mean [SD] age, 52 [10.0] years) were significantly older and had a longer (mean [SD], 46 [16.3] months) interval since taking the ASSET course (both P < .001). Overall resident cohort performance improved with increased anatomy knowledge, correct procedural steps, and decreased errors from 60% to 19% after the ASSET course regardless of clinical year of training (P < .001). For 21 of 40 residents (52%), correct vascular procedural steps plotted against anatomy knowledge (the 2 IPS components most improved with training) indicates the resident’s performance was within 1 nearest-neighbor classifier of experts after ASSET training. Five residents had no improvement with training. The Trauma Readiness Index for experts (mean [SD], 74 [4]) was significantly different compared with the trained residents (mean [SD], 48 [7] before training vs 63 [7] after training [P = .004] and vs 64 [6] 14 months later [P = .002]). Critical errors that might lead to patient death were identified by pretraining IPS decile of less than 0.5. At follow-up, frequency of resident critical errors was no different from experts. The IPSs ranged from 31.6% to 76.9% among residents for core trauma competency procedures. Modeling revealed that interval experience, rather than time since training, affected skill retention up to 18 months later. Only 4 experts and 16 residents (40%) adequately decompressed and confirmed entry into all 4 lower extremity compartments, Conclusions and Relevance This study found that ASSET training improved resident procedural skills for up to 18 months. Performance was highly variable. Interval experience after training affected performance. Pretraining skill identified competency of residents vs experts. Extremity vascular and fasciotomy performance evaluations suggest the need for specific anatomical training interventions in residents with IPS deciles less than 0.5.

Can Hyper-realistic Physical Models of Peripheral Vessel Exposure and Fasciotomy Replace Cadavers for Performance Assessment?

BACKGROUND Work-hour restrictions have reduced operative experience for residents. The Advanced Surgical Skills for Exposure in Trauma (ASSET) course fills this training gap. Cadaver use has limitations including cost and availability. Hyper-realistic synthetic models may provide an alternative to cadavers. We compared same surgeon performance between synthetic and cadaveric models to determine interchangeability for formative evaluation. METHODS Forty residents (<4 weeks after ASSET) and 35 faculty (mean, 2.5 ± 1.3 years after ASSET) exposed axillary, brachial, and femoral arteries, and performed lower extremity fasciotomy. Separate evaluators and random starting order between models were used for participants. Individual procedure scores and aggregate procedure scores, a trauma readiness index, evaluated participants. Students t and &khgr;2 tests were used where appropriate. p Values less than 0.05 were considered significant. RESULTS For same surgeons, faculty, but not residents, had higher trauma readiness index on the synthetic model (0.63 vs. 0.70, p < 0.01; 0.63 vs. 0.67, p = 0.06, respectively). Scores were not significantly different between models for residents except for the brachial artery exposure (0.68 vs. 0.75, p < 0.01), which was the least realistic of all procedures. Faculty did significantly better on the synthetic model in all procedures. All participants completed procedures nearly twice as quickly (5.61 ± 3.21 vs. 10.08 ± 4.66 minutes) and performed fewer errors on the synthetic model (113 vs. 53, p < 0.01; 118 vs. 76, p = 0.03, respectively). CONCLUSION Same surgeons performed procedures quicker and with fewer errors on the synthetic model. Residents performed similarly on both model types, this likely represents the unfamiliarity neophytes bring to new procedures. This suggests that the synthetic model, with easily discernible and standardized anatomy, may be useful in the early stages of training to understand critical procedural steps. The difficulty of the cadaver is more apt to assess and evaluate the experienced surgeon and identify opportunities for improvement. LEVEL OF EVIDENCE Prognostic, level III.