Matthew S. Holden

The Development and Validation of Hand Motion Analysis to Evaluate Competency in Central Line Catheterization

OBJECTIVES Traditionally, technical skills proficiency has been assessed by direct observation. While direct observation and feedback are essential components in technical skills learning, they demand considerable investment of faculty time, and as an assessment tool, direct observation is inherently subjective and has been criticized as unreliable. The purpose of this study was to determine if quantitative electromagnetic motion tracking is feasible and can discriminate between experts and nonexperts during simulated ultrasound (US)-guided insertion of a central venous catheter (CVC) guidewire. METHODS Ten nonexperts (junior emergency medicine residents) and 10 experts (critical care fellows or attending physicians) were recruited. Electromagnetic sensor probes were used to capture hand motion during an US-guided internal jugular cannulation on a standardized manikin. Hand, US, and needle motion were analyzed for the following metrics: total path length, total time, translational movements, and rotational movements. Subjects were also videotaped and evaluated using a modified, validated global rating scale (GRS) by a blinded expert. RESULTS There was a significant difference in almost all examined motion parameters between experts and nonexperts. Experts took 66% less time (50.2 seconds vs. 148.7 seconds, p < 0.001) and had significantly less right-hand and US motion (total path length and translational and rotational movements). Left-hand total path length was the only parameter that was not significantly different between groups. Concurrent validity of motion parameters was established by strong correlations (r2 > 0.74) to a previously published, modified GRS. CONCLUSIONS Electromagnetic hand and instrument motion analysis is technically feasible for assessing competence in US-guided insertion of a CVC guidewire in a simulation setting. In showing that it discriminates between the performances of nonexperts and experts, this study has provided evidence for construct validity. It also shows excellent correlation with a modified version of a previously validated GRS, providing evidence of concurrent validity.

Tracked Ultrasonography Snapshots Enhance Needle Guidance for Percutaneous Renal Access: A Pilot Study

BACKGROUND AND PURPOSE Although ultrasonography-guided percutaneous nephrostomy is relatively safe, a number of factors make it challenging for inexperienced operators. A computerized needle navigation technique using tracked ultrasonography snapshots was investigated to determine whether performance of percutaneous nephrostomy by inexperienced users could be improved. METHODS Ten operators performed the procedure on a phantom model with alternating needle guidance between conventional ultrasonography and tracked ultrasonography snapshots. The needle was reinserted until fluid backflow confirmed calyceal access. Needle trajectories were recorded using the real time needle navigation system for offline evaluation of operator performance. Recorded needle trajectories were used to measure needle motion path length inside the phantom tissue, number of reinsertions, total procedure time, and needle insertion time as end points of this study. RESULTS Needle path length measured inside the phantom tissue was significantly lower with ultrasonography snapshots guidance (295.0±23.1 mm, average±standard error of the mean) compared with control procedures (977.9±144.4 mm, P<0.01). This was associated with a significantly lower number of needle insertion attempts with ultrasonography snapshots (average 1.27±0.10 vs 2.83±0.31, P<0.01). The total procedure time and the needle insertion time were also significantly lower with ultrasonography snapshots guidance. CONCLUSION Tracked ultrasonography snapshots appear to improve the performance of percutaneous nephrostomy in these preliminary investigations, justifying further validation studies. The presented navigation system is reproducible because of commercially available hardware and open-source software components, facilitating its potential role in clinical practice.

Feasibility of Real-Time Workflow Segmentation for Tracked Needle Interventions

Computer-assisted training systems promote both training efficacy and patient health. An important component for providing automatic feedback in computer-assisted training systems is workflow segmentation: the determination of what task in the workflow is being performed. Our objective was to develop a workflow segmentation algorithm for needle interventions using needle tracking data. Needle tracking data were collected from ultrasound-guided epidural injections and lumbar punctures, performed by medical personnel. The workflow segmentation algorithm was tested in a simulated real-time scenario: the algorithm was only allowed access to data recorded at, or prior to, the time being segmented. Segmentation output was compared to the ground-truth segmentations produced by independent blinded observers. Overall, the algorithm was 93% accurate. It automatically segmented the ultrasound-guided epidural procedures with 81% accuracy and the lumbar punctures with 82% accuracy. Given that the manual segmentation consistency was only 84%, the algorithms accuracy was 93%. Using Cohens d statistic, a medium effect size (0.5) was calculated. Because the algorithm segments needle-based procedures with such high accuracy, expert observers can be augmented by this algorithm without a large decrease in ability to follow trainees in a workflow. The proposed algorithm is feasible for use in a computer-assisted needle placement training system.

Examination of Learning Trajectories for Simulated Lumbar Puncture Training Using Hand Motion Analysis

OBJECTIVES A prospective cohort study was conducted using hand motion analysis (HMA) to assess the acquisition and retention of technical proficiency among first-year medical students learning the lumbar puncture (LP) skill in a simulated setting. METHODS Twenty-five subjects attended three or four simulation sessions at 6-week intervals. The initial session consisted of an introduction to the procedure and a baseline HMA assessment. This was followed by a session involving deliberate practice and evaluation using HMA. Subject HMA metrics were followed over time and compared to performance benchmarks to determine the volume of practice required to achieve and maintain technical proficiency in the simulated setting. Performance benchmarks were derived from the assessment of experts using HMA. RESULTS Subject baseline metrics were significantly different from expert (p < 0.01). At the outset of session 2, none of the subjects achieved the performance benchmarks. At the outset of sessions 3 and 4, 40 and 60% of subjects, respectively, demonstrated retention of technical proficiency. However, there was evidence of significant skill erosion between sessions (p < 0.01). The mean number of practice attempts required to achieve technical proficiency declined between sessions. Regression analysis indicated that there was a significant training effect for all students (overall negative slopes) over time. When examining the group as a whole, the speed at which students reached the expert benchmark was not significantly associated with number of practices in the first three sessions, although for some individuals these factors did appear associated. A total of 76% of subjects retained technical proficiency by session 4 and required a mean of 14 practices (range = 5 to 19). CONCLUSIONS These results show that the majority of students require three to four sessions of deliberate practice to achieve a sustainable level of proficiency in the LP procedure. There is considerable variation in learning progression and retention of technical proficiency. These results have important implications for the design and resource requirements of a competency-based medical education program targeting LP training.

Surgical motion characterization in simulated needle insertion procedures

PURPOSE: Evaluation of surgical performance in image-guided needle insertions is of emerging interest, to both promote patient safety and improve the efficiency and effectiveness of training. The purpose of this study was to determine if a Markov model-based algorithm can more accurately segment a needle-based surgical procedure into its five constituent tasks than a simple threshold-based algorithm. METHODS: Simulated needle trajectories were generated with known ground truth segmentation by a synthetic procedural data generator, with random noise added to each degree of freedom of motion. The respective learning algorithms were trained, and then tested on different procedures to determine task segmentation accuracy. In the threshold-based algorithm, a change in tasks was detected when the needle crossed a position/velocity threshold. In the Markov model-based algorithm, task segmentation was performed by identifying the sequence of Markov models most likely to have produced the series of observations. RESULTS: For amplitudes of translational noise greater than 0.01mm, the Markov model-based algorithm was significantly more accurate in task segmentation than the threshold-based algorithm (82.3% vs. 49.9%, p<0.001 for amplitude 10.0mm). For amplitudes less than 0.01mm, the two algorithms produced insignificantly different results. CONCLUSION: Task segmentation of simulated needle insertion procedures was improved by using a Markov model-based algorithm as opposed to a threshold-based algorithm for procedures involving translational noise.

Real-time workflow detection using webcam video for providing real-time feedback in central venous catheterization training.

Purpose: Medical schools are shifting from a time-based approach to a competency-based education approach. A competency-based approach requires continuous observation and evaluation of trainees. The goal of Central Line Tutor is to be able to provide instruction and real-time feedback for trainees learning the procedure of central venous catheterization, without requiring a continuous expert observer. The purpose of this study is to test the accuracy of the workflow detection method of Central Line Tutor. This study also looks at the effectiveness of object recognition from a webcam video for workflow detection. Methods: Five trials of the procedure were recorded from Central Line Tutor. Five reviewers were asked to identify the timestamp of the transition points in each recording. Reviewer timestamps were compared to those identified by Central Line Tutor. Differences between these values were used to calculate average transitional delay. Results: Central Line Tutor was able to identify 100% of transition points in the procedure with an average transitional delay of -1.46 ± 0.81s. The average transitional delay of EM and webcam tracked steps were -0.35 ± 2.51s and -2.46 ± 3.57s respectively. Conclusions: Central line tutor was able to detect completion of all workflow tasks with minimal delay and may be used to provide trainees with real-time feedback. The results also show that object recognition from a webcam video is an effective method for detecting workflow tasks in the procedure of central venous catheterization.

Self-guided training for deep brain stimulation planning using objective assessment

ObjectiveDeep brain stimulation (DBS) is an increasingly common treatment for neurodegenerative diseases. Neurosurgeons must have thorough procedural, anatomical, and functional knowledge to plan electrode trajectories and thus ensure treatment efficacy and patient safety. Developing this knowledge requires extensive training. We propose a training approach with objective assessment of neurosurgeon proficiency in DBS planning.MethodsTo assess proficiency, we propose analyzing both the viability of the planned trajectory and the manner in which the operator arrived at the trajectory. To improve understanding, we suggest a self-guided training course for DBS planning using real-time feedback. To validate the proposed measures of proficiency and training course, two experts and six novices followed the training course, and we monitored their proficiency measures throughout.ResultsAt baseline, experts planned higher quality trajectories and did so more efficiently. As novices progressed through the training course, their proficiency measures increased significantly, trending toward expert measures.ConclusionWe developed and validated measures which reliably discriminate proficiency levels. These measures are integrated into a training course, which quantitatively improves trainee performance. The proposed training course can be used to improve trainees’ proficiency, and the quantitative measures allow trainees’ progress to be monitored.

A learning curve analysis of ultrasound-guided in-plane and out-of-plane vascular access training with Perk Tutor.

PURPOSE: Under ultrasound guidance, procedures that have been traditionally performed using landmark approaches have become safer and more efficient. However, inexperienced trainees struggle with coordinating probe handling and needle insertion. We aimed to establish learning curves to identify the rate of acquisition of in-plane and out-of-plane vascular access skill in novice medical trainees. METHODS: Thirty-eight novice participants were randomly assigned to perform either in-plane or out-of-plane insertions. Participants underwent baseline testing, four practice insertions (with 3D visualization assistance), and final testing; performance metrics were computed for all procedures. Five expert participants performed insertions in both approaches to establish expert performance metric benchmarks. RESULTS: In-plane novices (n=19) demonstrated significant final reductions in needle path inefficiency (45.8 vs. 127.1, p<0.05), needle path length (41.1 mm vs. 58.0 mm, p<0.05), probe path length (11.6 mm vs. 43.8 mm, p<0.01), and maximal distance between needle and ultrasound plane (3.1 mm vs. 5.5 mm, p<0.05) and surpassed expert benchmarks in average and maximal rotational error. Out-of-plane novices (n=19) demonstrated significant final reductions in all performance metrics, including needle path inefficiency (54.4 vs. 1102, p<0.01), maximum distance of needle past plane (0.0 mm vs. 7.3 mm, p<0.01), and total time of needle past plane (0.0 s vs. 3.4 s, p<0.01) and surpassed expert benchmarks in maximum distance and time of needle past plane. CONCLUSION: Our learning curves quantify improvement in in-plane and out-of-plane vascular access skill with 3D visualization over multiple attempts. The training session enables more than half of novices to approach expert performance benchmarks.

Neurosurgical burr hole placement using the Microsoft HoloLens

PURPOSE: Tracked navigation systems are generally impractical in bedside neurosurgical procedures, such as a twist-drill crainiostomy for the removal of a subdural hematoma, where the use of navigation could optimize the placement of the drill in relation to the underlying fluid. We use the Microsoft HoloLens to display a hologram floating in the patient’s head to mark a burr hole on the skull. METHODS: A 3D model of the head, hematoma and burr hole is created from CT and imported to the HoloLens. The hologram is interactively registered to the patient and the burr hole is marked on the skull. 3D Slicer, Unity, and Visual Studio were used for software development. The system was tested by 6 inexperienced and 1 experienced users. They each performed 6 registrations on phantoms with fiducial markers placed at 3 plausible burr hole locations on each side of the head. Registration accuracy was determined by measuring the distance between the holographic and physical markers. RESULTS: Inexperienced users placed 98% of the markers within the clinically acceptable range of 10 mm in an average time of 4:46 min. The experienced user placed 100% of the markers within the acceptable range in an average time of 2:52 min. CONCLUSION: It is feasible to mark a neurosurgical burr hole location with clinically acceptable accuracy using the Microsoft HoloLens, within an acceptable length of time. This technology may also prove useful for procedures that require higher accuracy of drill location and drain trajectory such as the placement of external ventricular drains.

Exploration using holographic hands as a modality for skills training in medicine.

PURPOSE: Gaining proficiency in technical skills involving specific hand motions is prevalent across all disciplines of medicine and particularly relevant in learning surgical skills such as knot tying. We propose a new form of self-directed learning where a pair of holographic hands is projected in front of the trainee using the Microsoft HoloLens and guides them through learning various basic hand motions relevant to surgery and medicine. This study looks at the feasibility and effectiveness of using holographic hands as a skills training modality for learning hand motions compared to the traditional methods of apprenticeship and video-based learning. METHODS: 9 participants were recruited and each learned 6 different hand motions from 3 different modalities (video, apprenticeship, HoloLens). Results of successful completion and feedback on effectiveness was obtained through a questionnaire. RESULTS: Participants had a considerable preference for learning from HoloLens and apprenticeship and a higher success rate of learning hand motions compared to video-based learning. Furthermore, learning with holographic hands was shown to be comparable to apprenticeship in terms of both effectiveness and success rate. However, more participants still selected apprenticeship as a preferred learning method compared to HoloLens. CONCLUSION: This initial pilot study shows promising results for using holographic hands as a new effective form of self-directed apprenticeship learning that can be applied to learning a wide variety of skills requiring hand motions in medicine. Work continues toward implementing this technology in knot tying and suture tutoring modules in our undergraduate medical curriculum.