Carol E. Reiley

Augmented Reality During Robot-assisted Laparoscopic Partial Nephrectomy: Toward Real-Time 3D-CT to Stereoscopic Video Registration

OBJECTIVES To investigate a markerless tracking system for real-time stereo-endoscopic visualization of preoperative computed tomographic imaging as an augmented display during robot-assisted laparoscopic partial nephrectomy. METHODS Stereoscopic video segments of a patient undergoing robot-assisted laparoscopic partial nephrectomy for tumor and another for a partial staghorn renal calculus were processed to evaluate the performance of a three-dimensional (3D)-to-3D registration algorithm. After both cases, we registered a segment of the video recording to the corresponding preoperative 3D-computed tomography image. After calibrating the camera and overlay, 3D-to-3D registration was created between the model and the surgical recording using a modified iterative closest point technique. Image-based tracking technology tracked selected fixed points on the kidney surface to augment the image-to-model registration. RESULTS Our investigation has demonstrated that we can identify and track the kidney surface in real time when applied to intraoperative video recordings and overlay the 3D models of the kidney, tumor (or stone), and collecting system semitransparently. Using a basic computer research platform, we achieved an update rate of 10 Hz and an overlay latency of 4 frames. The accuracy of the 3D registration was 1 mm. CONCLUSIONS Augmented reality overlay of reconstructed 3D-computed tomography images onto real-time stereo video footage is possible using iterative closest point and image-based surface tracking technology that does not use external navigation tracking systems or preplaced surface markers. Additional studies are needed to assess the precision and to achieve fully automated registration and display for intraoperative use.

Effects of visual force feedback on robot-assisted surgical task performance

OBJECTIVE Direct haptic (force or tactile) feedback is negligible in current surgical robotic systems. The relevance of haptic feedback in robot-assisted performances of surgical tasks is controversial. We studied the effects of visual force feedback, a haptic feedback surrogate, on tying surgical knots with fine sutures similar to those used in cardiovascular surgery. METHODS By using a modified da Vinci robotic system (Intuitive Surgical, Inc, Sunnyvale, Calif) equipped with force-sensing instrument tips and real-time visual force feedback overlays in the console image, 10 surgeons each tied 10 knots with and 10 knots without visual force feedback. Four surgeons had significant prior da Vinci experience, and the remaining 6 surgeons did not. Performance parameters, including suture breakage and secure knots, peak and standard deviation of applied forces, and completion times using 5-0 silk sutures, were recorded. Chi-square and Student t test analyses determined the differences between groups. RESULTS Among surgeon subjects with robotic experience, no differences in measured performance parameters were found between robot-assisted knot ties executed with and without visual force feedback. Among surgeons without robotic experience, however, visual force feedback was associated with lower suture breakage rates, peak applied forces, and standard deviations of applied forces. Visual force feedback did not impart differences in knot completion times or loose knots for either surgeon group. CONCLUSIONS Visual force feedback resulted in reduced suture breakage, lower forces, and decreased force inconsistencies among novice robotic surgeons, although elapsed time and knot quality were unaffected. In contrast, visual force feedback did not affect these metrics among surgeons experienced with the da Vinci system. These results suggest that visual force feedback primarily benefits novice robot-assisted surgeons, with diminishing benefits among experienced surgeons.

Review of methods for objective surgical skill evaluation

BackgroundRising health and financial costs associated with iatrogenic errors have drawn increasing attention to the dexterity of surgeons. With the advent of new technologies, such as robotic surgical systems and medical simulators, researchers now have the tools to analyze surgical motion with the goal of differentiating the level of technical skill in surgeons.MethodsThe review for this paper is obtained from a Google Scholar and PubMed search of the key words “objective surgical skill evaluation.” Only studies that included motion analysis were used.ResultsIn this paper, we provide a clinical motivation for the importance of surgical skill evaluation. We review the current methods of tracking surgical motion and the available data-collection systems. We also survey current methods of surgical skill evaluation and show that most approaches fall into one of three methods: (1) structured human grading; (2) descriptive statistics; or (3) statistical language models of surgical motion. We discuss the need for an encompassing approach to model human skill through statistical models to allow for objective skill evaluation.

Task versus Subtask Surgical Skill Evaluation of Robotic Minimally Invasive Surgery

Evaluating surgical skill is a time consuming, subjective, and difficult process. This paper compares two methods of identifying the skill level of a subject given motion data from a benchtop surgical task. In the first method, we build discrete Hidden Markov Models at the task level, and test against these models. In the second method, we build discrete Hidden Markov Models of surgical gestures, called surgemes, and evaluate skill at this level. We apply these techniques to 57 data sets collected from the da Vinci surgical system. Our current techniques have achieved accuracy levels of 100% using task level models and known gesture segmentation, 95% with task level models and unknown gesture segmentation, and 100% with the surgeme level models in correctly identifying the skill level. We observe that, although less accurate, the second method requires less prior label information. Also, the surgeme level classification provided more insights into what subjects did well, and what they did poorly.

Dynamic augmented reality for sensory substitution in robot-assisted surgical systems.

Teleoperated robot-assisted surgical systems provide surgeons with improved precision, dexterity, and visualization over traditional minimally invasive surgery. The addition of haptic (force and/or tactile) feedback has been proposed as a way to further enhance the performance of these systems. However, due to limitations in sensing and control technologies, implementing direct haptic feedback to the surgeons hands remains impractical for clinical application. A new, intuitive augmented reality system for presentation of force information through sensory substitution has been developed and evaluated. The augmented reality system consists of force-sensing robotic instruments, a kinematic tool tracker, and a graphic display that overlays a visual representation of force levels on top of the moving instrument tips. The system is integrated with the da Vinci Surgical System (Intuitive Surgical, Inc.) and tested by several users in a phantom knot tying task. The augmented reality system decreases the number of broken sutures, decreases the number of loose knots, and results in more consistent application of forces

Haptics for Robot-Assisted Minimally Invasive Surgery

Robot-assisted minimally invasive surgery (RMIS) holds great promise for improving the accuracy and dexterity of a surgeon while minimizing trauma to the patient. However, widespread clinical success with RMIS has been marginal and it is hypothesized by engineers and surgeons alike that the lack of haptic feedback presented to the surgeon is a limiting factor. The objective of our research is to acquire, display, and determine the utility of haptic information during RMIS. This overview paper examines the design, analysis, practicality, and effectiveness of various force estimation and display methods. In particular, we describe our experience in adding force feedback to an experimental version of the da Vinci surgical system, a commercially available teleoperated RMIS system.

One-on-One Spotlight with the RAS President [Student's Corner]

I asked our new IEEE Robotics & Automation Society (RAS) President Prof. Kazuhiro Kosuge a few questions so that we all can get to know him better. Prof. Kosuge was elected by the Administrative Committee (AdCom) to serve as president-elect under Prof. Bruno Siciliano, who was the president in 2008-2009, and succeeded Prof. Siciliano as president on 1 January 2010. Here are his thoughts on the students and Society.