Animesh Garg

Learning by observation for surgical subtasks: Multilateral cutting of 3D viscoelastic and 2D Orthotropic Tissue Phantoms

Automating repetitive surgical subtasks such as suturing, cutting and debridement can reduce surgeon fatigue and procedure times and facilitate supervised tele-surgery. Programming is difficult because human tissue is deformable and highly specular. Using the da Vinci Research Kit (DVRK) robotic surgical assistant, we explore a “Learning By Observation” (LBO) approach where we identify, segment, and parameterize motion sequences and sensor conditions to build a finite state machine (FSM) for each subtask. The robot then executes the FSM repeatedly to tune parameters and if necessary update the FSM structure. We evaluate the approach on two surgical subtasks: debridement of 3D Viscoelastic Tissue Phantoms (3d-DVTP), in which small target fragments are removed from a 3D viscoelastic tissue phantom; and Pattern Cutting of 2D Orthotropic Tissue Phantoms (2d-PCOTP), a step in the standard Fundamentals of Laparoscopic Surgery training suite, in which a specified circular area must be cut from a sheet of orthotropic tissue phantom. We describe the approach and physical experiments with repeatability of 96% for 50 trials of the 3d-DVTP subtask and 70% for 20 trials of the 2d-PCOTP subtask. A video is available at: http://j.mp/Robot-Surgery-Video-Oct-2014.

Transition State Clustering: Unsupervised Surgical Trajectory Segmentation for Robot Learning

A large and growing corpus of synchronized kinematic and video recordings of robot-assisted surgery has the potential to facilitate training and subtask automation. One of the challenges in segmenting such multi-modal trajectories is that demonstrations vary spatially, temporally, and contain random noise and loops (repetition until achieving the desired result). Segments of task trajectories are often less complex, less variable, and allow for easier detection of outliers. As manual segmentation can be tedious and error-prone, we propose a new segmentation method that combines hybrid dynamical systems theory and Bayesian non-parametric statistics to automatically segment demonstrations. Transition State Clustering (TSC) models demonstrations as noisy realizations of a switched linear dynamical system, and learns spatially and temporally consistent transition events across demonstrations. TSC uses a hierarchical Dirichlet Process Gaussian Mixture Model to avoid having to select the number of segments a priori. After a series of merging and pruning steps, the algorithm adaptively optimizes the number of segments. In a synthetic case study with two linear dynamical regimes, where demonstrations are corrupted with noise and temporal variations, TSC finds up to a 20% more accurate segmentation than GMM-based alternatives. On 67 recordings of surgical needle passing and suturing tasks from the JIGSAWS surgical training dataset [7], supplemented with manually annotated visual features, TSC finds 83% of needle passing segments and 73% of the suturing segments found by human experts. Qualitatively, TSC also identifies transitions overlooked by human annotators.

Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization

For supervised automation of multi-throw suturing in Robot-Assisted Minimally Invasive Surgery, we present a novel mechanical needle guide and a framework for optimizing needle size, trajectory, and control parameters using sequential convex programming. The Suture Needle Angular Positioner (SNAP) results in a 3x error reduction in the needle pose estimate in comparison with the standard actuator. We evaluate the algorithm and SNAP on a da Vinci Research Kit using tissue phantoms and compare completion time with that of humans from the JIGSAWS dataset [5]. Initial results suggest that the dVRK can perform suturing at 30% of human speed while completing 86% suture throws attempted. Videos and data are available at: berkeleyautomation.github.io/amts.

Evaluation of PC-ISO for Customized, 3D Printed, Gynecologic 192-Ir HDR Brachytherapy Applicators

The purpose of this study was to evaluate the radiation attenuation properties of PC‐ISO, a commercially available, biocompatible, sterilizable 3D printing material, and its suitability for customized, single‐use gynecologic (GYN) brachytherapy applicators that have the potential for accurate guiding of seeds through linear and curved internal channels. A custom radiochromic film dosimetry apparatus was 3D‐printed in PC‐ISO with a single catheter channel and a slit to hold a film segment. The apparatus was designed specifically to test geometry pertinent for use of this material in a clinical setting. A brachytherapy dose plan was computed to deliver a cylindrical dose distribution to the film. The dose plan used an 192Ir source and was normalized to 1500 cGy at 1 cm from the channel. The material was evaluated by comparing the film exposure to an identical test done in water. The Hounsfield unit (HU) distributions were computed from a CT scan of the apparatus and compared to the HU distribution of water and the HU distribution of a commercial GYN cylinder applicator. The dose depth curve of PC‐ISO as measured by the radiochromic film was within 1% of water between 1 cm and 6 cm from the channel. The mean HU was ‐10 for PC‐ISO and ‐1 for water. As expected, the honeycombed structure of the PC‐ISO 3D printing process created a moderate spread of HU values, but the mean was comparable to water. PC‐ISO is sufficiently water‐equivalent to be compatible with our HDR brachytherapy planning system and clinical workflow and, therefore, it is suitable for creating custom GYN brachytherapy applicators. Our current clinical practice includes the use of custom GYN applicators made of commercially available PC‐ISO when doing so can improve the patients treatment. PACS number: none

Tumor localization using automated palpation with Gaussian Process Adaptive Sampling

In surgical tumor removal, inaccurate localization can lead to removal of excessive healthy tissue and failure to completely remove cancerous tissue. Automated palpation with a tactile sensor has the potential to precisely estimate the geometry of embedded tumors during robot-assisted minimally invasive surgery (RMIS). We formulate tumor boundary localization as a Bayesian optimization model along implicit curves over estimated tissue stiffness. We propose a Gaussian Process Adaptive Sampling algorithm called Implicit Level Set Upper Confidence Bound (ILS-UCB), that prioritizes sampling near a level set of the estimate. We compare the ILS-UCB algorithm to two alternative palpation algorithms: (1) Expected Variance Reduction (EVR), which emphasizes exploration by minimizing variance, and (2) Upper Confidence Bound (UCB), which balances exploration with exploitation using only the estimated mean. We compare these algorithms in simulated experiments varying the levels of measurement noise and bias. We find that ILS-UCB significantly outperforms the other two algorithms as measured by the symmetric difference between tumor boundary estimate and ground truth, reducing error by up to 10×. Physical experiments on a dVRK show that our approach can localize the tumor boundary with approximately the same accuracy as a dense raster scan while requiring at least 10× fewer measurements.

A single-use haptic palpation probe for locating subcutaneous blood vessels in robot-assisted minimally invasive surgery

We present the design and evaluation of a novel low-cost palpation probe for Robot assisted Minimally Invasive Surgery (RMIS) for localizing subcutaneous blood vessels. It measures probe tip deflection using a Hall Effect sensor as the spherical tip is moved tangentially across a surface under automated control. The probe is intended to be single-use and disposable, built from 3D printed parts and commercially available electronics. The prototype has a cross-section of less than 15mm×10mm and fits on the end of an 8mm diameter needle driver in the Intuitive Surgical da Vinci® Research Kit (dVRK). We report experiments for quasi-static sliding palpation with silicone based tissue phantoms with embedded cylinders as subcutaneous blood vessel phantoms. We analyzed signal-to-noise ratios with multiple diameters of silicone cylinders (1.58-4.75 mm) at varying subcutaneous depths (1-5 mm) with a range of indentation depths (0-8 mm) and sliding speeds (0.5-21 mm/s). Results suggest that the probe can detect subcutaneous structures in phantoms of diameter 2.25 mm at a depth of up to 5mm below the tissue surface.

An algorithm for computing customized 3D printed implants with curvature constrained channels for enhancing intracavitary brachytherapy radiation delivery

Brachytherapy is a widely-used treatment modality for cancer in many sites in the body. In brachytherapy, small radioactive sources are positioned proximal to cancerous tumors. An ongoing challenge is to accurately place sources on a set of dwell positions to sufficiently irradiate the tumors while limiting radiation damage to healthy organs and tissues. In current practice, standardized applicators with internal channels are inserted into body cavities to guide the sources. These standardized implants are one-size-fits-all and are prone to shifting inside the body, resulting in suboptimal dosages. We propose a new approach that builds on recent results in 3D printing and steerable needle motion planning to create customized implants containing customized curvature-constrained internal channels that fit securely, minimize air gaps, and precisely guide radioactive sources through printed channels. When compared with standardized implants, customized implants also have the potential to provide better coverage: more potential source dwell positions proximal to tumors. We present an algorithm for computing curvature-constrained channels based on rapidly-expanding randomized trees (RRT). We consider a prototypical case of OB/GYN cervical and vaginal cancer with three treatment options: standardized ring implant (current practice), customized implant with linear channels, and customized implant with curved channels. Results with a two-parameter coverage metric suggest that customized implants with curved channels can offer significant improvement over current practice.

TSC-DL: Unsupervised trajectory segmentation of multi-modal surgical demonstrations with Deep Learning

The growth of robot-assisted minimally invasive surgery has led to sizable datasets of fixed-camera video and kinematic recordings of surgical subtasks. Segmentation of these trajectories into locally-similar contiguous sections can facilitate learning from demonstrations, skill assessment, and salvaging good segments from otherwise inconsistent demonstrations. Manual, or supervised, segmentation can be prone to error and impractical for large datasets. We present Transition State Clustering with Deep Learning (TSC-DL), a new unsupervised algorithm that leverages video and kinematic data for task-level segmentation, and finds regions of the visual feature space that correlate with transition events using features constructed from layers of pre-trained image classification Deep Convolutional Neural Networks (CNNs). We report results on three datasets comparing Deep Learning architectures (AlexNet and VGG), choice of convolutional layer, dimensionality reduction techniques, visual encoding, and the use of Scale Invariant Feature Transforms (SIFT). We find that the deep architectures extract features that result in up-to a 30.4% improvement in Silhouette Score (a measure of cluster tightness) over the traditional “shallow” features from SIFT. We also present cases where TSC-DL discovers human annotator omissions. Supplementary material, data and code is available at: http://berkeleyautomation.github.io/tsc-dl/.

Robot-Guided Open-Loop Insertion of Skew-Line Needle Arrangements for High Dose Rate Brachytherapy

We present a study in human-centered automation that has potential to reduce patient side effects from high dose rate brachytherapy (HDR-BT). To efficiently deliver radiation to the prostate while minimizing trauma to sensitive structures such as the penile bulb, we modified the Acubot-RND 7-axis robot to guide insertion of diamond-tip needles into desired skew-line geometric arrangements. We extend and integrate two algorithms: Needle Planning with Integer Programming (NPIP) and Inverse Planning with Integer Programming (IPIP) to compute skew-line needle and dose plans. We performed three physical experiments with anatomically correct phantom models to study performance: two with the robot and one control experiment with an expert human physician (coauthor Hsu) without the robot. All were able to achieve needle arrangements that meet the RTOG-0321 clinical dose objectives with zero trauma to the penile bulb. We analyze systematic and random errors in needle placement; total RMS error for the robot system operating without feedback ranged from 2.6 to 4.3 mm, which is comparable to the RMS error of 2.7 mm obtained in an earlier study for PPI-BT treatment using a robot with 3D ultrasound feedback.

Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning

In the Fundamentals of Laparoscopic Surgery (FLS) standard medical training regimen, the Pattern Cutting task requires residents to demonstrate proficiency by maneuvering two tools, surgical scissors and tissue gripper, to accurately cut a circular pattern on surgical gauze suspended at the corners. Accuracy of cutting depends on tensioning, wherein the gripper pinches a point on the gauze in R3 and pulls to induce and maintain tension in the material as cutting proceeds. An automated tensioning policy maps the current state of the gauze to output a direction of pulling as an action. The optimal tensioning policy depends on both the choice of pinch point and cutting trajectory. We explore the problem of learning a tensioning policy conditioned on specific cutting trajectories. Every timestep, we allow the gripper to react to the deformation of the gauze and progress of the cutting trajectory with a translation unit vector along an allowable set of directions. As deformation is difficult to analytically model and explicitly observe, we leverage deep reinforcement learning with direct policy search methods to learn tensioning policies using a finite-element simulator and then transfer them to a physical system. We compare the Deep RL tensioning policies with fixed and analytic (opposing the error vector with a fixed pinch point) policies on a set of 17 open and closed curved contours in simulation and 4 patterns in physical experiments with the da Vinci Research Kit (dVRK). Our simulation results suggest that learning to tension with Deep RL can significantly improve performance and robustness to noise and external forces.