Rogério Richa

Three-dimensional Motion Tracking for Beating Heart Surgery Using a Thin-plate Spline Deformable Model

Minimally invasive cardiac surgery offers important benefits for the patient but it also imposes several challenges for the surgeon. Robotic assistance has been proposed to overcome many of the difficulties inherent to the minimally invasive procedure, but so far no solutions for compensating physiological motion are present in the existing surgical robotic platforms. In beating heart surgery, cardiac and respiratory motions are important sources of disturbance, hindering the surgeon’s gestures and limiting the types of procedures that can be performed in a minimally invasive fashion. In this context, computer vision techniques can be used for retrieving the heart motion for active motion stabilization, which improves the precision and repeatability of the surgical gestures. However, efficient tracking of the heart surface is a challenging problem due to the heart surface characteristics, large deformations and the complex illumination conditions. In this article, we present an efficient method for active cancellation of cardiac motion where we combine an efficient algorithm for 3D tracking of the heart surface based on a thin-plate spline deformable model and an illumination compensation algorithm able to cope with arbitrary illumination changes. The proposed method has two novelties: the thin-plate spline model for representing the heart surface deformations and an efficient parametrization for 3D tracking of the beating heart using stereo images from a calibrated stereo endoscope. The proposed tracking method has been evaluated offline on in vivo images acquired by a DaVinci surgical robotic platform.

Towards robust 3D visual tracking for motion compensation in beating heart surgery

In the context of minimally invasive cardiac surgery, active vision-based motion compensation schemes have been proposed for mitigating problems related to physiological motion. However, robust and accurate visual tracking remains a difficult task. The purpose of this paper is to present a robust visual tracking method that estimates the 3D temporal and spatial deformation of the heart surface using stereo endoscopic images. The novelty is the combination of a visual tracking method based on a Thin-Plate Spline (TPS) model for representing the heart surface deformations with a temporal heart motion model based on a time-varying dual Fourier series for overcoming tracking disturbances or failures. The considerable improvements in tracking robustness facing specular reflections and occlusions are demonstrated through experiments using images of in vivo porcine and human beating hearts.

Visual tracking using the sum of conditional variance

The goal of this paper is to introduce a direct visual tracking method based on an image similarity measure called the sum of conditional variance (SCV). The SCV was originally proposed in the medical imaging domain for registering multi-modal images. In the context of visual tracking, the SCV is invariant to non-linear illumination variations, multi-modal and computationally inexpensive. Compared to information theoretic tracking methods, it requires less iterations to converge and has a significantly larger convergence radius. The novelty in this paper is a generalization of the efficient second-order minimization formulation for tracking using the SCV, allowing us to combine the efficient second-order approximation of the Hessian with a similarity metric invariant to non-linear illumination variations. The result is a visual tracking method that copes with non-linear illumination variations without requiring the estimation of photometric correction parameters at every iteration. We demonstrate the superior performance of the proposed method through comparative studies and tracking experiments under challenging illumination conditions and rapid motions.

Beating heart motion prediction for robust visual tracking

In the context of minimally invasive cardiac surgery, robotic assistance has significantly helped surgeons to overcome difficulties related to the minimally invasive procedure. Recently, techniques have been proposed for active canceling the beating heart motion for improving the accuracy of the surgical gestures. In this scenario, computer vision techniques can be applied for estimating the heart motion based solely on natural structures on the heart surface. However, visual tracking is complicated by the particular lighting conditions and clutter (smoke, liquids, etc) during surgery. Another challenging problem are the occasional occlusions by surgical instruments. In order to overcome these problems, we exploit the quasi-periodicity of the beating heart motion for increasing the robustness of the visual tracking task. In this paper, a novel time-varying dual Fourier series for modeling the quasi-periodic beating heart motion is proposed. For estimating the series parameters, an Extended Kalman Filter (EKF) is used. The proposed method is applied in a visual tracking task for bridging tracking disturbances and automatically reestablish tracking in cases of occlusions. The efficiency of the prediction method and the sensible improvements in the visual tracking task are demonstrated through in vivo experiments.

Unified Detection and Tracking of Instruments during Retinal Microsurgery

Methods for tracking an object have generally fallen into two groups: tracking by detection and tracking through local optimization. The advantage of detection-based tracking is its ability to deal with target appearance and disappearance, but it does not naturally take advantage of target motion continuity during detection. The advantage of local optimization is efficiency and accuracy, but it requires additional algorithms to initialize tracking when the target is lost. To bridge these two approaches, we propose a framework for unified detection and tracking as a time-series Bayesian estimation problem. The basis of our approach is to treat both detection and tracking as a sequential entropy minimization problem, where the goal is to determine the parameters describing a target in each frame. To do this we integrate the Active Testing (AT) paradigm with Bayesian filtering, and this results in a framework capable of both detecting and tracking robustly in situations where the target object enters and leaves the field of view regularly. We demonstrate our approach on a retinal tool tracking problem and show through extensive experiments that our method provides an efficient and robust tracking solution.

Data-Driven visual tracking in retinal microsurgery

In the context of retinal microsurgery, visual tracking of instruments is a key component of robotics assistance. The difficulty of the task and major reason why most existing strategies fail on in-vivo image sequences lies in the fact that complex and severe changes in instrument appearance are challenging to model. This paper introduces a novel approach, that is both data-driven and complementary to existing tracking techniques. In particular, we show how to learn and integrate an accurate detector with a simple gradient-based tracker within a robust pipeline which runs at framerate. In addition, we present a fully annotated dataset of retinal instruments in in-vivo surgeries, which we use to quantitatively validate our approach. We also demonstrate an application of our method in a laparascopy image sequence.

Visual tracking of surgical tools for proximity detection in retinal surgery

In retinal surgery, surgeons face difficulties such as indirect visualization of surgical targets, physiological tremor and lack of tactile feedback. Such difficulties increase the risks of incorrect surgical gestures which may cause retinal damage. In this context, robotic assistance has the potential to overcome current technical limitations and increase surgical safety. In this paper we present a method for robustly tracking surgical tools in retinal surgery for detecting proximity between surgical tools and the retinal surface. An image similarity function based on weighted mutual information is specially tailored for tracking under critical illumination variations, lens distortions, and rapid motion. The proposed method was tested on challenging conditions using a phantom eye and recorded human in vivo data acquired by an ophthalmic stereo microscope.

Robust 3d visual tracking for robotic-assisted cardiac interventions

In the context of minimally invasive cardiac surgery, active vision-based motion compensation schemes have been proposed for mitigating problems related to physiological motion. However, robust and accurate visual tracking is a difficult task. The purpose of this paper is to present a hybrid tracker that estimates the heart surface deformation using the outputs of multiple visual tracking techniques. In the proposed method, the failure of an individual technique can be circumvented by the success of others, enabling the robust estimation of the heart surface deformation with increased spatial resolution. In addition, for coping with the absence of visual information due to motion blur or occlusions, a temporal heart motion model is incorporated as an additional support for the visual tracking task. The superior performance of the proposed technique compared to existing techniques individually is demonstrated through experiments conducted on recorded images of an in vivo minimally invasive CABG using the DaVinci robotic platform.

Improving NCC-based direct visual tracking

Direct visual tracking can be impaired by changes in illumination if the right choice of similarity function and photometric model is not made. Tracking using the sum of squared differences, for instance, often needs to be coupled with a photometric model to mitigate illumination changes. More sophisticated similarities, e.g. mutual information and cross cumulative residual entropy, however, can cope with complex illumination variations at the cost of a reduction of the convergence radius, and an increase of the computational effort. In this context, the normalized cross correlation (NCC) represents an interesting alternative. The NCC is intrinsically invariant to affine illumination changes, and also presents low computational cost. This article proposes a new direct visual tracking method based on the NCC. Two techniques have been developed to improve the robustness to complex illumination variations and partial occlusions. These techniques are based on subregion clusterization, and weighting by a residue invariant to affine illumination changes. The last contribution is an efficient Newton-style optimization procedure that does not require the explicit computation of the Hessian. The proposed method is compared against the state of the art using a benchmark database with ground-truth, as well as real-world sequences.

Efficient 3D Tracking for Motion Compensation in Beating Heart Surgery

The design of physiological motion compensation systems for robotic-assisted cardiac Minimally Invasive Surgery (MIS) is a challenging research topic. In this domain, vision-based techniques have proven to be a practical way to retrieve the motion of the beating heart. However due to the complexity of the heart motion and its surface characteristics, efficient tracking is still a complicated task. In this paper, we propose an algorithm for tracking the 3D motion of the beating heart, based on a Thin-Plate Splines (TPS) parametric model. The novelty of our approach lies in that no explicit matching between the stereo camera images is required and consequently no intermediate steps such as rectification are needed. Experiments conducted on ex-vivo and in-vivo tissue show the effectiveness of the proposed algorithm for tracking surfaces undergoing complex deformations.