François Chadebecq

Shape-from-Template

We study a problem that we call Shape-from-Template, which is the problem of reconstructing the shape of a deformable surface from a single image and a 3D template. Current methods in the literature address the case of isometric deformations, and relax the isometry constraint to the convex inextensibility constraint, solved using the so-called maximum depth heuristic. We call these methods zeroth-order since they use image point locations (the zeroth-order differential structure) to solve the shape inference problem from a perspective image. We propose a novel class of methods that we call first-order. The key idea is to use both image point locations and their first-order differential structure. The latter can be easily extracted from a warp between the template and the input image. We give a unified problem formulation as a system of PDEs for isometric and conformal surfaces that we solve analytically. This has important consequences. First, it gives the first analytical algorithms to solve this type of reconstruction problems. Second, it gives the first algorithms to solve for the exact constraints. Third, it allows us to study the well-posedness of this type of reconstruction: we establish that isometric surfaces can be reconstructed unambiguously and that conformal surfaces can be reconstructed up to a few discrete ambiguities and a global scale. In the latter case, the candidate solution surfaces are obtained analytically. Experimental results on simulated and real data show that our isometric methods generally perform as well as or outperform state of the art approaches in terms of reconstruction accuracy, while our conformal methods largely outperform all isometric methods for extensible deformations.

Real-time mosaicing of fetoscopic videos using SIFT

Fetoscopic laser photo-coagulation of the placental vascular anastomoses remains the most effective therapy for twin-to-twin transfusion syndrome (TTTS) in monochorionic twin pregnancies. However, to ensure the success of the intervention, complete photo-coagulation of all anastomoses is needed. This is made difficult by the limited field of view of the fetoscopic video guidance, which hinders the surgeons ability to locate all the anastomoses. A potential solution to this problem is to expand the field of view of the placental surface by creating a mosaic from overlapping fetoscopic images. This mosaic can then be used for anastomoses localization and spatial orientation during surgery. However, this requires accurate and fast algorithms that can operate within the real-time constraints of fetal surgery. In this work, we present an image mosaicing framework that leverages the parallelism of modern GPUs and can process clinical fetoscopic images in real-time. Initial qualitative results on ex-vivo placental images indicate that the proposed framework can generate clinically useful mosaics from fetoscopic videos in real-time.

Measuring the size of neoplasia in colonoscopy using depth-from-defocus

Colonoscopy is the reference medical examination for the diagnosis and treatment of neoplasia in gastroenterology. During the examination, the expert explores the colon cavity with a gastroscope in order to detect neoplasias - abnormal growths of tissue - and to diagnose which ones could be malignant. The Paris classification of superficial neoplastic lesions is the gold standard set of criteria for this type of diagnosis. One of the major criteria is the size. However, this is tremendously difficult to accurately estimate from images. This is because the absolute scale of the observed tissues is not directly conveyed in the 2D endoscopic image. We propose an image-based method to estimate the size of neoplasias. The core idea is to combine Depth-From-Focus (DFF) and Depth-From-Defocus (DFD). This allows us to recover the absolute scale by automatically detecting the blur/unblur breakpoint while the expert pulls the gastroscope away from a neoplasia. Our method is passive: it uses the image data only and thus does not require hardware modification of the gastroscope. We report promising experimental results on phantom and patient datasets.

Enhanced imaging colonoscopy facilitates dense motion-based 3D reconstruction

We propose a novel approach for estimating a dense 3D model of neoplasia in colonoscopy using enhanced imaging endoscopy modalities. Estimating a dense 3D model of neoplasia is important to make 3D measurements and to classify the superficial lesions in standard frameworks such as the Paris classification. However, it is challenging to obtain decent dense 3D models using computer vision techniques such as Structure-from-Motion due to the lack of texture in conventional (white light) colonoscopy. Therefore, we propose to use enhanced imaging endoscopy modalities such as Narrow Band Imaging and chromoendoscopy to facilitate the 3D reconstruction process. Thanks to the use of these enhanced endoscopy techniques, visualization is improved, resulting in more reliable feature tracks and 3D reconstruction results. We first build a sparse 3D model of neoplasia using Structure-from-Motion from enhanced endoscopy imagery. Then, the sparse reconstruction is densified using a Multi-View Stereo approach, and finally the dense 3D point cloud is transformed into a mesh by means of Poisson surface reconstruction. The obtained dense 3D models facilitate classification of neoplasia in the Paris classification, in which the 3D size and the shape of the neoplasia play a major role in the diagnosis.

Using the Infocus-Breakpoint to estimate the scale of neoplasia in colonoscopy

Colonoscopy is the reference medical examination for early diagnosis and treatment of colonic diseases. This minimally invasive technique allows gastroenterologists to explore the colon and remove neoplasias - abnormal growth of tissue - such as polyps which may transform into malignant tumors. Shape, texture and size of polyps are of particular interest for determining their nature. However, the size of neoplasias is difficult to estimate because the absolute scale of the observed tissue is not directly conveyed in the 2D endoscopic images. We here improve our Infocus-Breakpoint (IB) technique, which estimates an imagewise scale by detecting the blur/ unblur breakpoint in a video sequence. We simultaneously track a neoplasia with a 2D affine transformation and estimate the amount of defocus blur. This leads to an areawise scale estimate with better accuracy.

A Continuum Robot and Control Interface for Surgical Assist in Fetoscopic Interventions

Twin–twin transfusion syndrome requires interventional treatment using a fetoscopically introduced laser to sever the shared blood supply between the fetuses. This is a delicate procedure relying on small instrumentation with limited articulation to guide the laser tip and a narrow field of view to visualize all relevant vascular connections. In this letter, we report on a mechatronic design for a comanipulated instrument that combines concentric tube actuation to a larger manipulator constrained by a remote centre of motion. A stereoscopic camera is mounted at the distal tip and used for imaging. Our mechanism provides enhanced dexterity and stability of the imaging device. We demonstrate that the imaging system can be used for computing geometry and enhancing the view at the operating site. Results using electromagnetic sensors for verification and comparison to visual odometry from the distal sensor show that our system is promising and can be developed further for multiple clinical needs in fetoscopic procedures.

A Combined EM and Visual Tracking Probabilistic Model for Robust Mosaicking: Application to Fetoscopy

Twin-to-Twin Transfusion Syndrome (TTTS) is a progressive pregnancy complication in which inter-twin vascular connections in the shared placenta result in a blood flow imbalance between the twins. The most effective therapy is to sever these connections by laser photo-coagulation. However, the limited field of view of the fetoscope hinders their identification. A potential solution is to augment the surgeons view by creating a mosaic image of the placenta. State-of-the-art mosaicking methods use feature-based approaches, which have three main limitations: (i) they are not robust against corrupt data e.g. blurred frames, (ii) temporal information is not used, (iii) the resulting mosaic suffers from drift. We introduce a probabilistic temporal model that incorporates electromagnetic and visual tracking data to achieve a robust mosaic with reduced drift. By assuming planarity of the imaged object, the nRT decomposition can be used to parametrize the state vector. Finally, we tackle the non-linear nature of the problem in a numerically stable manner by using the Square Root Unscented Kalman Filter. We show an improvement in performance in terms of robustness as well as a reduction of the drift in comparison to state-of-the-art methods in synthetic, phantom and ex vivo datasets.

How big is this neoplasia? Live colonoscopic size measurement using the Infocus-Breakpoint

Colonoscopy is the reference medical examination for early diagnosis and treatment of colonic diseases. This minimally invasive technique allows endoscopists to explore the colon cavity and remove neoplasias - abnormal growths of tissue - which may develop into malignant tumors. The size, shape and appearance of a neoplasia are essential cues for diagnostic. However, the size is difficult to estimate because the absolute scale of the observed tissue is not directly conveyed in the 2D colonoscopic images. An erroneous size estimate may lead to inappropriate treatment. There currently exist no solutions to reproducible neoplasia size measurement adapted to colonoscopy. We propose a colonoscopic size measurement system for neoplasias. By using a simple planar geometry, the key technical problem is reduced to resolving scale. Our core contribution is introducing the Infocus-Breakpoint (IB) that allows us to resolve scale from a regular colonoscopic video. We define the IB as the lower limit of the colonoscopes depth of field. The IB corresponds to a precise colonoscope to tissue distance, called the reference depth, which we calibrate preoperatively. We detect the IB intraoperatively thanks to two novel modules: deformable Blur-Estimating Tracking (BET) and Blur-Model Fitting (BMF). With our system, the endoscopist may interactively measure the length and area of a neoplasia in a 2D colonoscopic image directly. Our system needs no hardware modification to standard monocular colonoscopes, yet reaching a size measurement accuracy of the order of a millimeter, as shown on several phantom and patient datasets.

Adjoint Transformation Algorithm for Hand–Eye Calibration with Applications in Robotic Assisted Surgery

Hand–eye calibration aims at determining the unknown rigid transformation between the coordinate systems of a robot arm and a camera. Existing hand–eye algorithms using closed-form solutions followed by iterative non-linear refinement provide accurate calibration results within a broad range of robotic applications. However, in the context of surgical robotics hand–eye calibration is still a challenging problem due to the required accuracy within the millimetre range, coupled with a large displacement between endoscopic cameras and the robot end-effector. This paper presents a new method for hand–eye calibration based on the adjoint transformation of twist motions that solves the problem iteratively through alternating estimations of rotation and translation. We show that this approach converges to a solution with a higher accuracy than closed form initializations within a broad range of synthetic and real experiments. We also propose a stereo hand–eye formulation that can be used in the context of both our proposed method and previous state-of-the-art closed form solutions. Experiments with real data are conducted with a stereo laparoscope, the KUKA robot arm manipulator, and the da Vinci surgical robot, showing that both our new alternating solution and the explicit representation of stereo camera hand–eye relations contribute to a higher calibration accuracy.

Medical-grade Sterilizable Target for Fluid-immersed Fetoscope Optical Distortion Calibration

We have developed a calibration target for use with fluid-immersed endoscopes within the context of the GIFT-Surg (Guided Instrumentation for Fetal Therapy and Surgery) project. One of the aims of this project is to engineer novel, real-time image processing methods for intra-operative use in the treatment of congenital birth defects, such as spina bifida and the twin-to-twin transfusion syndrome. The developed target allows for the sterility-preserving optical distortion calibration of endoscopes within a few minutes. Good optical distortion calibration and compensation are important for mitigating undesirable effects like radial distortions, which not only hamper accurate imaging using existing endoscopic technology during fetal surgery, but also make acquired images less suitable for potentially very useful image computing applications, like real-time mosaicing. In this paper proposes a novel fabrication method to create an affordable, sterilizable calibration target suitable for use in a clinical setup. This method involves etching a calibration pattern by laser cutting a sandblasted stainless steel sheet. This target was validated using the camera calibration module provided by OpenCV, a state-of-the-art software library popular in the computer vision community.