Abed Malti

Monocular Template-Based 3D Reconstruction of Extensible Surfaces with Local Linear Elasticity

We propose a new approach for template-based extensible surface reconstruction from a single view. We extend the method of isometric surface reconstruction and more recent work on conformal surface reconstruction. Our approach relies on the minimization of a proposed stretching energy formalized with respect to the Poisson ratio parameter of the surface. We derive a patch-based formulation of this stretching energy by assuming local linear elasticity. This formulation unifies geometrical and mechanical constraints in a single energy term. We prevent local scale ambiguities by imposing a set of fixed boundary 3D points. We experimentally prove the sufficiency of this set of boundary points and demonstrate the effectiveness of our approach on different developable and non-developable surfaces with a wide range of extensibility.

Combining Conformal Deformation and Cook–Torrance Shading for 3-D Reconstruction in Laparoscopy

We propose a new monocular 3-D reconstruction method adapted for reconstructing organs in the abdominal cavity. It combines both motion and shading cues. The former uses a conformal deformation prior and the latter the Cook-Torrance reflectance model. Our method runs in two phases: first, a 3-D geometric and photometric template of the organ at rest is reconstructed in vivo. The geometric shape is reconstructed using rigid shape-from-motion while the surgeon is exploring-but not deforming-structures in the abdominal cavity. This geometric template is then used to retrieve the photometric properties. A nonparametric model of the lights direction of the laparoscope and the Cook-Torrance reflectance model of the organs tissue are estimated. Second, the surgeon manipulates and deforms the environment. Here, the 3-D template is conformally deformed to globally match a set of few correspondences between the 2-D image data provided by the monocular laparoscope and the 3-D template. Then, the coarse 3-D shape is refined using shading cues to obtain a final 3-D deformed shape. This second phase only relies on a single image. Therefore, it copes with both sequential processing and self-recovery from tracking failure. The proposed approach has been validated using 1) ex vivo and in vivo data with ground-truth, and 2) in vivo laparoscopic videos of a patients uterus. Our experimental results illustrate the ability of our method to reconstruct natural 3-D deformations typical in real surgical procedures.

Hand-eye calibration with epipolar constraints: Application to endoscopy

In this paper, we propose a new calibration method for a hand-eye system equipped with a camera undergoing radial distortion as a rigid endoscope. While classic methods propose either a separated estimation of the camera intrinsics and the hand-eye transform or a mixed non-linear estimation of both hand-eye and camera intrinsics assuming a pin-hole model, the proposed approach enables a simultaneous refinement of the hand-eye and the camera parameters including the distortion factor with only three frames of the calibrated pattern. We propose and compare two criteria to minimize: (i) the first one is based on the epipolar constraint between pairs of frames and (ii) the second one expresses the camera extrinsic with respect to hand-eye and world-grid transforms in the single frame reprojection error. We run bundle-adjustment on each criterion with respect to the distortion parameters, the camera intrinsics and the hand-eye transform. Our method allows us to recover more accurate 3D shapes when compared to state-of-the-art non-linear methods.

Template-Based conformal shape-from-motion-and-shading for laparoscopy

Shape-from-Shading (SfS) is one of the fundamental techniques to recover depth from a single view. Such a method has shown encouraging but limited results in laparoscopic surgery due to the complex reflectance properties of the organ tissues. On the other hand, Template-Based Deformable-Shape-from-Motion (DSfM) has been recently used to recover a coarse 3D shape in laparoscopy. We propose to combine both geometric and photometric cues to robustly reconstruct 3D human organs. Our method is dubbed Deformable-Shape-from-Motion-and-Shading (DSfMS). It tackles the limits of classical SfS and DSfM methods: First the photometric template is reconstructed using rigid SfM (Shape-from-Motion) while the surgeon is exploring --- but not deforming --- the peritoneal environment. Second a rough 3D deformed shape is computed using a recent method for elastic surface from a single laparoscopic image. Third a fine 3D deformed shape is recovered using shading and specularities. The proposed approach has been validated on both synthetic data and in-vivo laparoscopic videos of a uterus. Experimental results illustrate its effectiveness compared to SfS and DSfM.

A linear least-squares solution to elastic Shape-from-Template

We cast SfT (Shape-from-Template) as the search of a vector field (X, δX), composed of the pose X and the displacement δX that produces the deformation. We propose the first fully linear least-squares SfT method modeling elastic deformations. It relies on a set of Solid Boundary Constraints (SBC) to position the template at X in the deformed frame. The displacement is mapped by the stiffness matrix to minimize the amount of force responsible for the deformation. This linear minimization is subjected to the Reprojection Boundary Constraints (RBC) of the deformed shape X + δX on the deformed image. Compared to state-of-the-art methods, this new formulation allows us to obtain accurate results at a low computation cost.

A pixel-based approach to template-based monocular 3D reconstruction of deformable surfaces

Most of the previous work on template-based deformable 3D surface reconstruction using a single view is feature-based. We propose a pixel-based formulation in a variational framework; the unknown is the surface function. The color discrepancy between the template and the deformed images is formalized as a functional of the surface function. The main difficulty in such a formulation arises when the surface self-occludes which induces discontinuities in the discrepancy measure at the self-occlusion boundary. Based on previous work on 3D rigid surface reconstruction, we rigorously formalize the visibility as a continuous functional of the surface function. It is derived in the template for visible/self-occluded regions in the deformed image. The gradient of the color discrepancy is computed with respect to the surface function. The minimization smoothly updates the surface function to fit the self-occlusion boundary. Gradient descent is initialized from feature-based 3D reconstruction. Our experimental results on simulated and real data show that during the minimization of the color discrepancy, the self-occlusion boundary of the reconstructed surface moves to its correct location in the image. We show quantitatively that in the template image, the accuracy of visible/self-occluded areas is improved to a significant extent.

Live image parsing in uterine laparoscopy

Augmented Reality (AR) can improve the information delivery to surgeons. In laparosurgery, the primary goal of AR is to provide multimodal information overlaid in live laparoscopic videos. For gynecologic laparoscopy, the 3D reconstruction of uterus and its deformable registration to preoperative data form the major problems in AR. Shape-from-Shading (SfS) and inter-frame registration require an accurate identification of the uterus region, the occlusions due to surgical tools, specularities, and other tissues. We propose a cascaded patient-specific real-time segmentation method to identify these four important regions. We use a color based Gaussian Mixture Model (GMM) to segment the tools and a more elaborate color and texture model to segment the uterus. The specularities are obtained by a saturation test. We show that our segmentation improves SfS and inter-frame registration of the uterus.

Hand–eye and radial distortion calibration for rigid endoscopes

In this paper, we propose a non‐linear calibration method for hand–eye system equipped with a camera undergoing radial distortion as the rigid endoscope. Whereas classic methods propose either a separated estimation of the camera intrinsics and the hand–eye transform or a mixed non‐linear estimation of both hand–eye and camera intrinsics assuming a pin‐hole model, the proposed approach enables a simultaneous refinement of the hand–eye and the camera parameters including the distortion factor with only three frames of the calibrated pattern.

Shape-from-Polarization in laparoscopy

We present the first prototype of 3D laparoscope using polarized lenses. Polarization encapsulates light intensity and surface orientation, from which 3D can be retrieved. Our method first acquires 3 images of the same view at 3 different orientations of the polarizer (00, 450 and 900). Second it uses Shape-from-Polarization (SfP) to calculate the surface normals. Finally it integrates the normals to estimate the 3D surface. Qualitative and quantitative comparison to Shape-from-Shading (SfS) show the effectiveness of our method and its promising use in laparoscopic surgery.

Magnification-continuous static calibration model of a scanning-electron microscope

Abstract. We present a new calibration model of both static distortion and projection for a scanning-electron microscope (SEM). The proposed calibration model depends continuously on the magnification factor. State-of-the-art methods have proposed models to solve the static distortion and projection model but for a discrete set of low and high magnifications: at low magnifications, existing models assume static distortion and perspective projection. At high magnifications, existing models assume an orthogonal projection without presence of static distortion. However, a magnification-continuous model which defines continuous switch from low to high magnifications has not yet been proposed. We propose a magnification-continuous static calibration model of the SEM. The static distortion and intrinsics of the projection matrix are modeled by partial differential equations (PDEs) with respect to magnification. The approach is applied with success to the JEOL-JSM 820 in a secondary electron imaging mode for magnification ranging from 100× to 10k×. The final RMS reprojection error is about 0.9 pixels. This result together with two application-based experiments: the consistent measurements of the bending of a cantilever and a 3-D reconstruction of a nano-ball emphasize the relevance of the proposed approach.