Stefan Suwelack

Dense GPU‐enhanced surface reconstruction from stereo endoscopic images for intraoperative registration

PURPOSE In laparoscopic surgery, soft tissue deformations substantially change the surgical site, thus impeding the use of preoperative planning during intraoperative navigation. Extracting depth information from endoscopic images and building a surface model of the surgical field-of-view is one way to represent this constantly deforming environment. The information can then be used for intraoperative registration. Stereo reconstruction is a typical problem within computer vision. However, most of the available methods do not fulfill the specific requirements in a minimally invasive setting such as the need of real-time performance, the problem of view-dependent specular reflections and large curved areas with partly homogeneous or periodic textures and occlusions. METHODS In this paper, the authors present an approach toward intraoperative surface reconstruction based on stereo endoscopic images. The authors describe our answer to this problem through correspondence analysis, disparity correction and refinement, 3D reconstruction, point cloud smoothing and meshing. Real-time performance is achieved by implementing the algorithms on the gpu. The authors also present a new hybrid cpu-gpu algorithm that unifies the advantages of the cpu and the gpu version. RESULTS In a comprehensive evaluation using in vivo data, in silico data from the literature and virtual data from a newly developed simulation environment, the cpu, the gpu, and the hybrid cpu-gpu versions of the surface reconstruction are compared to a cpu and a gpu algorithm from the literature. The recommended approach toward intraoperative surface reconstruction can be conducted in real-time depending on the image resolution (20 fps for the gpu and 14fps for the hybrid cpu-gpu version on resolution of 640 × 480). It is robust to homogeneous regions without texture, large image changes, noise or errors from camera calibration, and it reconstructs the surface down to sub millimeter accuracy. In all the experiments within the simulation environment, the mean distance to ground truth data is between 0.05 and 0.6 mm for the hybrid cpu-gpu version. The hybrid cpu-gpu algorithm shows a much more superior performance than its cpu and gpu counterpart (mean distance reduction 26% and 45%, respectively, for the experiments in the simulation environment). CONCLUSIONS The recommended approach for surface reconstruction is fast, robust, and accurate. It can represent changes in the intraoperative environment and can be used to adapt a preoperative model within the surgical site by registration of these two models.

Physics‐based shape matching for intraoperative image guidance

PURPOSE Soft-tissue deformations can severely degrade the validity of preoperative planning data during computer assisted interventions. Intraoperative imaging such as stereo endoscopic, time-of-flight or, laser range scanner data can be used to compensate these movements. In this context, the intraoperative surface has to be matched to the preoperative model. The shape matching is especially challenging in the intraoperative setting due to noisy sensor data, only partially visible surfaces, ambiguous shape descriptors, and real-time requirements. METHODS A novel physics-based shape matching (PBSM) approach to register intraoperatively acquired surface meshes to preoperative planning data is proposed. The key idea of the method is to describe the nonrigid registration process as an electrostatic-elastic problem, where an elastic body (preoperative model) that is electrically charged slides into an oppositely charged rigid shape (intraoperative surface). It is shown that the corresponding energy functional can be efficiently solved using the finite element (FE) method. It is also demonstrated how PBSM can be combined with rigid registration schemes for robust nonrigid registration of arbitrarily aligned surfaces. Furthermore, it is shown how the approach can be combined with landmark based methods and outline its application to image guidance in laparoscopic interventions. RESULTS A profound analysis of the PBSM scheme based on in silico and phantom data is presented. Simulation studies on several liver models show that the approach is robust to the initial rigid registration and to parameter variations. The studies also reveal that the method achieves submillimeter registration accuracy (mean error between 0.32 and 0.46 mm). An unoptimized, single core implementation of the approach achieves near real-time performance (2 TPS, 7-19 s total registration time). It outperforms established methods in terms of speed and accuracy. Furthermore, it is shown that the method is able to accurately match partial surfaces. Finally, a phantom experiment demonstrates how the method can be combined with stereo endoscopic imaging to provide nonrigid registration during laparoscopic interventions. CONCLUSIONS The PBSM approach for surface matching is fast, robust, and accurate. As the technique is based on a preoperative volumetric FE model, it naturally recovers the position of volumetric structures (e.g., tumors and vessels). It cannot only be used to recover soft-tissue deformations from intraoperative surface models but can also be combined with landmark data from volumetric imaging. In addition to applications in laparoscopic surgery, the method might prove useful in other areas that require soft-tissue registration from sparse intraoperative sensor data (e.g., radiation therapy).

Context-aware Augmented Reality in laparoscopic surgery

Augmented Reality is a promising paradigm for intraoperative assistance. Yet, apart from technical issues, a major obstacle to its clinical application is the man-machine interaction. Visualization of unnecessary, obsolete or redundant information may cause confusion and distraction, reducing usefulness and acceptance of the assistance system. We propose a system capable of automatically filtering available information based on recognized phases in the operating room. Our system offers a specific selection of available visualizations which suit the surgeons needs best. The system was implemented for use in laparoscopic liver and gallbladder surgery and evaluated in phantom experiments in conjunction with expert interviews.

Pose-independent surface matching for intra-operative soft-tissue marker-less registration

One of the main challenges in computer-assisted soft tissue surgery is the registration of multi-modal patient-specific data for enhancing the surgeons navigation capabilities by observing beyond exposed tissue surfaces. A new approach to marker-less guidance involves capturing the intra-operative patient anatomy with a range image device and doing a shape-based registration. However, as the target organ is only partially visible, typically does not provide salient features and underlies severe non-rigid deformations, surface matching in this context is extremely challenging. Furthermore, the intra-operatively acquired surface data may be subject to severe systematic errors and noise. To address these issues, we propose a new approach to establishing surface correspondences, which can be used to initialize fine surface matching algorithms in the context of intra-operative shape-based registration. Our method does not require any prior knowledge on the relative poses of the input surfaces to each other, does not rely on the detection of prominent surface features, is robust to noise and can be used for overlapping surfaces. It takes into account (1) similarity of feature descriptors, (2) compatibility of multiple correspondence pairs, as well as (3) the spatial configuration of the entire correspondence set. We evaluate the algorithm on time-of-flight (ToF) data from porcine livers in a respiratory liver motion simulator. In all our experiments the alignment computed from the established surface correspondences yields a registration error below 1cm and is thus well suited for initializing fine surface matching algorithms for intra-operative soft-tissue registration.

Real-time surface reconstruction from stereo endoscopic images for intraoperative registration

Minimally invasive surgery is a medically complex discipline that can heavily benefit from computer assistance. One way to assist the surgeon is to blend in useful information about the intervention into the surgical view using Augmented Reality. This information can be obtained during preoperative planning and integrated into a patient-tailored model of the intervention. Due to soft tissue deformation, intraoperative sensor data such as endoscopic images has to be acquired and non-rigidly registered with the preoperative model to adapt it to local changes. Here, we focus on a procedure that reconstructs the organ surface from stereo endoscopic images with millimeter accuracy in real-time. It deals with stereo camera calibration, pixel-based correspondence analysis, 3D reconstruction and point cloud meshing. Accuracy, robustness and speed are evaluated with images from a test setting as well as intraoperative images. We also present a workflow where the reconstructed surface model is registered with a preoperative model using an optical tracking system. As preliminary result, we show an initial overlay between an intraoperative and a preoperative surface model that leads to a successful rigid registration between these two models.

OpenHELP (Heidelberg laparoscopy phantom): development of an open-source surgical evaluation and training tool

AbstractBackgroundApart from animal testing and clinical trials, surgical research and laparoscopic training mainly rely on phantoms. The aim of this project was to design a phantom with realistic anatomy and haptic characteristics, modular design and easy reproducibility. The phantom was named open-source Heidelberg laparoscopic phantom (OpenHELP) and serves as an open-source platform. MethodsThe phantom was based on an anonymized CT scan of a male patient. The anatomical structures were segmented to obtain digital three-dimensional models of the torso and the organs. The digital models were materialized via rapid prototyping. One flexible, using an elastic abdominal wall, and one rigid method, using a plastic shell, to simulate pneumoperitoneum were developed. Artificial organ production was carried out sequentially starting from raw gypsum models to silicone molds to final silicone casts. The reproduction accuracy was exemplarily evaluated for ten silicone rectum models by comparing the digital 3D surface of the original rectum with CT scan by calculating the root mean square error of surface variations. Haptic realism was also evaluated to find the most realistic silicone compositions on a visual analog scale (VAS, 0–10).ResultsThe rigid and durable plastic torso and soft silicone organs of the abdominal cavity were successfully produced. A simulation of pneumoperitoneum could be created successfully by both methods. The reproduction accuracy of ten silicone rectum models showed an average root mean square error of 2.26 (0–11.48) mm. Haptic realism revealed an average value on a VAS of 7.25 (5.2–9.6) for the most realistic rectum.ConclusionThe OpenHELP phantom proved to be feasible and accurate. The phantom was consecutively applied frequently in the field of computer-assisted surgery at our institutions and is accessible as an open-source project at www.open-cas.org for the academic community.

Quadratic Corotated Finite Elements for Real-Time Soft Tissue Registration

Organ motion due to respiration and contact with surgical instruments can significantly degrade the accuracy of image-guided surgery. In most applications, the ensuing soft tissue deformations have to be compensated in order to register preoperative planning data to the patient. Biomechanical models can be used to perform registration based on sparse intraoperative sensor data. Using elasticity theory, the approach can be formulated as a boundary value problem with displacement boundary conditions. In this paper, we propose to use corotated finite elements (FE) with quadratic shape functions as a robust and accurate model for real-time soft-tissue registration. A detailed numerical analysis reveals that quadratic FE perform significantly better than linear corotated FE for high resolution meshes. We also show that the method achieves nearly the same registration accuracy as a complex nonlinear viscoelastic material model. Furthermore, a phantom experiment demonstrates how the model can be used for intraoperative liver registration.

A biomechanical liver model for intraoperative soft tissue registration

Organ motion due to respiration and contact with surgical instruments can significantly degrade the accuracy of image guided surgery. In most applications the ensuing soft tissue deformations have to be compensated in order to register preoperative planning data to the patient. Biomechanical models can be used to perform an accurate registration based on sparse intraoperative sensor data. Using elasticity theory, the approach can be formulated as a boundary value problem with displacement boundary conditions. In this paper, several models of the liver from the literature and a new simplified model are evaluated with regards to their application to intraoperative soft tissue registration. We construct finite element models of a liver phantom using the different material laws. Thereafter, typical deformation pattern that occur during surgery are imposed by applying displacement boundary conditions. A comparative numerical study shows that the maximal registration error of all non-linear models stays below 1.1mm, while the linear model produces errors up to 3.9mm. It can be concluded that linear elastic models are not suitable for the registration of the liver and that a geometrically non-linear formulation has to be used. Although the stiffness parameters of the non-linear materials differ considerably, the calculated displacement fields are very similar. This suggests that a difficult patient-specific parameterization of the model might not be necessary for intraoperative soft tissue registration. We also demonstrate that the new simplified model achieves nearly the same registration accuracy as complex quasi-linear viscoelastic models.

From stereo image sequences to smooth and robust surface models using temporal information and Bilateral postprocessing

Reconstruction of surface models from camera images has many applications in robotics such as surface registration or object recognition. In this paper, we describe a workflow in which we extract depth information from stereo image sequences to generate a surface model. We present our solutions to correspondence analysis, disparity correction and refinement, as well as 3D reconstruction, point cloud smoothing and meshing. One important feature of the correspondence analysis that we evaluate in detail is the use of temporal information. Another emphasis is on correcting and smoothing the disparity images as well as the reconstructed point cloud without losing too much detail. We, hence, introduce our application of the Bilateral filter on disparity images and our usage of least squares smoothing. The components of the workflow were evaluated using three image sources: Endoscopic images from the daVinci® telemanipulator; images from a stereo camera integrated in the ARMAR III humanoid robot; synthetic data. Depending on the image resolution and the application, the workflow reconstructs surface models in real-time. We show that by using temporal information we obtain more accurate and robust correspondences. Additionally, the Bilateral filter was especially useful in refining the correspondences extracted from endoscopic images as well as the synthetic data sets, whereas the least squares method showed good results in smoothing the point cloud of ARMAR III images. Overall, the presented approach achieves good results for different camera settings and image types, especially with respect to the real-time requirement.

Intraoperative on-the-fly organ-mosaicking for laparoscopic surgery

The goal of computer-assisted surgery is to provide the surgeon with guidance during an intervention using augmented reality (AR). To display preoperative data correctly, soft tissue deformations that occur during surgery have to be taken into consideration. Optical laparoscopic sensors, such as stereo endoscopes, can produce a 3D reconstruction of single stereo frames for registration. Due to the small field of view and the homogeneous structure of tissue, reconstructing just a single frame in general will not provide enough detail to register and update preoperative data due to ambiguities. In this paper, we propose and evaluate a system that combines multiple smaller reconstructions from different viewpoints to segment and reconstruct a large model of an organ. By using GPU-based methods we achieve near real-time performance. We evaluated the system on an ex-vivo porcine liver (4.21mm± 0.63) and on two synthetic silicone livers (3.64mm ± 0.31 and 1.89mm ± 0.19) using three different methods for estimating the camera pose (no tracking, optical tracking and a combination).