Mingxing Hu

Biopsy Site Re-localisation Based on the Computation of Epipolar Lines from Two Previous Endoscopic Images

Tracking biopsy sites in endoscopic images can be useful to provide a visual aid for the guidance of surgical tools, for example when endoscopic guided biopsy is required. A new method for re-localisation of these sites is presented in this paper. It makes use of epipolar geometry properties between three images of the same site observed from different viewpoints with an endoscope. Two epipolar lines are derived from the two first images in the third image where the site needs to be re-localised. Their intersection corresponds to the location of the biopsy site. This method was tested with gastroscopic data from 2 patients with 9 series of three images of the oesophagus. The re-localisation error was estimated at less than 1.5 millimetres by a clinical endoscopist, which is sufficient for most clinical endoscopic applications.

3D reconstruction of internal organ surfaces for minimal invasive surgery

While Minimally Invasive Surgery (MIS) offers great benefits to patients compared with open surgery surgeons suffer from a restricted field-of-view and obstruction from instruments. We present a novel method for 3D reconstruction of soft tissue, which can provide a wider field-of-view with 3D information for surgeons, including restoration of missing data. The paper focuses on the use of Structure from Motion (SFM) techniques to solve the missing data problem and application of competitive evolutionary agents to improve the robustness to missing data and outliers. The method has been evaluated with synthetic data, images from a phantom heart model, and in vivo MIS image sequences using the da Vinci telerobotic surgical system.

Image guidance for robotic minimally invasive coronary artery bypass.

A novel system for image guidance in totally endoscopic coronary artery bypass (TECAB) is presented. Key requirement is the availability of 2D-3D registration techniques that can deal with non-rigid motion and deformation. Image guidance for TECAB is mainly required before the mechanical stabilisation of the heart, when the most dominant source of misregistration is the deformation and non-rigid motion of the heart. To augment the images in the endoscope of the da Vinci robot, we have to find the transformation from the coordinate system of the preoperative imaging modality to the system of the endoscopic cameras. In a first step we build a 4D motion model of the beating heart. Intraoperatively we can use the ECG or video processing to determine the phase of the cardiac cycle, as well as the heart and respiratory frequencies. We then take the heart surface from the motion model and register it to the stereo endoscopic images of the da Vinci robot resp. of a validation system using photo-consistency. To take advantage of the fact that there is a whole image sequence available for registration, we use the different phases together to get the registration. We found the similarity function to be much smoother when using more phases. This also showed promising behaviour in convergence tests. Images of the vessels available in the preoperative coordinate system can then be transformed to the camera system and projected into the calibrated endoscope view using two video mixers with chroma keying. It is hoped that the augmented view can improve the efficiency of TECAB surgery and reduce the conversion rate to more conventional procedures.

Registration of the endoluminal surfaces of the colon derived from prone and supine CT colonography

PURPOSE Computed tomographic (CT) colonography is a relatively new technique for detecting bowel cancer or potentially precancerous polyps. CT scanning is combined with three-dimensional (3D) image reconstruction to produce a virtual endoluminal representation similar to optical colonoscopy. Because retained fluid and stool can mimic pathology, CT data are acquired with the bowel cleansed and insufflated with gas and patient in both prone and supine positions. Radiologists then match visually endoluminal locations between the two acquisitions in order to determine whether apparent pathology is real or not. This process is hindered by the fact that the colon, essentially a long tube, can undergo considerable deformation between acquisitions. The authors present a novel approach to automatically establish spatial correspondence between prone and supine endoluminal colonic surfaces after surface parameterization, even in the case of local colon collapse. METHODS The complexity of the registration task was reduced from a 3D to a 2D problem by mapping the surfaces extracted from prone and supine CT colonography onto a cylindrical parameterization. A nonrigid cylindrical registration was then performed to align the full colonic surfaces. The curvature information from the original 3D surfaces was used to determine correspondence. The method can also be applied to cases with regions of local colonic collapse by ignoring the collapsed regions during the registration. RESULTS Using a development set, suitable parameters were found to constrain the cylindrical registration method. Then, the same registration parameters were applied to a different set of 13 validation cases, consisting of 8 fully distended cases and 5 cases exhibiting multiple colonic collapses. All polyps present were well aligned, with a mean (+/- std. dev.) registration error of 5.7 (+/- 3.4) mm. An additional set of 1175 reference points on haustral folds spread over the full endoluminal colon surfaces resulted in an error of 7.7 (+/- 7.4) mm. Here, 82% of folds were aligned correctly after registration with a further 15% misregistered by just onefold. CONCLUSIONS The proposed method reduces the 3D registration task to a cylindrical registration representing the endoluminal surface of the colon. Our algorithm uses surface curvature information as a similarity measure to drive registration to compensate for the large colorectal deformations that occur between prone and supine data acquisitions. The method has the potential to both enhance polyp detection and decrease the radiologists interpretation time.

Reconstruction of a 3D surface from video that is robust to missing data and outliers: Application to minimally invasive surgery using stereo and mono endoscopes

Minimally invasive surgery (MIS) offers great benefits to patients compared with open surgery. Nevertheless during MIS surgeons often need to contend with a narrow field-of-view of the endoscope and obstruction from other surgical instruments. He/she may also need to relate the surgical scene to information derived from previously acquired 3D medical imaging. We thus present a new framework to reconstruct the 3D surface of an internal organ from endoscopic images which is robust to measurement noise, missing data and outliers. This can provide 3D surface with a wide field-of-view for surgeons, and it can also be used for 3D-3D registration of the anatomy to pre-operative CT/MRI data for use in image guided interventions. Our proposed method first removes most of the outliers using an outlier removal method that is based on the trilinear constraints over three images. Then data that are missing from one or more of the video images (missing data) and 3D structure are recovered using the structure from motion (SFM) technique. Evolutionary agents are applied to improve both the efficiency of data recovery and robustness to outliers. Furthermore, an incremental bundle adjustment strategy is used to refine the camera parameters and 3D structure and produce a more accurate 3D surface. Experimental results with synthetic data show that the method is able to reconstruct surfaces in the presence of feature tracking errors (up to 5 pixel standard deviation) and a large amount of missing data (up to 50%). Experiments on a realistic phantom model and in vivo data further demonstrate the good performance of the proposed approach in terms of accuracy (1.7 mm residual phantom surface error) and robustness (50% missing data rate, and 20% outliers in in vivo experiments).

Non-rigid Reconstruction of the Beating Heart Surface for Minimally Invasive Cardiac Surgery

This paper presents a new method to reconstruct the beating heart surface based on the non-rigid structure from motion technique using preprocessed endoscopic images. First the images captured at the same phase within each heart cycle are automatically extracted from the original image sequence to reduce the dimension of the deformation subspace. Then the remaining residual non-rigid motion is restricted to lie within a low-dimensional subspace and a probabilistic model is used to recover the 3D structure and camera motion simultaneously. Outliers are removed iteratively based on the reprojection error, Missing data are also recovered with an Expectation Maximization algorithm. As a result the camera can move around the operation scene to build a 3D surface with a wide field-of-view for intra-operative procedures. The method has been evaluated with synthetic data, heart phantom data, and in vivo data from a da Vinci surgical system.

Epipolar geometry estimation based on evolutionary agents

This paper presents a novel approach based on the use of evolutionary agents for epipolar geometry estimation. In contrast to conventional nonlinear optimization methods, the proposed technique employs each agent to denote a minimal subset to compute the fundamental matrix, and considers the data set of correspondences as a 1D cellular environment, in which the agents inhabit and evolve. The agents execute some evolutionary behavior, and evolve autonomously in a vast solution space to reach the optimal (or near optima) result. Then three different techniques are proposed in order to improve the searching ability and computational efficiency of the original agents. Subset template enables agents to collaborate more efficiently with each other, and inherit accurate information from the whole agent set. Competitive evolutionary agent (CEA) and finite multiple evolutionary agent (FMEA) apply a better evolutionary strategy or decision rule, and focus on different aspects of the evolutionary process. Experimental results with both synthetic data and real images show that the proposed agent-based approaches perform better than other typical methods in terms of accuracy and speed, and are more robust to noise and outliers.

Re-localisation of a biopsy site in endoscopic images and characterisation of its uncertainty

Endoscopy guided probe-based optical biopsy is a new method for detecting sites for tissue biopsy and treatment. After detection, it can be useful to provide a visual aid in the endoscopic images to the endoscopist for example for guidance of forceps to the biopsy sites detected optically. A new method for re-localisation of these sites during the endoscopic examination is presented in this paper. It makes use of a sequence of endoscopic images, where the biopsy site location is known, in order to derive the same number of epipolar lines as images in the sequence projected onto a subsequent target image where the re-localised biopsy site needs to be computed. The location of the re-localised biopsy site is found by minimisation of the sum of squared distances to the epipolar lines. The method also determines analytically the uncertainty of the re-localised biopsy site. This provides the endoscopist with a confidence region around the re-localised biopsy site and a measure of the re-localisation precision. Simulations confirmed that the analytical uncertainty has the potential to be a good estimation of the experimental uncertainty. The method was tested on a physical phantom and on real data from four patients with eight sequences of images acquired during gastroscopy. The re-localisation precision and accuracy were estimated at 1 millimetre or better, which is sufficient for re-localisation of optical biopsy sites.

A Novel Algorithm for Heart Motion Analysis Based on Geometric Constraints

Recently, much attention has been focused on heart motion analysis for minimally invasive beating-heart surgery. Unfortunately existing techniques usually require the camera(s) to be fixed during the motion analysis, which can restrict its usefulness during surgery. In this paper we present a novel method for heart motion analysis using geometric constraint, which can estimate the motion from a moving camera and employ multiple image features to improve robustness to noise. Our approach combines the benefits of geometry estimation for obtaining an accurate and robust solution with the proper treatment of respiratory motion. The proposed method can be applied to not only beating heart surgery, but also to other procedures involving periodic organ motion, such as lung and liver.

Establishing spatial correspondence between the inner colon surfaces from prone and supine CT colonography

Colonography is an important screening tool for colorectal lesions. This paper presents a method for establishing spatial correspondence between prone and supine inner colon surfaces reconstructed from CT colonography. The method is able to account for the large deformations and torsions of the colon occurring through movement between the two positions. Therefore, we parameterised the two surfaces in order to provide a 2D indexing system over the full length of the colon using the Ricci flow method. This provides the input to a non-rigid B-spline registration in 2D space which establishes a correspondence for each surface point of the colon in prone and supine position. The method was validated on twelve clinical cases and demonstrated promising registration results over the majority of the colon surface.