Kai-Che Liu

Automatic Distortion Correction of Endoscopic Images Captured With Wide-Angle Zoom Lens

Operation in minimally invasive surgery is more difficult since the surgeons perform operations without haptic feedback or depth perception. Moreover, the field of view perceived by the surgeons through endoscopy is usually quite limited. The goal of this paper is to allow surgeons to see wide-angle images from endoscopy without the drawback of lens distortion. The proposed distortion correction process consists of lens calibration and real-time image warping. The calibration step is to estimate the parameters in the lens distortion model. We propose a fully automatic Hough-entropy-based calibration algorithm, which provides calibration results comparable to the previous manual calibration method. To achieve real-time correction, we use graphics processing unit to warp the image in parallel. In addition, surgeons may adjust the focal length of a lens during the operation. Real-time distortion correction of a zoomable lens is impossible by using traditional calibration methods because the tedious calibration process has to repeat again if focal length is changed. We derive a formula to describe the relationship between the distortion parameter, focal length, and image boundary. Hence, we can estimate the focal length for a zoomable lens from endoscopic images online and achieve real-time lens distortion correction.

Deformable three-dimensional model architecture for interactive augmented reality in minimally invasive surgery

BackgroundSurgical procedures have undergone considerable advancement during the last few decades. More recently, the availability of some imaging methods intraoperatively has added a new dimension to minimally invasive techniques. Augmented reality in surgery has been a topic of intense interest and research.MethodsAugmented reality involves usage of computer vision algorithms on video from endoscopic cameras or cameras mounted in the operating room to provide the surgeon additional information that he or she otherwise would have to recognize intuitively. One of the techniques combines a virtual preoperative model of the patient with the endoscope camera using natural or artificial landmarks to provide an augmented reality view in the operating room. The authors’ approach is to provide this with the least number of changes to the operating room. Software architecture is presented to provide interactive adjustment in the registration of a three-dimensional (3D) model and endoscope video.ResultsAugmented reality including adrenalectomy, ureteropelvic junction obstruction, and retrocaval ureter and pancreas was used to perform 12 surgeries. The general feedback from the surgeons has been very positive not only in terms of deciding the positions for inserting points but also in knowing the least change in anatomy.ConclusionsThe approach involves providing a deformable 3D model architecture and its application to the operating room. A 3D model with a deformable structure is needed to show the shape change of soft tissue during the surgery. The software architecture to provide interactive adjustment in registration of the 3D model and endoscope video with adjustability of every 3D model is presented.

Stereoscopic visualization of laparoscope image using depth information from 3D model

Laparoscopic surgery is indispensable from the current surgical procedures. It uses an endoscope system of camera and light source, and surgical instruments which pass through the small incisions on the abdomen of the patients undergoing laparoscopic surgery. Conventional laparoscope (endoscope) systems produce 2D colored video images which do not provide surgeons an actual depth perception of the scene. In this work, the problem was formulated as synthesizing a stereo image of the monocular (conventional) laparoscope image by incorporating into them the depth information from a 3D CT model. Various algorithms of the computer vision including the algorithms for the feature detection, matching and tracking in the video frames, and for the reconstruction of 3D shape from shading in the 2D laparoscope image were combined for making the system. The current method was applied to the laparoscope video at the rate of up to 5 frames per second to visualize its stereo video. A correlation was investigated between the depth maps calculated with our method with those from the shape from shading algorithm. The correlation coefficients between the depth maps were within the range of 0.70-0.95 (P<0.05). A t-test was used for the statistical analysis.

A stereoscopic video conversion scheme based on spatio-temporal analysis of MPEG videos

In this article, an automatic stereoscopic video conversion scheme which accepts MPEG-encoded videos as input is proposed. Our scheme is depth-based, relying on spatio-temporal analysis of the decoded video data to yield depth perception cues, such as temporal motion and spatial contrast, which reflect the relative depths between the foreground and the background areas. Our scheme is shot-adaptive, demanding that shot change detection and shot classification be performed for tuning of algorithm or parameters that are used for depth cue combination. The above-mentioned depth estimation is initially block-based, followed by a locally adaptive joint trilateral upsampling algorithm to reduce the computing load significantly. A recursive temporal filter is used to reduce the possible depth fluctuations (and also artifacts in the synthesized images) resulting from wrong depth estimations. The traditional Depth-Image-Based-Rendering algorithm is used to synthesize the left- and right-view frames for 3D display. Subjective tests show that videos converted by our scheme provide comparable perceived depth and visual quality with those converted from the depth data calculated by stereo vision techniques. Also, our scheme is shown to outperform the well-known TriDef software in terms of human’s perceived 3D depth. Based on the implementation by using “OpenMP” parallel programming model, our scheme is capable of executing in real-time on a multi-core CPU platform.

Wide-angle distortion correction by Hough transform and gradient estimation

Wide-angle cameras have been widely used in surveillance and endoscopic imaging. An automatic distortion correction method is very useful for these applications. Traditional methods extract corners or curved straight lines for estimating distortion parameters. Hough transform is a powerful tool to assess straightness. However, previous methods usually require some human intervention or only focus on using a single curve. In this paper, we propose a new method based on Hough transform by considering all curves into the estimation of distortion parameters. By considering the relationship between distortion parameters and curves, our method is fully automatic and does not require manual selection of curves in an image. Experiments on synthetic and real datasets have been conducted. The results of our method are also compared with other Hough Transform based methods in quantitative measures. The experimental results show that the accuracy of the proposed automatic method is comparable to those of other manual line-based methods.

Quality enhancement based on retinex and pseudo-HDR synthesis algorithms for endoscopic images

In this paper, we present a quality enhancement scheme for endoscopic images. Traditional algorithms might be able to enhance the image contrast, but possible over-enhancement also lead to bad overall visual quality which prevents surgeons from accurate examination or operations of instruments in Minimal Invasive Surgery (MIS). Our proposed scheme integrates the well-known retinex algorithm with a pseudo-HDR (High Dynamic Range) synthesis process, designed to compose of three parts: multiscale retinex with gamma correction (MSR-G), local brightness range expansion (brightness diversity), and bilateral-filter-based HDR image fusion. Experiment results demonstrate that the proposed scheme is able to enhance image details and keep the overall visual quality good as well, with respect to other existing methods.

A landmark based registration technique for minimally invasive spinal surgery

This paper proposed a technique for skin curve registration based on landmark detection and calibrated camera-projector system for medical purpose. The technique applies the algorithm which registers a precalculated 3D model for the 2D images of the video frames. The algorithm first calculates the 3D locations of landmarks based upon the calibrated camera projector system and the matching the landmarks in the 3D model using CT-scan. We then generate the image to be projected on the skin curve of a patient from the registered 3D spinal model, and it is created for projecting on the pork or real patient for surgeons to process surgical procedure by utilizing their natural eyes system. The experiments demonstrate the proposed registration method on both animal and real patient and evaluated by several surgeons for spinal surgery.

3D Spinal Cord and Nerves Segmentation from STIR-MRI

In this paper, we present a system for spinal cord and nerves segmentation from STIR-MRI. We propose an user interactive segmentation method for 3D images, which is extended from the 2D random walker algorithm and implemented with a slice-section strategy. After obtaining the 3D segmentation result, we build the 3D spinal cord and nerves model for each view using VTK, which is an open-source, freely available software. Then we obtain the point cloud of the spinal cord and nerves surface by registering the three surface models constructed from three STIR-MRI images of different directions. In the experimental results, we show the 3D segmentation results of spinal cord and nerves from the STIR-MRI (Short Tau Inversion Recovery - Magnetic Resonance Imaging)images in three different views, and also display the reconstructed 3D surface model.

A Specular Reflection Suppression Method for Endoscopic Images

In this paper, we proposed a specular reflection suppression method for endoscopic images. To reduce the impact of specular reflection on visual quality, the proposed method is composed of two parts: reflection detection and reflection suppression. To detect reflection regions, a thresholding algorithm is used. To suppress the effect of specular reflection, a modified image inpainting algorithm is developed. Our simulation results show that the proposed method can not only detect but also suppress specular reflection in endoscopic images. In addition, the proposed specular reflection suppression method outperforms some existing methods.

Distinguishing normal and pulmonary edema chest x-ray using Gabor filter and SVM

Pulmonary edema, i.e. excess of extravascular fluid in lungs, is a common manifestation of various clinical conditions. Although the etiology of the pulmonary edema is deduced with the help of history, physical examination and various biochemical and radiological investigations, computer aided evaluation of pulmonary edema will be helpful for physicians in determining the course of management for the condition. In this study we present texture analysis of chest x-ray, using Gabor filter and one of the machine learning techniques, Support Vector Machine (SVM), to distinguish the normal chest x-ray from the chest x-ray of pulmonary edema. This is an initial step towards computer aided quantitative assessment of the pulmonary edema using chest x-ray.