Yen-Yu Wang

Stereoscopic visualization of laparoscope image using depth information from 3D model

Laparoscopic surgery is indispensable from the current surgical procedures. It uses an endoscope system of camera and light source, and surgical instruments which pass through the small incisions on the abdomen of the patients undergoing laparoscopic surgery. Conventional laparoscope (endoscope) systems produce 2D colored video images which do not provide surgeons an actual depth perception of the scene. In this work, the problem was formulated as synthesizing a stereo image of the monocular (conventional) laparoscope image by incorporating into them the depth information from a 3D CT model. Various algorithms of the computer vision including the algorithms for the feature detection, matching and tracking in the video frames, and for the reconstruction of 3D shape from shading in the 2D laparoscope image were combined for making the system. The current method was applied to the laparoscope video at the rate of up to 5 frames per second to visualize its stereo video. A correlation was investigated between the depth maps calculated with our method with those from the shape from shading algorithm. The correlation coefficients between the depth maps were within the range of 0.70-0.95 (P<0.05). A t-test was used for the statistical analysis.

Distinguishing normal and pulmonary edema chest x-ray using Gabor filter and SVM

Pulmonary edema, i.e. excess of extravascular fluid in lungs, is a common manifestation of various clinical conditions. Although the etiology of the pulmonary edema is deduced with the help of history, physical examination and various biochemical and radiological investigations, computer aided evaluation of pulmonary edema will be helpful for physicians in determining the course of management for the condition. In this study we present texture analysis of chest x-ray, using Gabor filter and one of the machine learning techniques, Support Vector Machine (SVM), to distinguish the normal chest x-ray from the chest x-ray of pulmonary edema. This is an initial step towards computer aided quantitative assessment of the pulmonary edema using chest x-ray.

Surface reconstruction from endoscopic image sequence

The current study describes a technique for reconstructing a three dimensional surface from multiple images of the (single camera) endoscopic video. A 3D shape of the scene in each image frame of the endoscopic video was reconstructed using shape from shading technique. The characteristic feature points on the 2D images were detected using SURF algorithm and the features were matched using BRIEF and Hamming distance criteria. The matched feature points between the consecutive frames were used to find the transformation matrix to align the 3D surfaces of the consecutive video frames. Using the transformation matrix, the 3D surfaces were stitched (registered) together. The method was applied on a recorded video of the laparoscopic surgery. The reconstructed surface provides a wider view and depth information to the viewers.

Stereoscopic augmented reality for single camera endoscopy: a virtual study

AbstractEndoscopic surgery causes less tissue injury compared to open surgical techniques, thus promoting more rapid recuperation and reduced post-operative pain. Endoscopy, however, allows the surgeon to visualise only the anatomical surface of the surgical site, with a relatively narrow field of view. Moreover, the 2D video captured by the conventional endoscope does not provide depth perception of the surgical scene. In this study, these limitations have been addressed with the development of an augmented reality (AR) system with stereoscopic visualisation. A phantom and its 3D CT model were used, respectively, to form the real and virtual parts of the AR. The virtual environment camera pose was tracked using algorithms for image feature detection, feature matching and Perspective-n-Point applied on the endoscopic image and the 3D virtual model-rendered image. The endoscope video frame- and the virtual model-rendered images were superimposed to form the AR composite view. The depth buffer (z-buffer) of...

Laparoscopic video augmentation with infrared image information

Sub-surface information during a laparoscopic surgery is captured with a near infrared imaging where the tissue is highlighted with the help of an indocyanine dye injected to the patient before the surgery. The surgeons can use such system during the surgery, however, they need to swap the light source between infra-red and the white light to complete the surgery. Swapping the light source causes inconvenience and increase in the time of the surgery. The current study aims to develop a method for superimposing and tracking the infrared imagery on the laparoscopic video acquired under white light. In this study, the infrared imagery was captured and registered to one of the white light imagery from the endoscope video using a 2D image registration technique. The registration technique was based on maximization of mutual information between two images. The edges on the registered infrared image was detected using canny edge detection algorithm. A homography matrix between consecutive video frames under white light was calculated after characteristic feature point detection and matching. The homography matrix was applied on the registered infrared image (only edge) and a new registered edge image was generated. The process continued for every new frame of the video under white light. The system was applied with a video acquisition rate of 20 frames per second. The system needs to be evaluated for its influence on the time of surgery.

An Application of Robot Arm for Dynamic Field of View Expansion During Laparoscopy

An expanding mosaic view is one of the possible solutions for narrow field of view during endoscopic surgery. The current work presents a system to create an expanding mosaic view of the video during laparoscopy where the laparoscope is held and moved by a customized robot arm.

Stereoscopic Augmented Reality for Single Camera Endoscope Using Optical Tracker: A Study on Phantom

Introduction: Endoscopic surgery causes reduced injury to tissues which helps in rapid recovery and less painful post-operative period of patients. However, its narrow field of view and loss of depth perception in endoscope image make surgeons task difficult. Moreover, in endoscopic surgery surgeons can only see the surface of the surgical anatomy. In this study these limitations have been addressed with the development of an augmented reality system with 3D visualization. Method: The system is comprising of infrared based optical tracker, 2D endoscope system, computer and a 3D monitor. The system was calibrated to bring all the components to a single reference frame of the optical tracker. A phantom and its 3D CT model were used, respectively, to form the real and virtual part of the augmented reality. The endoscope camera position was tracked in 3D using a tracker tool mounted on it. The camera in the virtual environment was updated with the new positions of the endoscope camera. The endoscope video frame and the virtual model rendered image were superimposed to form augmented reality view. The depth map of the virtual scene was further used to make an stereo pair of the augmented reality. Results: The AR system was examined with endoscope video of phantom. It could produce well aligned real and virtual component. The contour different of the two components was 5 to 10 mm. Conclusion: An AR system for endoscopic surgery was successfully implemented on a phantom. It needs to include motion and deformation model to be applied on real patients.

Three Dimensional Panorama Visualization for Endoscopic Video

Introduction: Endoscopy is widely used in the surgical world. Reduced surgical injury during endoscopic surgery makes the patient’s life easy. However, the endoscopic surgery becomes a challenge for the novice surgeons because of the narrow field of view and the lack of 3D perception in the 2D endoscope image. Such limitations of the endoscope may be addressed with a 3D panorama created with the endoscopic 2D images. Few studies have been reported to create a 3D panorama from endoscopic images. However, to our knowledge, no study has reported a 3D panorama system which uses only a single camera image from the conventional endoscope to make a 3D shape of the organs, and further stitch those 3D shapes to create a 3D panorama of the surgical scene. Such system would enable surgeons to have an extended field of view of the surgical scene with a 3D perception. Method: Images from the endoscopic surgery video were used in this study. A shape from shading (SfS) algorithm was applied to create a 3D shape of the organs in the images. Characteristic feature points were identified on the images using a feature detection algorithm (SURF). The matching feature points in the consecutive images were found using a feature matching algorithm (BRIEF). An iterative closest point (ICP) algorithm was applied to stitch 3D shapes from the consecutive images to synthesize the 3D panorama. Results: The method was applied on 100 consecutive video frames from an endoscopic video of a patient. The root mean square error for the registration of the consecutive image feature points was <4 mm. The 3D panorama from the method helps in visualizing a larger area of the surgical anatomy. An improved version of the method may be applied to a real time video from the endoscope.

Stereoscopic laparoscopy using depth information from 3D model

Laparoscopic surgery is indispensable from the current surgical procedures. Conventional laparoscope (endoscope) systems produce 2D colored video images which do not provide surgeons an actual depth perception of the scene. In this work, the problem was formulated as synthesizing a stereo image of the monocular (conventional) laparoscope image by incorporating into them the depth information from a 3D CT model. The current method was applied to the laparoscope video at the rate of up to 5 frames per second to visualize its stereo video. Correlation coefficients between the depth maps calculated with our method with those from the shape from shading algorithm were within the range of 0.70 to 0.95 (P <; 0.05).