Joseph Marino

Supine and Prone Colon Registration Using Quasi-Conformal Mapping

In virtual colonoscopy, CT scans are typically acquired with the patient in both supine (facing up) and prone (facing down) positions. The registration of these two scans is desirable so that the user can clarify situations or confirm polyp findings at a location in one scan with the same location in the other, thereby improving polyp detection rates and reducing false positives. However, this supine-prone registration is challenging because of the substantial distortions in the colon shape due to the patients change in position. We present an efficient algorithm and framework for performing this registration through the use of conformal geometry to guarantee that the registration is a diffeomorphism (a one-to-one and onto mapping). The taeniae coli and colon flexures are automatically extracted for each supine and prone surface, employing the colon geometry. The two colon surfaces are then divided into several segments using the flexures, and each segment is cut along a taenia coli and conformally flattened to the rectangular domain using holomorphic differentials. The mean curvature is color encoded as texture images, from which feature points are automatically detected using graph cut segmentation, mathematic morphological operations, and principal component analysis. Corresponding feature points are found between supine and prone and are used to adjust the conformal flattening to be quasi-conformal, such that the features become aligned. We present multiple methods of visualizing our results, including 2D flattened rendering, corresponding 3D endoluminal views, and rendering of distortion measurements. We demonstrate the efficiency and efficacy of our registration method by illustrating matched views on both the 2D flattened colon images and in the 3D volume rendered colon endoluminal view. We analytically evaluate the correctness of the results by measuring the distance between features on the registered colons.

Context Preserving Maps of Tubular Structures

When visualizing tubular 3D structures, external representations are often used for guidance and display, and such views in 2D can often contain occlusions. Virtual dissection methods have been proposed where the entire 3D structure can be mapped to the 2D plane, though these will lose context by straightening curved sections. We present a new method of creating maps of 3D tubular structures that yield a succinct view while preserving the overall geometric structure. Given a dominant view plane for the structure, its curve skeleton is first projected to a 2D skeleton. This 2D skeleton is adjusted to account for distortions in length, modified to remove intersections, and optimized to preserve the shape of the original 3D skeleton. Based on this shaped 2D skeleton, a boundary for the map of the object is obtained based on a slicing path through the structure and the radius around the skeleton. The sliced structure is conformally mapped to a rectangle and then deformed via harmonic mapping to match the boundary placement. This flattened map preserves the general geometric context of a 3D object in a 2D display, and rendering of this flattened map can be accomplished using volumetric ray casting. We have evaluated our method on real datasets of human colon models.

Planar Visualization of Treelike Structures

We present a novel method to create planar visualizations of treelike structures (e.g., blood vessels and airway trees) where the shape of the object is well preserved, allowing for easy recognition by users familiar with the structures. Based on the extracted skeleton within the treelike object, a radial planar embedding is first obtained such that there are no self-intersections of the skeleton which would have resulted in occlusions in the final view. An optimization procedure which adjusts the angular positions of the skeleton nodes is then used to reconstruct the shape as closely as possible to the original, according to a specified view plane, which thus preserves the global geometric context of the object. Using this shape recovered embedded skeleton, the object surface is then flattened to the plane without occlusions using harmonic mapping. The boundary of the mesh is adjusted during the flattening step to account for regions where the mesh is stretched over concavities. This parameterized surface can then be used either as a map for guidance during endoluminal navigation or directly for interrogation and decision making. Depth cues are provided with a grayscale border to aid in shape understanding. Examples are presented using bronchial trees, cranial and lower limb blood vessels, and upper aorta datasets, and the results are evaluated quantitatively and with a user study.

Virtually assisted optical colonoscopy

We present a set of tools used to enhance the optical colonoscopy procedure in a novel manner with the aim of improving both the accuracy and efficiency of this procedure. In order to better present the colon information to the gastroenterologist performing a conventional (optical) colonoscopy, we undistort the radial distortion of the fisheye view of the colonoscope. The radial distortion is modeled with a function that converts the fisheye view to the perspective view, where the shape and size of polyps can be more readily observed. The conversion, accelerated on the graphics processing unit and running in real-time, calculates the corresponding position in the fisheye view of each pixel on the perspective image. We also merge our previous work in computer-aided polyp detection for virtual colonoscopy into the optical colonoscopy environment. The physical colonoscope path in the optical colonoscopy is approximated with the hugging corner shortest path, which is correlated with the centerline in the virtual colonoscopy. With the estimated distance that the colonoscope has been inserted, we are able to provide the gastroenterologist with visual cues along the observation path as to the location of possible polyps found by the detection process. In order to present the information to the gastroenterologist in a non-intrusive manner, we have developed a friendly user interface to enhance the optical colonoscopy without being cumbersome, distracting, or resulting in a more lackadaisical inspection by the gastroenterologist.

Corresponding Supine and Prone Colon Visualization Using Eigenfunction Analysis and Fold Modeling

We present a method for registration and visualization of corresponding supine and prone virtual colonoscopy scans based on eigenfunction analysis and fold modeling. In virtual colonoscopy, CT scans are acquired with the patient in two positions, and their registration is desirable so that physicians can corroborate findings between scans. Our algorithm performs this registration efficiently through the use of Fiedler vector representation (the second eigenfunction of the Laplace-Beltrami operator). This representation is employed to first perform global registration of the two colon positions. The registration is then locally refined using the haustral folds, which are automatically segmented using the 3D level sets of the Fiedler vector. The use of Fiedler vectors and the segmented folds presents a precise way of visualizing corresponding regions across datasets and visual modalities. We present multiple methods of visualizing the results, including 2D flattened rendering and the corresponding 3D endoluminal views. The precise fold modeling is used to automatically find a suitable cut for the 2D flattening, which provides a less distorted visualization. Our approach is robust, and we demonstrate its efficiency and efficacy by showing matched views on both the 2D flattened colons and in the 3D endoluminal view. We analytically evaluate the results by measuring the distance between features on the registered colons, and we also assess our fold segmentation against 20 manually labeled datasets. We have compared our results analytically to previous methods, and have found our method to achieve superior results. We also prove the hot spots conjecture for modeling cylindrical topology using Fiedler vector representation, which allows our approach to be used for general cylindrical geometry modeling and feature extraction.

Conformal geometry based supine and prone colon registration

In virtual colonoscopy, CT scans are typically acquired with the patient in both supine and prone positions. The registration of these two scans is desirable so that the physician can clarify situations or confirm polyp findings at a location in one scan with the same location in the other, thereby improving polyp detection rates and reducing false positives. However, this supine-prone registration is challenging because of the substantial distortions in the colon shape due to the patients position shifting. We present an efficient algorithm and framework for performing this registration through the use of conformal geometry to guarantee the registration is a diffeomorphism. The colon surface is conformally flattened to a rectangle using holomorphic differentials. The flattened domains of supine and prone are aligned by the harmonic map with feature correspondence constraints. We demonstrate the efficiency and efficacy of our method by measuring the distance between features on the registered colons.

Crowdsourcing for identification of polyp-free segments in virtual colonoscopy videos

Virtual colonoscopy (VC) allows a physician to virtually navigate within a reconstructed 3D colon model searching for colorectal polyps. Though VC is widely recognized as a highly sensitive and specific test for identifying polyps, one limitation is the reading time, which can take over 30 minutes per patient. Large amounts of the colon are often devoid of polyps, and a way of identifying these polyp-free segments could be of valuable use in reducing the required reading time for the interrogating radiologist. To this end, we have tested the ability of the collective crowd intelligence of non-expert workers to identify polyp candidates and polyp-free regions. We presented twenty short videos flying through a segment of a virtual colon to each worker, and the crowd was asked to determine whether or not a possible polyp was observed within that video segment. We evaluated our framework on Amazon Mechanical Turk and found that the crowd was able to achieve a sensitivity of 80.0% and specificity of 86.5% in identifying video segments which contained a clinically proven polyp. Since each polyp appeared in multiple consecutive segments, all polyps were in fact identified. Using the crowd results as a first pass, 80% of the video segments could in theory be skipped by the radiologist, equating to a significant time savings and enabling more VC examinations to be performed.

Computer-aided detection of colonic polyps using volume rendering

This work utilizes a novel pipeline for the computer-aided detection (CAD) of colonic polyps, assisting radiologists in locating polyps when using a virtual colonoscopy system. Our CAD pipeline automatically detects polyps while reducing the number of false positives (FPs). It integrates volume rendering and conformal colon flattening with texture and shape analysis. The colon is first digitally cleansed, segmented, and extracted from the CT dataset of the abdomen. The colon surface is then mapped to a 2D rectangle using conformal mapping. Using this colon flattening method, the CAD problem is converted from 3D into 2D. The flattened image is rendered using a direct volume rendering of the 3D colon dataset with a translucent transfer function. Suspicious polyps are detected by applying a clustering method on the 2D volume rendered image. The FPs are reduced by analyzing shape and texture features of the suspicious areas detected by the clustering step. Compared with shape-based methods, ours is much faster and much more efficient as it avoids computing curvature and other shape parameters for the whole colon wall. We tested our method with 178 datasets and found it to be 100% sensitive to adenomatous polyps with a low rate of FPs. The CAD results are seamlessly integrated into a virtual colonoscopy system, providing the radiologists with visual cues and likelihood indicators of areas likely to contain polyps, and allowing them to quickly inspect the suspicious areas and further exploit the flattened colon view for easy navigation and bookmark placement.

Crowd-assisted polyp annotation of virtual colonoscopy videos

Virtual colonoscopy (VC) allows a radiologist to navigate through a 3D colon model reconstructed from a computed tomography scan of the abdomen, looking for polyps, the precursors of colon cancer. Polyps are seen as protrusions on the colon wall and haustral folds, visible in the VC y-through videos. A complete review of the colon surface requires full navigation from the rectum to the cecum in antegrade and retrograde directions, which is a tedious task that takes an average of 30 minutes. Crowdsourcing is a technique for non-expert users to perform certain tasks, such as image or video annotation. In this work, we use crowdsourcing for the examination of complete VC y-through videos for polyp annotation by non-experts. The motivation for this is to potentially help the radiologist reach a diagnosis in a shorter period of time, and provide a stronger confirmation of the eventual diagnosis. The crowdsourcing interface includes an interactive tool for the crowd to annotate suspected polyps in the video with an enclosing box. Using our work flow, we achieve an overall polyps-per-patient sensitivity of 87.88% (95.65% for polyps ≥5mm and 70% for polyps <5mm). We also demonstrate the efficacy and effectiveness of a non-expert user in detecting and annotating polyps and discuss their possibility in aiding radiologists in VC examinations.

Automatic speed and direction control along constrained navigation paths

For many virtual reality applications, a pre-calculated fly-through path is the de facto standard navigation method. Such a path is convenient for users and ensures coverage of critical areas throughout the scene. Traditional applications use constant camera speed, allow for fully user-controlled manual speed adjustment, or use automatic speed adjustment based on heuristics from the scene. We introduce two novel methods for constrained path navigation and exploration in virtual environments which rely on the natural orientation of the users head during scene exploration. Utilizing head tracking to obtain the users area of focus, we perform automatic camera speed adjustment to allow for natural off-axis scene examination. We expand this to include automatic camera navigation along the pre-computed path, abrogating the need for any navigational inputs from the user. Our techniques are applicable for any scene with a pre-computed navigation path, including medical applications such as virtual colonoscopy, coronary fly-through, or virtual angioscopy, and graph navigation. We compare the traditional methods (constant speed and manual speed adjustment) and our two methods (automatic speed adjustment and automatic speed/direction control) to determine the effect of speed adjustment on system usability, mental load, performance, and user accuracy. Through this evaluation we observe the effect of automatic speed adjustment compared to traditional methods. We observed no negative impact from automatic navigation, and the users performed as well as with the manual navigation.