Kurtis Keller

Augmented Reality Visualization for Laparoscopic Surgery

We present the design and a prototype implementation of a three-dimensional visualization system to assist with laparoscopic surgical procedures. The system uses 3D visualization, depth extraction from laparoscopic images, and six degree-of-freedom head and laparoscope tracking to display a merged real and synthetic image in the surgeon’s video-see-through head-mounted display. We also introduce a custom design for this display. A digital light projector, a camera, and a conventional laparoscope create a prototype 3D laparoscope that can extract depth and video imagery.

High-Performance Wide-Area Optical Tracking: The HiBall Tracking System

Since the early 1980s, the Tracker Project at the University of North Carolina at Chapel Hill has been working on wide-area head tracking for virtual and augmented environments. Our long-term goal has been to achieve the high performance required for accurate visual simulation throughout our entire laboratory, beyond into the hallways, and eventually even outdoors. In this article, we present results and a complete description of our most recent electro-optical system, the HiBall Tracking System. In particular, we discuss motivation for the geometric configuration and describe the novel optical, mechanical, electronic, and algorithmic aspects that enable unprecedented speed, resolution, accuracy, robustness, and flexibility.

The HiBall Tracker: high-performance wide-area tracking for virtual and augmented environments

Our HiBall Tracking System generates over 2000 head-pose estimates per second with less than one millisecond of latency, and less than 0.5 millimeters and 0.02 degrees of position and orientation noise, everywhere in a 4.5 by 8.5 meter room. The system is remarkably responsive and robust, enabling VR applications and experiments that previously would have been difficult or even impossible. Previously we published descriptions of only the Kalman filter-based software approach that we call Single-Constraint-at-a-Time tracking. In this paper we describe the complete tracking system, including the novel optical, mechanical, electrical, and algorithmic aspects that enable the unparalleled performance.

Augmented reality guidance for needle biopsies: an initial randomized, controlled trial in phantoms.

We report the results of a randomized, controlled trial to compare the accuracy of standard ultrasound-guided needle biopsy to biopsies performed using a 3D Augmented Reality (AR) guidance system. A board-certified radiologist conducted 50 core biopsies of breast phantoms, with biopsies randomly assigned to one of the methods in blocks of five biopsies each. The raw ultrasound data from each biopsy was recorded. Another board-certified radiologist, blinded to the actual biopsy guidance mechanism, evaluated the ultrasound recordings and determined the distance of the biopsy from the ideal position. A repeated measures analysis of variance indicated that the head-mounted display method led to a statistically significantly smaller mean deviation from the desired target than did the standard display method (2.48 mm for control versus 1.62 mm for augmented reality, p<0.02). This result suggests that AR systems can offer improved accuracy over traditional biopsy guidance methods.

Video see-through design for merging of real and virtual environments

A head mounted display (HMD) that is optically opaque with respect to the outer world can be provided with see-through capability by mounting video cameras to the outside of the helmet. Stereoscopic views of the physical space surrounding the wearer are captured by the video cameras, and are projected to the display screens inside the HMD. The conditions necessary for mounting the cameras to a pre-existing, non-see-through HMD are described, and the advantages and disadvantages of different mounting schemes are discussed, together with the design of the video cameras lenses to be used with this specific application.<<ETX>>

Thin-foil magnetic force system for high-numerical-aperture microscopy

Forces play a key role in a wide range of biological phenomena from single-protein conformational dynamics to transcription and cell division, to name a few. The majority of existing microbiological force application methods can be divided into two categories: those that can apply relatively high forces through the use of a physical connection to a probe and those that apply smaller forces with a detached probe. Existing magnetic manipulators utilizing high fields and high field gradients have been able to reduce this gap in maximum applicable force, but the size of such devices has limited their use in applications where high force and high-numerical-aperture (NA) microscopy must be combined. We have developed a magnetic manipulation system that is capable of applying forces in excess of 700 pN on a 1 mum paramagnetic particle and 13 nN on a 4.5 mum paramagnetic particle, forces over the full 4pi sr, and a bandwidth in excess of 3 kHz while remaining compatible with a commercially available high-NA microscope objective. Our system design separates the pole tips from the flux coils so that the magnetic-field geometry at the sample is determined by removable thin-foil pole plates, allowing easy change from experiment to experiment. In addition, we have combined the magnetic manipulator with a feedback-enhanced, high-resolution (2.4 nm), high-bandwidth (10 kHz), long-range (100 mum xyz range) laser tracking system. We demonstrate the usefulness of this system in a study of the role of forces in higher-order chromosome structure and function.

Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources

We present a novel design for an optical see-through augmented reality display that offers a wide field of view and supports a compact form factor approaching ordinary eyeglasses. Instead of conventional optics, our design uses only two simple hardware components: an LCD panel and an array of point light sources (implemented as an edge-lit, etched acrylic sheet) placed directly in front of the eye, out of focus. We code the point light sources through the LCD to form miniature see-through projectors. A virtual aperture encoded on the LCD allows the projectors to be tiled, creating an arbitrarily wide field of view. Software rearranges the target augmented image into tiled sub-images sent to the display, which appear as the correct image when observed out of the viewers accommodation range. We evaluate the design space of tiled point light projectors with an emphasis on increasing spatial resolution through the use of eye tracking. We demonstrate feasibility through software simulations and a real-time prototype display that offers a 110° diagonal field of view in the form factor of large glasses and discuss remaining challenges to constructing a practical display.

Simulation-based design and rapid prototyping of a parallax-free, orthoscopic video see-through head-mounted display

We built a video see-through head-mounted display with zero eye offset from commercial components and a mount fabricated via rapid prototyping. The orthoscopic HMDs layout was created and optimized with a software simulator. We describe simulator and HMD design, we show the HMD in use and demonstrate zero parallax.

Augmented Reality Guidance for Needle Biopsies: A Randomized, Controlled Trial in Phantoms

We report the results of a randomized, controlled tnal to compare the accuracy of standard ultrasound-guided needle biopsy to biopsies performed using a 3D Augmented Reality (AR) guidance system. Fifty core biopsies of breast phantoms were conducted by a board-certified radiologist, with each set of five biopsies randomly assigned to one of the methods. The raw ultrasound data from each biopsy was recorded. Another board-certified radiologist, blinded to the actual biopsy guidance mechanism, evaluated the ultrasound recordings and determined the distance of the biopsy from the ideal position. A repeated measures analysis of variance indicated that the head-mounted display method led to a statistically significantly smaller mean deviation from the desired target than did the CRT display method. (2.48mm for control versus 1.62mm for augmented reality, p < 0.02). This result suggests that AR systems can offer improved accuracy over traditional biopsy guidance methods.

Three-dimensional force microscope: A nanometric optical tracking and magnetic manipulation system for the biomedical sciences

We report here the development of a three-dimensional (3D) magnetic force microscope for applying forces to and measuring responses of biological systems and materials. This instrument combines a conventional optical microscope with a free-floating or specifically bound magnetic bead used as a mechanical probe. Forces can be applied by the bead to microscopic structures of interest (specimens), while the reaction displacement of the bead is measured. This enables 3D mechanical manipulations and measurements to be performed on specimens in fluids. Force is generated by the magnetically permeable bead in reaction to fields produced by external electromagnets. The displacement is measured by interferometry using forward light scattered by the bead from a focused laser beam. The far-field interference pattern is imaged on a quadrant photodetector from which the 3D displacement can be computed over a limited range about the focal point. The bead and specimen are mounted on a 3D translation stage and feedback t...