Gentaro Hirota

Superior augmented reality registration by integrating landmark tracking and magnetic tracking

Accurate registration between real and virtual objects is crucial for augmented reality applications. Existing tracking methods are individually inadequate: magnetic trackers are inaccurate, mechanical trackers are cumbersome, and vision-based trackers are computationally problematic. We present a hybrid tracking method that combines the accuracy of vision-based tracking with the robustness of magnetic tracking without compromising real-time performance or usability. We demonstrate excellent registration in three sample applications. CR

Technologies for augmented reality systems: realizing ultrasound-guided needle biopsies

We present a real-time stereoscopic video-see-through augmented reality (AR) system applied to the medical procedure known as ultrasound-guided needle biopsy of the breast. The AR system was used by a physician during procedures on breast models and during non-invasive examinations of human subjects. The system merges rendered live ultrasound data and geometric elements with stereo images of the patient acquired through head-mounted video cameras and presents these merged images to the physician in a head-mounted display. The physician sees a volume visualization of the ultrasound data directly under the ultrasound probe, properly registered within the patient and with the biopsy needle. Using this system, a physician successfully guided a needle into an artificial tumor within a training phantom of a human breast. We discuss the construction of the AR system and the issues and decisions which led to the system architecture and the design of the video see-through head-mounted display. We designed methods to properly resolve occlusion of the real and synthetic image elements. We developed techniques for realtime volume visualization of timeand position-varying ultrasound data. We devised a hybrid tracking system which achieves improved registration of synthetic and real imagery and we improved on previous techniques for calibration of a magnetic tracker. CR

Augmented reality guidance for needle biopsies: an initial randomized, controlled trial in phantoms.

We report the results of a randomized, controlled trial to compare the accuracy of standard ultrasound-guided needle biopsy to biopsies performed using a 3D Augmented Reality (AR) guidance system. A board-certified radiologist conducted 50 core biopsies of breast phantoms, with biopsies randomly assigned to one of the methods in blocks of five biopsies each. The raw ultrasound data from each biopsy was recorded. Another board-certified radiologist, blinded to the actual biopsy guidance mechanism, evaluated the ultrasound recordings and determined the distance of the biopsy from the ideal position. A repeated measures analysis of variance indicated that the head-mounted display method led to a statistically significantly smaller mean deviation from the desired target than did the standard display method (2.48 mm for control versus 1.62 mm for augmented reality, p<0.02). This result suggests that AR systems can offer improved accuracy over traditional biopsy guidance methods.

Fast volume-preserving free form deformation using multi-level optimization

We present an efficient algorithm for preserving the total volume of a solids wdergoing free-form deformation using discrete level-of-detail representations. Given the boundary representation of a solid and user-specified deformation, the algorithm computes the new node positions of the de,formation lattice, while minimizing the elastic energy subject to the volumepreserving criterion. During each iteration, a non-linear optimizer computes the volume deviation and its derivatives based on a triangular approximation, which requires a finely tessellated mesh to achieve the desired accuracy. To reduce the computational cost, we exploit the multi-level representations of the boundary sugaces to greatly accelerate the performance of the non-linear optimizer. This technique also provides interactive response by progressively refining the solution. Furthermore, it is generally applicable to lattice-based free-form deformation and its variants. Our implementation has been applied to several compkx solids. We have been able to achieve an order of magnitude performance improvement over the conventional methods.

Towards Performing Ultrasound-Guided Needle Biopsies from within a Head-Mounted Display

Augmented reality is applied to ultrasound-guided needle biopsy of the human breast. In a tracked stereoscopic head-mounted display, a physician sees the ultrasound imagery “emanating” from the transducer, properly registered with the patient and the biopsy needle. A physician has successfully used the system to guide a needle into a synthetic tumor within a breast phantom and examine a human patient in preparation for a cyst aspiration.

An implicit finite element method for elastic solids in contact

Focuses on the simulation of mechanical contact between nonlinearly elastic objects such as the components of the human body. The computation of the reaction forces that act on the contact surfaces (contact forces) is the key for designing a reliable contact handling algorithm. In traditional methods, contact forces are often defined as discontinuous functions of deformation, which leads to poor convergence characteristics. This problem becomes especially serious in areas with complicated self contact such as skin folds. We introduce a novel penalty finite element formulation based on the concept of material depth, the distance between a particle inside an object and the objects boundary. By linearly interpolating pre-computed material depths at node points, contact forces can be analytically integrated over contact surfaces without increasing the computational cost. The continuity achieved by this formulation supports an efficient and reliable solution of the nonlinear system. This algorithm is implemented as part of our implicit finite element program for static, quasistatic and dynamic analysis of nonlinear viscoelastic solids. We demonstrate its effectiveness on an animation showing realistic effects such as folding skin and sliding contacts of the tissues involved in knee flexion. The finite element model of the leg and its internal structures was derived from the Visible Human data set.

Augmented Reality Guidance for Needle Biopsies: A Randomized, Controlled Trial in Phantoms

We report the results of a randomized, controlled tnal to compare the accuracy of standard ultrasound-guided needle biopsy to biopsies performed using a 3D Augmented Reality (AR) guidance system. Fifty core biopsies of breast phantoms were conducted by a board-certified radiologist, with each set of five biopsies randomly assigned to one of the methods. The raw ultrasound data from each biopsy was recorded. Another board-certified radiologist, blinded to the actual biopsy guidance mechanism, evaluated the ultrasound recordings and determined the distance of the biopsy from the ideal position. A repeated measures analysis of variance indicated that the head-mounted display method led to a statistically significantly smaller mean deviation from the desired target than did the CRT display method. (2.48mm for control versus 1.62mm for augmented reality, p < 0.02). This result suggests that AR systems can offer improved accuracy over traditional biopsy guidance methods.

An improved finite element contact model for anatomical simulations

This work focuses on the simulation of mechanical contact between nonlinearly elastic objects, such as the components of the human body. In traditional methods, contact forces are often defined as discontinuous functions of deformations, which leads to poor convergence characteristics and high-frequency noises. We introduce a novel penalty method for finite-element simulation based on the concept of material depth, which is the distance between a particle inside an object and the object’s boundary. By linearly interpolating precomputed material depths at node points, contact forces can be analytically integrated over contact surfaces without raising the computational cost. The continuity achieved by this formulation reduces oscillation and artificial acceleration, resulting in a more reliable simulation algorithm.

Dynamic virtual convergence for video see-through head-mounted displays: maintaining maximum stereo overlap throughout a close-range work space

We present a technique that allows users of video see-through head-mounted displays to work at close range without the typical loss of stereo perception due to reduced nasal side stereo overlap in most of todays commercial HMDs. Our technique dynamically selects parts of the imaging frustums acquired by wide-angle head-mounted cameras and re-projects them for the narrower field-of-view displays. In addition to dynamically maintaining maximum stereo overlap for objects at a heuristically estimated working distance, it also reduces the accommodation-vergence conflict, at the expense of a newly introduced disparity-vergence conflict. We describe the hardware (assembled from commercial components) and software implementation of our system and report on our experience while using this technique within two different AR applications. (Plus color plates).

Fast volume-preserving free-form deformation using multi-level optimization

Abstract We present a fast algorithm for preserving the total volume of a solid undergoing free-form deformation using level-of-detail representations. Given the boundary representation of a solid and user-specified deformation, the algorithm computes the new node positions of the deformation lattice, while minimizing the elastic energy subject to the volume-preserving criterion. During each iteration, a non-linear optimizer computes the volume deviation and its derivatives based on a triangular approximation, which requires a finely tessellated mesh to achieve the desired accuracy. To reduce the computational cost, we exploit the multi-resolution representations of the boundary surfaces to greatly accelerate the performance of the non-linear optimizer. This technique also provides interactive response by progressively refining the solution. Furthermore, it is generally applicable to lattice-based free-form deformation and its variants. Our implementation has been applied to several complex solids. We have been able to achieve an order of magnitude performance improvement over the conventional methods.