Chunyu Gao

A widget framework for augmented interaction in SCAPE

We have previously developed a collaborative infrastructure called SCAPE - an acronym for Stereoscopic Collaboration in Augmented and Projective Environments - that integrates the traditionally separate paradigms of virtual and augmented reality. In this paper, we extend SCAPE by formalizing its underlying mathematical framework and detailing three augmented Widgets constructed via this framework: CoCylinder, Magnifier, and CoCube. These devices promote intuitive ways of selecting, examining, and sharing synthetic objects, and retrieving associated documentary text. Finally we present a testbed application to showcase SCAPEs capabilities for interaction in large, augmented virtual environments.

Engineering of head-mounted projective displays

Head-mounted projective displays (HMPDs) are a novel type of head-mounted display. A HMPD consists of a miniature projection lens mounted upon the users head and retroreflective sheeting material placed strategically in the environment. First, the imaging concept of a HMPD is reviewed and its potential advantages and disadvantages are discussed. The design and a bench prototype implementation are then presented. Finally, the effects of retroreflective materials on the imaging properties and the optical properties of HMPDs are comprehensively investigated.

Scape: supporting stereoscopic collaboration in augmented and projective environments

Our collaborative augmented reality system for multiple users simultaneously creates outside-in workbench views and inside-out life-size walkthrough views, bridging the virtual and augmented reality paradigms. Here we introduce a collaborative infrastructure - stereoscopic collaboration in augmented and projective environments(Scape). Scape seamlessly merges the paradigm of virtual reality with that of augmented reality in a single system and provides multimodality interface devices and unique widget interfaces to facilitate collaborative work in 3D environments. A custom-designed cross-platform application-programming interface (API) allows high and medium-level controls over the Scape workspace to enable augmented interaction and collaboration.

A polarized head-mounted projective display

The lack of image brightness is a common problem in optical see-through head-mounted displays (OST-HMD) where a beamsplitter is required to combine views from HMD image source and the direct-view of a real world scene. This problem is further aggregated in a head-mounted projective display (HMPD) due to the fact that light passes through the beamsplitter multiple times. In this paper, we present a novel design of an ultra-bright polarized head-mounted projective display (p-HMPD). The image brightness observed by a viewer theoretically is four-times brighter than existing designs. We further demonstrate a design with currently available technology that leads to a display in which the observed image is significantly brighter than existing designs. Finally, experimental results from a bench setup are presented.

Design of a bright polarized head-mounted projection display

In optical see-through head-mounted displays, it has been a common challenge that the displayed image lacks brightness and contrast compared with the direct view of a real-world scene. Consequently, such displays are usually used in dimmed lighting conditions, which limits the feasibility of applying such information displays outdoors or in scenarios where well-lit environments, such as in operation rooms, are required. The lack of image brightness is aggravated in the design of a see-through head-mounted projection display (HMPD). For instance, the overall flux transfer efficiency of existing HMPD designs is less than 10%. The design of a polarized head-mounted projection display (p-HMPD) is presented. The images of a p-HMPD system can potentially be three times brighter than those in existing HMPD designs. It is further demonstrated that the p-HMPD design is able to dramatically improve image brightness, contrast, and color vividness with experimental results. Finally, the design of a compact optical system and helmet prototype is described.

A Refractive Camera for Acquiring Stereo and Super-resolution Images

We propose a novel depth sensing system composed of a single camera, and a transparent plate which is placed in front of the camera and rotates about the optical axis of the camera. The camera takes a sequence of images as the plate rotates, which provide the equivalent of a large number of stereo pairs. Compared with conventional multi-camera stereo systems, the use of a single camera for capturing stereo pairs helps improve the accuracy of detecting correspondences. The availability of the large number of stereo pairs also reduces matching ambiguities, even for objects with low texture. By using both the images and the estimated depth map, we show that the proposed system is also capable of generating super-resolution images. Experimental results on reconstructing 3D structures and recovering highresolution images are presented.

Using a head-mounted projective display in interactive augmented environments

Head-mounted projective displays (HMPD) have been recently proposed as an alternative to conventional eyepiece-type head-mounted displays (HMDs). HMPDs consist of a pair of miniature projection lenses, beamsplitters, and displays mounted on the helmet and retro-reflective sheeting materials placed strategically in the environment. The HMPD technology is first reviewed briefly, which includes its features and capabilities and a comparison with conventional visualization techniques, as well as our recent implementation of a compact HMPD prototype. Then we present some preliminary findings on retro-reflective materials and discuss a framework for collaborative AR environments, which supports at least three modes of collaboration: interactive local collaboration in an AR environment, passive distant collaboration, and interactive distant collaboration. Finally, two preliminary application examples of the HMPD technology for interactive collaboration in augmented environments are included, which demonstrate some of the HMPD characteristics and embody the framework for distant collaboration.

Single camera stereo using planar parallel plate

A system of using a planar parallel plate to achieve single camera stereo has been proposed by Nishimoto and Shirai (Feb. 1987). Their work was based on an assumption that the optical axis of the camera was equally displaced by a tilted planar plate with two fixed tilt angles. Such assumption is invalid in general. We propose a general framework to more accurately model such a single camera system using a planar parallel plate and to deal with arbitrary orientation of the plate. Our model has no limitation on the FOV of the camera. The proposed framework includes: a mathematical formulation to calculate the displacement of scene points; a plate calibration method to determine the plate intrinsic parameters as well as its extrinsic pose, and a generalized correspondence method which is capable of dealing with arbitrary number of images captured from different plate poses. Experimental result is included to validate the proposed framework.

Calibration of a head-mounted projective display for augmented reality systems

In augmented reality (AR) applications, registering a virtual object with its real counterpart accurately and comfortably is one of the basic and challenging issues in the sense that the size, depth, geometry, as well as physical attributes of the virtual objects have to be rendered precisely relative to a physical reference, which is well known as the calibration or registration problem. This paper presents a systematic calibration process to address static registration in a custom-designed augmented reality system, which is based upon the recent advancement of head-mounted projective display (HMPD) technology. Following a concise review of the HMPD concept and system configuration, we present in detail a computational model for system calibration, describe the calibration procedures to obtain estimations of the unknown transformations, and include the calibration results, evaluation experiments and results.

Design Analysis of a High-Resolution Panoramic Camera Using Conventional Imagers and a Mirror Pyramid

Wide field of view (FOV) and high-resolution image acquisition is highly desirable in many vision-based applications. Several systems have reported the use of reflections off mirror pyramids to capture high-resolution, single-viewpoint, and wide-FOV images. Using a dual mirror pyramid (DMP) panoramic camera as an example, in this paper, we examine how the pyramid geometry, and the selection and placement of imager clusters can be optimized to maximize the overall panoramic FOV, sensor utilization efficiency, and image uniformity. The analysis can be generalized and applied to other pyramid-based designs