Ferdi Smit

An image-warping VR-architecture: design, implementation and applications

We describe an architecture that provides a programmable display layer in order to allow the execution of custom programs on consecutive display frames. This replaces the default display behavior of repeating application frames until an update is available. The architecture is implemented using a multi-GPU system. We will show three applications of the architecture typical to VR. First, smooth motion is provided by generating intermediate display frames by per-pixel depth-image warping using 3D motion fields. Smooth motion can be beneficial for walk-throughs of large scenes. Second, we implement fine-grained latency reduction at the display frame level using a synchronized prediction of simulation objects and the viewpoint. This improves the average quality and consistency of latency reduction. Third, a crosstalk reduction algorithm for consecutive display frames is implemented, which improves the quality of stereoscopic images.

Three extensions to subtractive crosstalk reduction

Stereo displays suffer from crosstalk, an effect that reduces or even inhibits the viewers ability to correctly fuse stereoscopic images. In this paper, three extensions for improved software crosstalk reduction are introduced. First, we propose a reduction method operating in CIELAB color space to find a perceptually better color match for crosstalk corrected pixels. Second, we introduce a geometry-based reduction method that operates on fused 3D pixels. Finally, a run-time optimization is introduced that avoids the need to process each pixel. We evaluated our CIELAB-based method using the Visible Differences Predictor (VDP). Our results show that we are able to significantly improve crosstalk reduction compared to previously used methods that operate in RGB color space. The combination of our methods provides an improved, real-time software crosstalk reduction framework, applicable to a wider range of scenes, delivering better quality, higher performance, and more flexibility.

Non-Uniform Crosstalk Reduction for Dynamic Scenes

Stereo displays suffer from crosstalk, an effect that reduces or even inhibits the viewers ability to correctly perceive depth. Previous work on software crosstalk reduction focussed on the preprocessing of static scenes which are viewed from a fixed viewpoint. However, in virtual environments scenes are dynamic, and are viewed from various viewpoints in real-time on large display areas. In this paper, three methods are introduced for reducing crosstalk in virtual environments. A non-uniform crosstalk model is described, which can be used to accurately reduce crosstalk on large display areas. In addition, a novel temporal algorithm is used to address the problems that occur when reducing crosstalk in dynamic scenes. This way, high-frequency jitter caused by the erroneous assumption of static scenes can be eliminated. Finally, a perception based metric is developed that allows us to quantify crosstalk. We provide a detailed description of the methods, discuss their tradeoffs, and compare their performance with existing crosstalk reduction methods

The design and implementation of a VR-architecture for smooth motion

We introduce an architecture for smooth motion in virtual environments. The system performs forward depth image warping to produce images at video refresh rates. In addition to color and depth, our 3D warping approach records per-pixel motion information during rendering of the three-dimensional scene. These enhanced depth images are used to perform per-pixel advection, which considers object motion and view changes. Our dual graphics card architecture is able to render the 3D scene at the highest possible frame rate on one graphics card, while doing the depth image warping on a second graphics engine at video refresh rate. This architecture allows us to compensate for visual artifacts, also called motion judder, arising when the rendering frame rate is lower than the video refresh rate. The evaluation of our method shows motion judder can be effectively removed.

A Programmable Display Layer for Virtual Reality System Architectures

Display systems typically operate at a minimum rate of 60 Hz. However, existing VR-architectures generally produce application updates at a lower rate. Consequently, the display is not updated by the application every display frame. This causes a number of undesirable perceptual artifacts. We describe an architecture that provides a programmable display layer (PDL) in order to generate updated display frames. This replaces the default display behavior of repeating application frames until an update is available. We will show three benefits of the architecture typical to VR. First, smooth motion is provided by generating intermediate display frames by per-pixel depth-image warping using 3D motion fields. Smooth motion eliminates various perceptual artifacts due to judder. Second, we implement fine-grained latency reduction at the display frame level using a synchronized prediction of simulation objects and the viewpoint. This improves the average quality and consistency of latency reduction. Third, a crosstalk reduction algorithm for consecutive display frames is implemented, which improves the quality of stereoscopic images. To evaluate the architecture, we compare image quality and latency to that of a classic level-of-detail approach.

Special section: IEEE Virtual Reality (VR) 2009: A shared-scene-graph image-warping architecture for VR: Low latency versus image quality

Designing low end-to-end latency system architectures for virtual reality is still an open and challenging problem. We describe the design, implementation and evaluation of a client-server depth-image warping architecture that updates and displays the scene graph at the refresh rate of the display. Our approach works for scenes consisting of dynamic and interactive objects. The end-to-end latency is minimized as well as smooth object motion generated. However, this comes at the expense of image quality inherent to warping techniques. To improve image quality, we present a novel way of detecting and resolving occlusion errors due to warping. Furthermore, we investigate the use of asynchronous data transfers to increase the architectures performance in a multi-GPU setting. Besides polygonal rendering, we also apply image-warping techniques to iso-surface rendering. Finally, we evaluate the architecture and its design trade-offs by comparing latency and image quality to a conventional rendering system. Our experience with the system confirms that the approach facilitates common interaction tasks such as navigation and object manipulation.

Virtual Environments: Graphtracker: A topology projection invariant optical tracker

In this paper, we describe a new optical tracking algorithm for pose estimation of interaction devices in virtual and augmented reality. Given a 3D model of the interaction device and a number of camera images, the primary difficulty in pose reconstruction is to find the correspondence between 2D image points and 3D model points. Most previous methods solved this problem by the use of stereo correspondence. Once the correspondence problem has been solved, the pose can be estimated by determining the transformation between the 3D point cloud and the model. Our approach is based on the projective invariant topology of graph structures. The topology of a graph structure does not change under projection: in this way we solve the point correspondence problem by a subgraph matching algorithm between the detected 2D image graph and the model graph. In addition to the graph tracking algorithm, we describe a number of related topics. These include a discussion on the counting of topologically different graphs, a theoretical error analysis, and a method for automatically estimating a device model. Finally, we show and discuss experimental results for the position and orientation accuracy of the tracker.

Technical Section: A simulator-based approach to evaluating optical trackers

We describe a software framework to evaluate the performance of model-based optical trackers in virtual environments. The framework can be used to evaluate and compare the performance of different trackers under various conditions, to study the effects of varying intrinsic and extrinsic camera properties, and to study the effects of environmental conditions on tracker performance. The framework consists of a simulator that, given various input conditions, generates a series of images. The input conditions of the framework model important aspects, such as the interaction task, input device geometry, camera properties and occlusion. As a concrete case, we illustrate the usage of the proposed framework for input device tracking in a near-field desktop virtual environment. We compare the performance of an in-house tracker with an ARToolkitPlus-based tracker under a fixed set of conditions. We also show how the framework can be used to assess the quality of various camera placements given a pre-recorded interaction task. Finally, we use the framework to determine the minimum required camera resolution for a desktop, Workbench and CAVE environment, and study the influence of random noise on tracker accuracy. The framework is shown to provide an efficient and simple method to study various conditions affecting optical tracker performance. Furthermore, it can be used as a valuable development tool to aid in the construction of optical trackers.

An Interactive, Multi-Modal Approach to Analysing High-Resolution Image Mass Spectrometry Data

The output resolution of imaging mass spectrometers is increasing rapidly due to advances in engineering and the use of tiling. Imaging-MS data is often displayed as a total-ion-count (TIC) image; however, anatomical structures are not easily identifiable from TIC images. For this purpose, additional high-resolution images that originate from different imaging modalities, such as stained histological data, are preferred. These modalities are most useful when fused; i.e., when the corresponding images are spatially aligned with respect to each other. The viewing and analysis of such data is ideally performed in real-time and at the highest possible resolution, allowing users to interactively query the combination of all fused data at the highest detail. However, proper alignment between modalities and interactively presenting large volumes of data is as of yet a challenge. We present a system for the simultaneous viewing and analysis of high-resolution data from different imaging modalities. Fusion is provided in such a way that interaction in one modality can be mapped to different modalities. For example, anatomical structures can be identified from histological data and their spatial extent mapped to a corresponding region-of-interest in the image MS data, allowing the analysis of its chemical compounds. In turn, the MS data can be analysed and filtered, for example using multi-variate analysis such as PCA, and the result mapped back to structures in other modalities. Level-of-detail, region-of-interest and asynchronous data processing algorithms ensure that the system can be operated interactively at the highest resolution.

A framework for performance evaluation of model-based optical trackers

We describe a software framework to evaluate the performance of model-based optical trackers in virtual environments. The framework can be used to evaluate and compare the performance of different trackers under various conditions, to study the effects of varying intrinsic and extrinsic camera properties, and to study the effects of environmental conditions on tracker performance. The framework consists of a simulator that, given various input conditions, generates a series of images. The input conditions of the framework model important aspects, such as the interaction task, input device geometry, camera properties and occlusion. As a concrete case, we illustrate the usage of the proposed framework for input device tracking in a near-field desktop virtual environment. We compare the performance of an in-house tracker with the ARToolkit tracker under a fixed set of conditions. We also show how the framework can be used to find the optimal camera parameters given a pre-recorded interaction task. Finally, we use the framework to determine the minimum required camera resolution for a desktop, Workbench and CAVE environment. The framework is shown to provide an efficient and simple method to study various conditions affecting optical tracker performance. Furthermore, it can be used as a valuable development tool to aid in the construction of optical trackers.