Cha Lee

Integrating the physical environment into mobile remote collaboration

We describe a framework and prototype implementation for unobtrusive mobile remote collaboration on tasks that involve the physical environment. Our system uses the Augmented Reality paradigm and model-free, markerless visual tracking to facilitate decoupled, live updated views of the environment and world-stabilized annotations while supporting a moving camera and unknown, unprepared environments. In order to evaluate our concept and prototype, we conducted a user study with 48 participants in which a remote expert instructed a local user to operate a mock-up airplane cockpit. Users performed significantly better with our prototype (40.8 tasks completed on average) as well as with static annotations (37.3) than without annotations (28.9). 79% of the users preferred our prototype despite noticeably imperfect tracking.

The role of latency in the validity of AR simulation

It is extremely challenging to run controlled studies comparing multiple Augmented Reality (AR) systems. We use an AR simulation approach, in which a Virtual Reality (VR) system is used to simulate multiple AR systems. To investigate the validity of this approach, in our first experiment we carefully replicated a well-known study by Ellis et al. using our simulator, obtaining comparable results. We include a discussion on general issues we encountered with replicating a prior study. In our second experiment further exploring the validity of AR simulation, we investigated the effects of simulator latency on the results from experiments conducted in an AR simulator. We found simulator latency to have a significant effect on 3D tracing, however there was no interaction between simulator latency and artificial latency. Based on the results from these two experiments, we conclude that simulator latency is not inconsequential in determining task performance. Simulating visual registration is not sufficient to simulate the overall perception of registration errors in an AR system. We also need to keep simulator latency at a minimum. We discuss the impact of these results on the use of the AR simulation approach.

Depth-Fused 3D Imagery on an Immaterial Display

We present an immaterial display that uses a generalized form of depth-fused 3D (DFD) rendering to create unencumbered 3D visuals. To accomplish this result, we demonstrate a DFD display simulator that extends the established depth-fused 3D principle by using screens in arbitrary configurations and from arbitrary viewpoints. The feasibility of the generalized DFD effect is established with a user study using the simulator. Based on these results, we developed a prototype display using one or two immaterial screens to create an unencumbered 3D visual that users can penetrate, examining the potential for direct walk-through and reach-through manipulation of the 3D scene. We evaluate the prototype system in formative and summative user studies and report the tolerance thresholds discovered for both tracking and projector errors.

An immaterial depth-fused 3D display

We present an immaterial display that uses a generalized form of depth-fused 3D (DFD) rendering to create unencumbered 3D visuals. To accomplish this result, we demonstrate a DFD display simulator that extends the established depth-fused 3D principle by using screens in arbitrary configurations and from arbitrary viewpoints. The performance of the generalized DFD effect is established with a user study using the simulator. Based on these results, we developed a prototype display using two immaterial screens to create an unencumbered 3D visual that users can penetrate, enabling the potential for direct walk-through and reach-through manipulation of the 3D scene.

A replication study testing the validity of AR simulation in VR for controlled experiments

It is extremely challenging to run controlled studies comparing multiple Augmented Reality (AR) systems. We use an “AR simulation” approach, in which a Virtual Reality (VR) system is used to simulate multiple AR systems. In order to validate this approach, we carefully replicated a well-known study by Ellis et al. using our simulator, obtaining comparable results.

Examining the equivalence of simulated and real AR on a visual following and identification task

Mixed Reality (MR) simulation, in which a Virtual Reality (VR) system is used to simulate both the real and virtual components of an Augmented Reality (AR) system, has been proposed as a method for evaluating AR systems with greater levels of experimental control. However, factors such as the latency of the MR simulator may impact the validity of experimental results obtained with MR simulation. We present a study evaluating the effects of simulator latency on the equivalence of results from an MR simulator and a real AR system. We designed an AR experiment which required the participants to visually follow a virtual pipe around a small room filled with real targets and to find and identify the targets which were intersected by the pipe. We show that, with a 95% confidence interval, the results from all three simulated AR conditions fall well within one standard deviation of the real AR case.

Evaluating the impact of recovery density on augmented reality tracking

Natural feature tracking systems for augmented reality are highly accurate, but can suffer from lost tracking. When registration is lost, the system must be able to re-localize and recover tracking. Likewise, when a camera is new to a scene, it must be able to perform the related task of localization. Localization and re-localization can only be performed at certain points or when viewing particular objects or parts of the scene with a sufficient number and quality of recognizable features to allow for tracking recovery. We explore how the density of such recovery locations/poses influences the time it takes users to resume tracking. We focus our evaluation on two generalized techniques for localization: keyframe-based and model-based. For the keyframe-based approach we assume a constant collection rate for keyframes. We find that at practical collection rates, the task of localization to a previously acquired keyframe that is shown to the user does not become more time-consuming as the interval between keyframes increases. For a localization approach using model data, we consider a grid of points around the model at which localization is guaranteed to succeed. We find that the user interface is crucial to successful localization. Localization can occur quickly if users do not need to orient themselves to marked localization points. When users are forced to mentally register themselves with a map of the scene, localization quickly becomes impractical as the distance to the next localization point increases. We contend that our results will help future designers of localization techniques to better plan for the effects of their proposed solutions.

A prototype setup to integrate the physical environment into mobile remote collaboration

In the accompanying paper [1], we describe a framework and prototype implementation for unobtrusive mobile remote collaboration on tasks that involve the physical environment. Our system uses model-free, markerless visual tracking to facilitate decoupled, live updated views of the environment and world-stabilized annotations while supporting a moving camera and unknown, unprepared environments. We conducted a user study with 48 participants to evaluate our concept. In this demo, we will present our system prototype and the setup used in the user study: a remote expert instructs a local user to operate a mock-up airplane. Users will be able to try out our interface as well as the two interfaces used as baseline in the study.