Carmine Elvezio

Virtual Replicas for Remote Assistance in Virtual and Augmented Reality

In many complex tasks, a remote subject-matter expert may need to assist a local user to guide actions on objects in the local users environment. However, effective spatial referencing and action demonstration in a remote physical environment can be challenging. We introduce two approaches that use Virtual Reality (VR) or Augmented Reality (AR) for the remote expert, and AR for the local user, each wearing a stereo head-worn display. Both approaches allow the expert to create and manipulate virtual replicas of physical objects in the local environment to refer to parts of those physical objects and to indicate actions on them. This can be especially useful for parts that are occluded or difficult to access. In one approach, the expert points in 3D to portions of virtual replicas to annotate them. In another approach, the expert demonstrates actions in 3D by manipulating virtual replicas, supported by constraints and annotations. We performed a user study of a 6DOF alignment task, a key operation in many physical task domains, comparing both approaches to an approach in which the expert uses a 2D tablet-based drawing system similar to ones developed for prior work on remote assistance. The study showed the 3D demonstration approach to be faster than the others. In addition, the 3D pointing approach was faster than the 2D tablet in the case of a highly trained expert.

ParaFrustum: visualization techniques for guiding a user to a constrained set of viewing positions and orientations

Many tasks in real or virtual environments require users to view a target object or location from one of a set of strategic viewpoints to see it in context, avoid occlusions, or view it at an appropriate angle or distance. We introduce ParaFrustum, a geometric construct that represents this set of strategic viewpoints and viewing directions. ParaFrustum is inspired by the look-from and look-at points of a computer graphics camera specification, which precisely delineate a location for the camera and a direction in which it looks. We generalize this approach by defining a ParaFrustum in terms of a look-from volume and a look-at volume, which establish constraints on a range of acceptable locations for the users eyes and a range of acceptable angles in which the users head can be oriented. Providing tolerance in the allowable viewing positions and directions avoids burdening the user with the need to assume a tightly constrained 6DoF pose when it is not required by the task. We describe two visualization techniques for virtual or augmented reality that guide a user to assume one of the poses defined by a ParaFrustum, and present the results of a user study measuring the performance of these techniques. The study shows that the constraints of a tightly constrained ParaFrustum (e.g., approximating a conventional camera frustum) require significantly more time to satisfy than those of a loosely constrained one. The study also reveals interesting differences in participant trajectories in response to the two techniques.

A user interface for assistive grasping

There has been considerable interest in producing grasping platforms using non-invasive, low bandwidth brain computer interfaces(BCIs). Most of this work focuses on low level control of simple hands. Using complex hands improves the versatility of a grasping platform at the cost of increasing its complexity. In order to control more complex hands with these low bandwidth signals, we need to use higher level abstractions. Here, we present a user interface which allows the user to combine the speed and convenience of offline preplanned grasps with the versatility of an online planner. This system incorporates a database of pre-planned grasps with the ability to refine these grasps using an online planner designed for arbitrarily complex hands. Only four commands are necessary to control the entire grasping pipeline, allowing us to use a low cost, noninvasive commercial BCI device to produce robust grasps that reflect user intent. We demonstrate the efficacy of this system with results from five subjects and present results using this system to grasp unknown objects.

Travel in large-scale head-worn VR: Pre-oriented teleportation with WIMs and previews

We demonstrate an interaction technique that allows a user to point at a world-in-miniature representation of a city-scale virtual environment and perform efficient and precise teleportation by pre-orienting an avatar. A preview of the post-teleport view of the full-scale virtual environment updates interactively as the user adjusts the position, yaw, and pitch of the avatars head with a pair of 6DoF-tracked controllers. We describe design decisions and contrast with alternative approaches to virtual travel.

Providing Assistance for Orienting 3D Objects Using Monocular Eyewear

Many tasks require that a user rotate an object to match a specific orientation in an external coordinate system. This includes tasks in which one object must be oriented relative to a second prior to assembly and tasks in which objects must be held in specific ways to inspect them. Research has investigated guidance mechanisms for some 6DOF tasks, using wide--field-of-view, stereoscopic virtual and augmented reality head-worn displays (HWDs). However, there has been relatively little work directed toward smaller field-of-view lightweight monoscopic HWDs, such as Google Glass, which may remain more comfortable and less intrusive than stereoscopic HWDs in the near future. We have designed and implemented a novel visualization approach and three additional visualizations representing different paradigms for guiding unconstrained manual 3DOF rotation, targeting these monoscopic HWDs. We describe our exploration of these paradigms and present the results of a user study evaluating the relative performance of the visualizations and showing the advantages of our new approach.

[POSTER] Interactive Visualizations for Monoscopic Eyewear to Assist in Manually Orienting Objects in 3D

Assembly or repair tasks often require objects to be held in specific orientations to view or fit together. Research has addressed the use of AR to assist in these tasks, delivered as registered overlaid graphics on stereoscopic head-worn displays. In contrast, we are interested in using monoscopic head-worn displays, such as Google Glass. To accommodate their small monoscopic field of view, off center from the users line of sight, we are exploring alternatives to registered overlays. We describe four interactive rotation guidance visualizations for tracked objects intended for these displays.

Mercury: A Messaging Framework for Modular UI Components

In recent years, the entity--component--system pattern has become a fundamental feature of the software architectures of game-development environments such as Unity and Unreal, which are used extensively in developing 3D user interfaces. In these systems, UI components typically respond to events, requiring programmers to write application-specific callback functions. In some cases, components are organized in a hierarchy that is used to propagate events among vertically connected components. When components need to communicate horizontally, programmers must connect those components manually and register/unregister events as needed. Moreover, events and callback signatures may be incompatible, making modular UIs cumbersome to build and share within or across applications. To address these problems, we introduce a messaging framework, Mercury, to facilitate communication among components. We provide an overview of Mercury, outline its underlying protocol and how it propagates messages to responders using relay nodes, describe a reference implementation in Unity, and present example systems built using Mercury to explain its advantages.

A framework to facilitate reusable, modular widget design for real-time interactive systems

Game engines have become popular development platforms for real-time interactive systems. Contemporary game engines, such as Unity and Unreal, feature component-based architectures, in which an objects appearance and behavior is determined by a collection of component scripts added to that object. This design pattern allows common functionality to be contained within component scripts and shared among different types of objects. In this paper, we describe a flexible framework that enables programmers to design modular, reusable widgets for real-time interactive systems using a collection of component scripts. We provide a reference implementation written in C# for the Unity game engine. Making an object, or a group of objects, part of our managed widget framework can be accomplished with just a few drag-and-drop operations in the Unity Editor. While our framework provides hooks and default implementations for common widget behavior (e.g., initialization, refresh, and toggling visibility), programmers can also define custom behavior for a particular widget or combine simple widgets into a hierarchy and build arbitrarily rich ones. Finally, we provide an overview of an accompanying library of scripts that support functionality for testing and networking.

Collaborative Virtual Reality for Low-Latency Interaction

In collaborative virtual environments, users must often perform tasks requiring coordinated action between multiple parties. Some cases are symmetric, in which users work together on equal footing, while others are asymmetric, in which one user may have more experience or capabilities than another (e.g., one may guide another in completing a task). We present a multi-user virtual reality system that supports interactions of both these types. Two collaborating users, whether co-located or remote, simultaneously manipulate the same virtual objects in a physics simulation, in tasks that require low latency networking to perform successfully. We are currently applying this approach to motor rehabilitation, in which a therapist and patient work together.

Collaborative exploration of urban data in virtual and augmented reality

From emergency planning to real estate, many domains can benefit from collaborative exploration of urban environments in VR and AR. We have created an interactive experience that allows multiple users to explore live datasets in context of an immersive scale model of the urban environment with which they are related.