Remo Pillat

A unified framework for individualized avatar-based interactions

This paper presents a framework to interactively control avatars in remote environments. The system, called AMITIES, serves as the central component that connects people controlling avatars (inhabiters), various manifestations of these avatars (surrogates), and people interacting with these avatars (participants). A multiserver–client architecture, based on a low-demand network protocol, connects the participant environment(s), the inhabiter station(s), and the avatars. A human-in-the-loop metaphor provides an interface for remote operation, with support for multiple inhabiters, multiple avatars, and multiple participants. Custom animation blending routines and a gesture-based interface provide inhabiters with an intuitive avatar control paradigm. This gesture control is enhanced by genres of program-controlled behaviors that can be triggered by events or inhabiter choices for individual or groups of avatars. This mixed (agency and gesture-based) control paradigm reduces the cognitive and physical loads on the inhabiter while supporting natural bidirectional conversation between participants and the virtual characters or avatar counterparts, including ones with physical manifestations, for example, robotic surrogates. The associated system affords the delivery of personalized experiences that adapt to the actions and interactions of individual users, while staying true to each virtual characters personality and backstory. In addition to its avatar control paradigm, AMITIES provides processes for character and scenario development, testing, and refinement. It also has integrated capabilities for session recording and event tagging, along with automated tools for reflection and after-action review. We demonstrate effectiveness by describing an instantiation of AMITIES, called TeachLivE, that is widely used by colleges of education to prepare new teachers and provide continuing professional development to existing teachers. Finally, we show the systems flexibility by describing a number of other diverse applications, and presenting plans to enhance capabilities and application areas.

AMITIES: avatar-mediated interactive training and individualized experience system

This paper presents an architecture to control avatars and virtual characters in remote interaction environments. A human-in-the-loop (interactor) metaphor provides remote control of multiple virtual characters, with support for multiple interactors and multiple observers. Custom animation blending routines and a gesture-based interface provide interactors with an intuitive digital puppetry paradigm. This paradigm reduces the cognitive and physical loads on the interactor while supporting natural bi-directional conversation between a user and the virtual characters or avatar counterparts. A multi-server-client architecture, based on a low-demand network protocol, connects the user environment, interactor station(s) and observer station(s). The associated system affords the delivery of personalized experiences that adapt to the actions and interactions of individual users, while staying true to each virtual characters personality and backstory. This approach has been used to create experiences designed for training, education, rehabilitation, remote presence and other-related applications.

Continuum of virtual-human space: towards improved interaction strategies for physical-virtual avatars

In this work, a broad continuum of 3D space that encapsulates avatars, ranging from artificial to real in shape, appearance and intelligence is defined. The research focuses on the control of physical-virtual avatars that occupy a specific region in this space that may be suitable for interacting with elements in the environment. To facilitate this control, a paradigm called microposes is developed that overcomes the need for high network bandwidth during remote tele-operation of avatars. The avatar itself uses a control strategy that interprets the received microposes data and executes motions that appear natural and human-like in the presence of data loss and noise. The physical-virtual avatar is used in several training and learning scenarios. Results during testing reveal a reduced bandwidth requirement during remote tele-operation of physical virtual avatars and a good motion tracking performance with respect to a commanded pose from the inhabiter.

Compliance estimation during bilateral teleoperation of a robotic arm

A method for estimating compliance properties of objects in the workspace of remotely located robotic manipulators is presented herein. An architecture that supports decoupled remote teleoperation is briefly described, and used to control the manipulator during recovery of an objects compliance parameters. These parameters are based on the Kelvin-Voigt contact model and estimated in simulation as well as experimentally using offline and online estimation techniques. A comparison of the estimation techniques reveals that the proposed RANSAC-based technique is better suited to compliance recovery over existing methods, even in the presence of sensor noise. The recovered parameters are used to update a reconfigurable compliance object in simulation to closely match that of the real-world object, thereby enabling the use of realistic haptic feedback during remote teleoperation.

Technical Report: Exploring Human Surrogate Characteristics

This report highlights some of the historical evolution of our research involving the characteristics that are essential for effective human-surrogate interactions. In this report, a consolidated glossary of terms related to human-surrogate interaction is described, following which an attempt at defining a consolidated space of surrogate characteristics is made. The rationale behind the space definition is to provide an easy way to categorize existing and future systems, and help identify areas in which the research community might focus its efforts.

Advances in Shader Lamps Avatars for telepresence

Since the introduction of the animatronic Shader Lamps Avatar in 2009, several aspects of the systems design have been improved to create a more effective telepresence experience. This includes an analysis of the tradeoffs between front and rear projection and bethardware deviceween rigidly aligning the projector with the display surface or needing to track it. A new control mode is implemented for the pan-tilt unit that enables the avatar head pose to better emulate real human motion and a new hardware device allows the motorized wheelchair that carries the avatar to be controlled by the remote inhabiter. With the recognition of the importance of high quality audio, the audio components for both the inhabiter and the avatar have also been updated. Finally, some preliminary live tests are presented and areas of continued research are discussed.