Iwan de Kok

Continuous Interaction with a Virtual Human

This paper presents our progress in developing a Virtual Human capable of being an attentive speaker. Such a Virtual Human should be able to attend to its interaction partner while it is speaking—and modify its communicative behavior on-the-fly based on what it observes in the behavior of its partner. We report new developments concerning a number of aspects, such as scheduling and interrupting multimodal behavior, automatic classification of listener responses, generation of response eliciting behavior, and strategies for generating appropriate reactions to listener responses. On the basis of this progress, a task-based setup for a responsive Virtual Human was implemented to carry out two user studies, the results of which are presented and discussed in this paper.

A Multimodal System for Real-Time Action Instruction in Motor Skill Learning

We present a multimodal coaching system that supports online motor skill learning. In this domain, closed-loop interaction between the movements of the user and the action instructions by the system is an essential requirement. To achieve this, the actions of the user need to be measured and evaluated and the system must be able to give corrective instructions on the ongoing performance. Timely delivery of these instructions, particularly during execution of the motor skill by the user, is thus of the highest importance. Based on the results of an empirical study on motor skill coaching, we analyze the requirements for an interactive coaching system and present an architecture that combines motion analysis, dialogue management, and virtual human animation in a motion tracking and 3D virtual reality hardware setup. In a preliminary study we demonstrate that the current system is capable of delivering the closed-loop interaction that is required in the motor skill learning domain.

The Intelligent Coaching Space: A Demonstration

Here we demonstrate our Intelligent Coaching Space, an immersive virtual environment in which users learn a motor action (e.g. a squat) under the supervision of a virtual coach. We detail how we assess the ability of the coachee in executing the motor action, how the intelligent coaching space and its features are realized and how the virtual coach leads the coachee through a coaching session.

Deictic gestures in coaching interactions

In motor skill coaching interaction coaches use several techniques to improve the motor skill of the coachee. Through goal setting, explanations, instructions and feedback the coachee is motivated and guided to improve the motor skill. These verbal speech actions are often accompanied by iconic or deictic gestures and other nonverbal acts, such as demonstrations. We are building a virtual coach that is capable of the same behaviour. In this paper we have taken a closer look at the form, type and timing of deictic gestures in our corpus of human-human coaching interactions. We show that a significant amount of the deictic gestures actually touch the referred object, that most of the gestures are complimentary (contrary to previous research) and often occur before the lexical affiliate.