Arno Hartholt

Multi-party, Multi-issue, Multi-strategy Negotiation for Multi-modal Virtual Agents

We present a model of negotiation for virtual agents that extends previous work to be more human-like and applicable to a broader range of situations, including more than two negotiators with different goals, and negotiating over multiple options. The agents can dynamically change their negotiating strategies based on the current values of several parameters and factors that can be updated in the course of the negotiation. We have implemented this model and done preliminary evaluation within a prototype training system and a three-party negotiation with two virtual humans and one human.

All Together Now

While virtual humans are proven tools for training, education and research, they are far from realizing their full potential. Advances are needed in individual capabilities, such as character animation and speech synthesis, but perhaps more importantly, fundamental questions remain as to how best to integrate these capabilities into a single framework that allows us to efficiently create characters that can engage users in meaningful and realistic social interactions. This integration requires in-depth, inter-disciplinary understanding few individuals, or even teams of individuals, possess. We help address this challenge by introducing the ICT Virtual Human Toolkit, which offers a flexible framework for exploring a variety of different types of virtual human systems, from virtual listeners and question-answering characters to virtual role-players. We show that due to its modularity, the Toolkit allows researchers to mix and match provided capabilities with their own, lowering the barrier of entry to this multi-disciplinary research challenge.

At the Virtual Frontier: Introducing Gunslinger, a Multi-Character, Mixed-Reality, Story-Driven Experience

Gunslinger isan interactive-entertainment application of virtual humans that transforms an iconic Wild West movie scene into a vivid semblance of reality. The project combines virtual humans technology with Hollywood storytelling and set building into an engaging, mixed-reality, story-driven experience, where a single participant can interact verbally and non-verbally with multiple virtual characters that are imbedded in a real, physical saloon. The Gunslinger project pushes the frontier of virtual humans research by combining question-answering dialogue techniques with explicit story representation. It incorporates speech recognition techniques and visual sensing to recognize multimodal user input. It further extends existing behavior generation methods such as SmartBody to drive tightly coupled dialogue amongst characters. These capabilities strive to seek a balance between open ended dialogue interaction and carefully crafted narrative.

Virtual Reality Exposure Therapy for Combat-Related Posttraumatic Stress Disorder

Virtual reality (VR) technology is rapidly evolving to support prolonged exposure (PE) therapy, a proven treatment for combat-related posttraumatic stress disorder. Building on the successful 2007 Virtual Iraq/Afghanistan VRET system, a team of behavioral scientists, software engineers, and virtual artists has created Bravemind, a flexible VR system that offers significantly enhanced PE treatment possibilities. The first Web extra at http://youtu.be/EiYg-kMNMtQ is a video demonstration of an original early virtual reality exposure therapy (VRET) prototype that shows a small section of an Iraqi city with a landing helicopter (2004). The second Web extra at http://youtu.be/_cS-ynWZmeQ is a video demonstration of VRET that simulates driving a Humvee in a rural part of Iraq, showcasing several encounters, including IED and road-side attacks (2007). The third Web extra at http://youtu.be/78QXX_F4mc8 is a video demonstration of VRET that simulates an overview of several Iraqi city areas (2007). The fourth Web extra at http://youtu.be/_AnixslkVLU is a video demonstration of VRET that simulates a patrol entering interior buildings in Iraq (2007). The fifth Web extra at http://youtu.be/S22aQ-DqKKU is a video demonstration of an original VRET tablet interface that allows the clinician to change virtual reality settings and trigger encounters (2007). The sixth Web extra at http://youtu.be/C-fspuLo4vw is a video demonstration of the Bravemind VRET prototype showing a variety of driving and dismounted scenarios with encounters in Iraq and Afghanistan (2013). The sixth Web extra at http://youtu.be/HSPDomDAigg is a video collection of Iraqi and Afghanistan VRET scenarios within the Bravemind prototype (2013).

Virtual Reality as a Tool for Delivering PTSD Exposure Therapy and Stress Resilience Training

The incidence of post-traumatic stress disorder (PTSD) in returning Operation Enduring Freedom and Operation Iraqi Freedom military personnel has created a significant behavioral health care challenge. One emerging form of treatment for combat-related PTSD that has shown promise involves the delivery of exposure therapy using immersive virtual reality (VR). Initial outcomes from open clinical trials have been positive, and fully randomized controlled trials are currently in progress. Inspired by the initial success of our research using VR to emotionally engage and successfully treat persons undergoing exposure therapy for PTSD, we have developed a similar VR-based approach to deliver resilience training prior to an initial deployment. The STress Resilience In Virtual Environments (STRIVE) project aims to create a set of combat simulations (derived from our existing virtual Iraq/Afghanistan PTSD exposure therapy system) that are part of a multiepisode interactive narrative experience. Users can be immersed within challenging virtual combat contexts and interact with virtual characters as part of an experiential approach for learning emotional coping strategies believed to enhance stress resilience. This article describes the development and evaluation of the virtual Iraq/Afghanistan exposure therapy system and then details its current transition into the STRIVE tool for predeployment stress resilience training.

Developing INOTS to support interpersonal skills practice

The Immersive Naval Officer Training System (INOTS) is a blended learning environment that merges traditional classroom instruction with a mixed reality training setting. INOTS supports the instruction, practice and assessment of interpersonal communication skills. The goal of INOTS is to provide a consistent training experience to supplement interpersonal skills instruction for Naval officer candidates without sacrificing trainee throughput and instructor control. We developed an instructional design from cognitive task analysis interviews with experts to serve as a framework for system development. We also leveraged commercial student response technology and research technologies including natural language recognition, virtual humans, realistic graphics, intelligent tutoring and automated instructor support tools. In this paper, we describe our methodologies for developing a blended learning environment, and our challenges adding mixed reality and virtual human technologies to a traditional classroom to support interpersonal skills training.1 2

Virtual Reality Applications for the Assessment and Treatment of PTSD

Posttraumatic stress disorder (PTSD) continues to be a significant issue for veterans of the Iraq and Afghanistan combat environments. A growing literature has now evolved that supports the idea that Virtual Reality Exposure Therapy (VRET) may be a useful approach for addressing this healthcare challenge. This chapter will describe the history and implementation of the Virtual Iraq/Afghanistan system (now referred to as BRAVEMIND) for the delivery of VRET for PTSD in United States Service Members and Veterans. The original VR system, implemented in 2007, was rebuilt in 2013 using the latest available software and it leveraged feedback from clinical users of the first system to inform its design. The BRAVEMIND content and functionality was also expanded to support the specific VRET needs of combat medics/corpsmen and persons who have experienced military sexual trauma (MST). The chapter presents the context, rationale, and the early research with phobias that informed the development of these VR systems. It also reviews the literature applying the system for the treatment of PTSD and discusses new research efforts on the use of this software for the assessment of PTSD.

A demonstration of the perception system in SimSensei, a virtual human application for healthcare interviews

We present the SimSensei system, a fully automatic virtual agent that conducts interviews to assess indicators of psychological distress. With this demo, we focus our attention on the perception part of the system, a multimodal framework which captures and analyzes user state behavior for both behavioral understanding and interactional purposes. We will demonstrate real-time user state sensing as a part of the SimSensei architecture and discuss how this technology enabled automatic analysis of behaviors related to psychological distress.

Facial cartography: interactive scan correspondence

We present a semi-automatic technique for computing surface correspondences between 3D facial scans in different expressions, such that scan data can be mapped into a common domain for facial animation. The technique can accurately correspond high-resolution scans of widely differing expressions -- without requiring intermediate pose sequences -- such that they can be used, together with reflectance maps, to create high-quality blendshape-based facial animation. We optimize correspondences through a combination of Image, Shape, and Internal forces, as well as Directable forces to allow a user to interactively guide and refine the solution. Key to our method is a novel representation, called an Active Visage, that balances the advantages of both deformable templates and correspondence computation in a 2D canonical domain. We show that our semi-automatic technique achieves more robust results than automated correspondence alone, and is more precise than is practical with unaided manual input.

Virtual Suspect William

We evaluate an algorithm which computes the responses of an agent that plays the role of a suspect in simulations of police interrogations. The algorithm is based on a cognitive model - the response model - that is centred around keeping track of interpersonal relations. The model is parametrized in such a way that different personalities of the virtual suspect can be defined. In the evaluation we defined three different personalities and had participants guess the personality based on the responses the model provided in an interaction with the participant. We investigate what factors contributed to the ability of a virtual agent to show behaviour that was recognized by participants as belonging to a persona.