Dan Bohus

The RavenClaw dialog management framework: Architecture and systems

In this paper, we describe RavenClaw, a plan-based, task-independent dialog management framework. RavenClaw isolates the domain-specific aspects of the dialog control logic from domain-independent conversational skills, and in the process facilitates rapid development of mixed-initiative systems operating in complex, task-oriented domains. System developers can focus exclusively on describing the dialog task control logic, while a large number of domain-independent conversational skills such as error handling, timing and turn-taking are transparently supported and enforced by the RavenClaw dialog engine. To date, RavenClaw has been used to construct and deploy a large number of systems, spanning different domains and interaction styles, such as information access, guidance through procedures, command-and-control, medical diagnosis, etc. The framework has easily adapted to all of these domains, indicating a high degree of versatility and scalability.

Facilitating multiparty dialog with gaze, gesture, and speech

We study how synchronized gaze, gesture and speech rendered by an embodied conversational agent can influence the flow of conversations in multiparty settings. We begin by reviewing a computational framework for turn-taking that provides the foundation for tracking and communicating intentions to hold, release, or take control of the conversational floor. We then present implementation aspects of this model in an embodied conversational agent. Empirical results with this model in a shared task setting indicate that the various verbal and non-verbal cues used by the avatar can effectively shape the multiparty conversational dynamics. In addition, we identify and discuss several context variables which impact the turn allocation process.

Dialog in the open world: platform and applications

We review key challenges of developing spoken dialog systems that can engage in interactions with one or multiple participants in relatively unconstrained environments. We outline a set of core competencies for open-world dialog, and describe three prototype systems. The systems are built on a common underlying conversational framework which integrates an array of predictive models and component technologies, including speech recognition, head and pose tracking, probabilistic models for scene analysis, multiparty engagement and turn taking, and inferences about user goals and activities. We discuss the current models and showcase their function by means of a sample recorded interaction, and we review results from an observational study of open-world, multiparty dialog in the wild.

Learning to Predict Engagement with a Spoken Dialog System in Open-World Settings

We describe a machine learning approach that allows an open-world spoken dialog system to learn to predict engagement intentions in situ, from interaction. The proposed approach does not require any developer supervision, and leverages spatiotemporal and attentional features automatically extracted from a visual analysis of people coming into the proximity of the system to produce models that are attuned to the characteristics of the environment the system is placed in. Experimental results indicate that a system using the proposed approach can learn to recognize engagement intentions at low false positive rates (e.g. 2--4%) up to 3--4 seconds prior to the actual moment of engagement.

LARRI: A Language-Based Maintenance and Repair Assistant

LARRI (Language-based Agent for Retrieval of Repair Information) is a dialo- gue-based system for support of maintenance and repair domains, characterized by large amounts of documentation and procedural information. LARRI is based on an architecture developed by Carnegie Mellon University for the DARPA Communicator programme and is integrated with a wearable computer system developed by the Wearable Computing Group at CMU. The system adapts an architecture developed and optimised for a telephone-based problem solving task (travel planning), and applies it to a very different domain — aircraft mainteance. Following two field trials in which LARRI was used by professional aircraft mechanics, we found that our architecture extended readily to a multimodal and multi-media framework. At the same time we found that assumptions that were reasonable in a services domain turn out to be inappropriate for a maintenance domain. Apart from the need to manage integration between input modes and output modalities, we found that the system needed to support multiple categories of tasks and that a different balance between user and system goals was required. A significant problem in the maintenance domain is the need to assimilate and make available for language processing appropriate domain information.

Constructing accurate beliefs in spoken dialog systems

We propose a novel approach for constructing more accurate beliefs over concept values in spoken dialog systems by integrating information across multiple turns in the conversation. In particular, we focus our attention on updating the confidence score of the top hypothesis for a concept, in light of subsequent user responses to system confirmation actions. Our data-driven approach bridges previous work in confidence annotation and correction detection, providing a unified framework for belief updating. The approach significantly outperforms heuristic rules currently used in most spoken dialog systems

Conquest---An Open-Source Dialog System for Conferences

We describe ConQuest, an open-source, reusable spoken dialog system that provides technical program information during conferences. The system uses a transparent, modular and open infrastructure, and aims to enable applied research in spoken language interfaces. The conference domain is a good platform for applied research since it permits periodical redeployments and evaluations with a real user-base. In this paper, we describe the systems functionality, overall architecture, and we discuss two initial deployments.

Managing Human-Robot Engagement with Forecasts and... um ... Hesitations

We explore methods for managing conversational engagement in open-world, physically situated dialog systems. We investigate a self-supervised methodology for constructing forecasting models that aim to anticipate when participants are about to terminate their interactions with a situated system. We study how these models can be leveraged to guide a disengagement policy that uses linguistic hesitation actions, such as filled and non-filled pauses, when uncertainty about the continuation of engagement arises. The hesitations allow for additional time for sensing and inference, and convey the systems uncertainty. We report results from a study of the proposed approach with a directions-giving robot deployed in the wild.

ONLINE SUPERVISED LEARNING OF NON-UNDERSTANDING RECOVERY POLICIES

Spoken dialog systems typically use a limited number of non- understanding recovery strategies and simple heuristic policies to engage them (e.g. first ask user to repeat, then give help, then transfer to an operator). We propose a supervised, online method for learning a non-understanding recovery policy over a large set of recovery strategies. The approach consists of two steps: first, we construct runtime estimates for the likelihood of success of each recovery strategy, and then we use these estimates to construct a policy. An experiment with a publicly available spoken dialog system shows that the learned policy produced a 12.5% relative improvement in the non-understanding recovery rate.

Decisions about turns in multiparty conversation: from perception to action

We present a decision-theoretic approach for guiding turn taking in a spoken dialog system operating in multiparty settings. The proposed methodology couples inferences about multiparty conversational dynamics with assessed costs of different outcomes, to guide turn-taking decisions. Beyond considering uncertainties about outcomes arising from evidential reasoning about the state of a conversation, we endow the system with awareness and methods for handling uncertainties stemming from computational delays in its own perception and production. We illustrate via sample cases how the proposed approach makes decisions, and we investigate the behaviors of the proposed methods via a retrospective analysis on logs collected in a multiparty interaction study.