Donna K. Byron

Toward Conversational Human-Computer Interaction

The belief that humans will be able to interact with computers in conversational speech has long been a favorite subject in science fiction, reflecting the persistent belief that spoken dialogue would be the most natural and powerful user interface to computers. With recent improvements in computer technology and in speech and language processing, such systems are starting to appear feasible. There are significant technical problems that still need to be solved before speech-driven interfaces become truly conversational. This article describes the results of a 10-year effort building robust spoken dialogue systems at the University of Rochester.

An architecture for a generic dialogue shell

This paper describes our work on dialogue systems that can mimic human conversation, with the goal of providing intuitive access to a wide range of applications by expanding the users options in the interaction. We concentrate on practical dialogue: dialogues in which the participants need to accomplish some objective or perform some task. Two hypotheses regarding practical dialogue motivate our research. First, that the conversational competence required for practical dialogues, while still complex, is significantly simpler to achieve than general human conversational competence. And second, that within the genre of practical dialogue, the bulk of the complexity in the language interpretation and dialogue management is independent of the task being performed. If these hypotheses are true, then it should be possible to build a generic dialogue shell for practical dialogue, by which we mean the full range of components required in a dialogue system, including speech recognition, language processing, dialogue management and response planning, built in such a way as to be readily adapted to new applications by specifying the domain and task models. This paper documents our progress and what we have learned so far based on building and adapting systems in a series of different problem solving domains.

Resolving Pronominal Reference to Abstract Entities

This paper describes PHORA, a technique for resolving pronominal reference to either individual or abstract entities. It defines processes for evoking abstract referents from discourse and for resolving both demonstrative and personal pronouns. It successfully interprets 72% of test pronouns, compared to 37% for a leading technique without these features.

Usability of Conversational Agents by Patients with Inadequate Health Literacy: Evidence from Two Clinical Trials

Embodied Conversational Agents (ECA) are computer-animated characters that simulate face-to-face conversation with patients. These agents can be programmed with best practices in human-human health communication and used for automated health education and behavior change counseling interventions. Evidence is presented from two ongoing clinical trials demonstrating that patients at different levels of health literacy find these agents acceptable and easy to use for automated health communication interventions. Innovative computer interface systems can be used to ensure that inadequate health literacy not serve as a barrier to interventions using health information technology.

Noun Phrase Generation for Situated Dialogs

We report on a study examining the generation of noun phrases within a spoken dialog agent for a navigation domain. The task is to provide real-time instructions that direct the user to move between a series of destinations within a large interior space. A subtask within sentence planning is determining what form to choose for noun phrases. This choice is driven by both the discourse history and spatial context features derived from the direction-followers position, e.g. his view angle, distance from the target referent and the number of similar items in view. The algorithm was developed as a decision tree and its output was evaluated by a group of human judges who rated 62.6% of the expressions generated by the system to be as good as or better than the language originally produced by human dialog partners.

A Preliminary Model of Centering in Dialog

The centering framework explains local coherence by relating local focus and the form of referring expressions. It has proven useful in monolog, but its utility for multi-party discourse has not been shown, and a variety of issues must be tackled to adapt the model for dialog. This paper reports our application of three naive models of centering theory for dialog. These results will be used as baselines for evaluating future models.

Prosody and the resolution of pronominal anaphora

In this paper, we investigate the acoustic prosodic marking of demonstrative and personal pronouns in task-oriented dialog. Although it has been hypothesized that acoustic marking affects pronoun resolution, we find that the prosodic information extracted from the data is not sufficient to predict antecedent type reliably. Inter-speaker variation accounts for much of the prosodic variation that we find in our data. We conclude that prosodic cues should be handled with care in robust, speaker-independent dialog systems.

The first challenge on generating instructions in virtual environments

This paper describes the First Challenge on Generating Instructions in Virtual Environments (GIVE-1). GIVE is a shared task for generation systems which give real-time natural-language instructions to users in a virtual 3D world. These systems are evaluated by connecting users and NLG systems over the Internet. We describe the design and results of GIVE-1 as well as the participating NLG systems, and validate the experimental methodology by comparing the results collected over the Internet with results from a more traditional laboratory-based experiment.

Report on the First NLG Challenge on Generating Instructions in Virtual Environments (GIVE)

We describe the first installment of the Challenge on Generating Instructions in Virtual Environments (GIVE), a new shared task for the NLG community. We motivate the design of the challenge, describe how we carried it out, and discuss the results of the system evaluation.

Utilizing visual attention for cross-modal coreference interpretation

In this paper, we describe an exploratory study to develop a model of visual attention that could aid automatic interpretation of exophors in situated dialog. The model is intended to support the reference resolution needs of embodied conversational agents, such as graphical avatars and robotic collaborators. The model tracks the attentional state of one dialog participant as it is represented by his visual input stream, taking into account the recency, exposure time, and visual distinctness of each viewed item. The model correctly predicts the correct referent of 52% of referring expressions produced by speakers in human-human dialog while they were collaborating on a task in a virtual world. This accuracy is comparable with reference resolution based on calculating linguistic salience for the same data.