Nigel Ward

Prosodic features which cue back-channel responses in English and Japanese

Back-channel feedback, responses such as uh-uh from a listener, is a pervasive feature of conversation. It has long been thought that the production of back-channel feedback depends to a large extent on the actions of the other conversation partner, not just on the volition of the one who produces them. In particular, prosodic cues from the speaker have long been thought to play a role, but have so far eluded identification. We have earlier suggested that an important prosodic cue involved, in both English and Japanese, is a region of low pitch late in an utterance (Ward, 1996). This paper discusses issues in the definition of back-channel feedback, presents evidence for our claim, surveys other factors which elicit or inhibit backchannel responses, and mentions a few related phenomena and theoretical issues.

Responding to subtle, fleeting changes in the user's internal state

In human-to-human interaction, people sometimes are able to pick up and respond sensitively to the others internal state as it shifts moment by moment over the course of an exchange. To find out whether such an ability is worthwhile for computer human interfaces, we built a semi-automated tutoring-type spoken dialog system. The system inferred information about the users \scare{ephemeral emotions}, such as confidence, confusion, pleasure, and dependency, from the prosody of his utterances and the context. It used this information to select the most appropriate acknowledgement form at each moment. In doing so the system was following some of the basic social conventions for real-time interaction. Users rated the system with this ability more highly than a version without.

Achieving rapport with turn-by-turn, user-responsive emotional coloring

People in dialog use a rich set of nonverbal behaviors, including variations in the prosody of their utterances. Such behaviors, often emotion-related, call for appropriate responses, but todays spoken dialog systems lack the ability to do this. Recent work has shown how to recognize user emotions from prosody and how to express system-side emotions with prosody, but demonstrations of how to combine these functions to improve the user experience have been lacking. Working with a corpus of conversations with students about graduate school, we analyzed the emotional states of the interlocutors, utterance-by-utterance, using three dimensions: activation, evaluation, and power. We found that the emotional coloring of the speakers utterance could be largely predicted from the emotion shown by her interlocutor in the immediately previous utterance. This finding enabled us to build Gracie, the first spoken dialog system that recognizes a users emotional state from his or her speech and gives a response with appropriate emotional coloring. Evaluation with 36 subjects showed that they felt significantly more rapport with Gracie than with either of two controls. This shows that dialog systems can tap into this important level of interpersonal interaction using todays technology.

A tool for taking class notes

Students still take class notes using pencil and paper-although digital documents are more legible, easier to search in and easier to edit--in part because of the lack of software to support note-taking. Class notes are characterized by free spatial organization, many small chunks of text, and a dense mix of text and graphic elements. These characteristics imply that a note-taking system should use pen, keyboard and mouse-or-equivalent; allow the swift entry of text at any desired position; and minimize the need to switch between input tools. A system with these properties was built and used by 10 subjects in a controlled study and by four users in their classes. Some users preferred our system to pencil and paper, suggesting that taking class notes with the computer is feasible.

Prosodic and temporal features for language modeling for dialog

If we can model the cognitive and communicative processes underlying speech, we should be able to better predict what a speaker will do. With this idea as inspiration, we examine a number of prosodic and timing features as potential sources of information on what words the speaker is likely to say next. In spontaneous dialog we find that word probabilities do vary with such features. Using perplexity as the metric, the most informative of these included recent speaking rate, volume, and pitch, and time until end of utterance. Using simple combinations of such features to augment trigram language models gave up to a 8.4% perplexity benefit on the Switchboard corpus, and up to a 1.0% relative reduction in word error rate (0.3% absolute) on the Verbmobil II corpus.

Issues in word choice

This paper discusses word choice for natural language generation. It examines 11 issues, the solutions that have been proposed for them, and their implications for design. The issues are:How are appropriate words chosen?How is conciseness ensured?When does choice stop?How are patterns of lexicalization respected?How are interactions among choices handrled?How are the correct parts of speech chosen?How are words chosen to satisfy constituency?What ensures that a word stands in the correct relation to its neighbors?How is word order determined?Are all words chosen in the same way?In what order are the factors considered?This paper also discusses FIG, a generator which incorporates novel solutions to many of these issues. FIG violates common assumptions about the roles of modularity and grammar in generator design. Analysis of FIG leads to 4 principles for generator design, as follows:Have an explicit representation of the status of the generation process at each point in time.Use a single, unified representation.Do not rely on the details of the structure of the input.Treat most choices as emergent.

A Framework for Model-based Evaluation of Spoken Dialog Systems

Improvements in the quality, usability and acceptability of spoken dialog systems can be facilitated by better evaluation methods. To support early and efficient evaluation of dialog systems and their components, this paper presents a tripartite framework describing the evaluation problem. One part models the behavior of user and system during the interaction, the second one the perception and judgment processes taking place inside the user, and the third part models what matters to system designers and service providers. The paper reviews available approaches for some of the model parts, and indicates how anticipated improvements may serve not only developers and users but also researchers working on advanced dialog functions and features.

A parallel approach to syntax for generation

Abstract To produce good utterances from nontrivial inputs a natural language generator should consider many words in parallel, which raises the question of how to handle syntax in a parallel generator. If a generator is incremental and centered on the task of word choice, then the role of syntax is merely to help evaluate the appropriateness of words. One way to do this is to represent syntactic knowledge as an inventory of “syntactic constructions” and to have many constructions active in parallel at run-time. If this is done then the syntactic form of utterances can be emergent, resulting from synergy among constructions, and there is no need to build up or manipulate representations of syntactic structure. This approach is implemented in FIG, an incremental generator based on spreading activation, in which syntactic knowledge is represented in the same network as world knowledge and lexical knowledge.

A Responsive Dialog System

Modeling language as people really use it is an elusive goal. Today, thanks to advances in speech recognition, dialog systems capable of understanding the meaning of user input and replying with appropriate information exist, but there are as yet no systems which interact naturally with humans. Two problems are: 1. Priority is given to understanding and responding accurately; but for human dialog, being responsive and interactive is also important. 2. The granularity of interaction is the sentence; but for human dialog, interaction happens more frequently, in real time, often with overlapping utterances; Given that such responsiveness is important to human language use, the question arises: how do we build systems with these abilities? The obvious approach is to add these abilities to a meaning based speech system. An alternative approach is to take these abilities as a basic foundation, and to layer meaning-based processing on top of this, subsumption-style (Ward, 1997).

Software for taking notes in class

Although digital devices are replacing paper and pencil in ever more domains, class notes have so far resisted this trend. In part this is because class notes and the note-taking process are unique and different from the tasks supported by existing software. A system to support classroom note-taking should embody seven design principles. A prototype system based on these principles was used in the laboratory and in the classroom by three long-term users. Some users prefered note-taking with the system over pencil and paper, suggesting that taking lecture-notes with the computer is feasible.