Michael Johnston

MATCH: An Architecture for Multimodal Dialogue Systems

Mobile interfaces need to allow the user and system to adapt their choice of communication modes according to user preferences, the task at hand, and the physical and social environment. We describe a multimodal application architecture which combines finite-state multimodal language processing, a speech-act based multimodal dialogue manager, dynamic multimodal output generation, and user-tailored text planning to enable rapid prototyping of multimodal interfaces with flexible input and adaptive output. Our testbed application MATCH (Multimodal Access To City Help) provides a mobile multimodal speech-pen interface to restaurant and sub-way information for New York City.

Robust understanding in multimodal interfaces

Multimodal grammars provide an effective mechanism for quickly creating integration and understanding capabilities for interactive systems supporting simultaneous use of multiple input modalities. However, like other approaches based on hand-crafted grammars, multimodal grammars can be brittle with respect to unexpected, erroneous, or disfluent input. In this article, we show how the finite-state approach to multimodal language processing can be extended to support multimodal applications combining speech with complex freehand pen input, and evaluate the approach in the context of a multimodal conversational system (MATCH). We explore a range of different techniques for improving the robustness of multimodal integration and understanding. These include techniques for building effective language models for speech recognition when little or no multimodal training data is available, and techniques for robust multimodal understanding that draw on classification, machine translation, and sequence edit methods. We also explore the use of edit-based methods to overcome mismatches between the gesture stream and the speech stream.

MATCHKiosk: a multimodal interactive city guide

Multimodal interfaces provide more flexible and compelling interaction and can enable public information kiosks to support more complex tasks for a broader community of users. MATCHKiosk is a multimodal interactive city guide which provides users with the freedom to interact using speech, pen, touch or multimodal inputs. The system responds by generating multimodal presentations that synchronize synthetic speech with a life-like virtual agent and dynamically generated graphics.

MVA: The Multimodal Virtual Assistant

The Multimodal Virtual Assistant (MVA) is an application that enables users to plan an outing through an interactive multimodal dialog with a mobile device. MVA demonstrates how a cloud-based multimodal language processing infrastructure can support mobile multimodal interaction. This demonstration will highlight incremental recognition, multimodal speech and gesture input, contextually-aware language understanding, and the targeted clarification of potentially incorrect segments within user input.

Speech and Multimodal Interaction in Mobile Search

With the widespread adoption of high-speed wireless networks symbiotically complemented by the burgeoning demand for smart mobile devices, access to the Internet is evolving from personal computers (PCs) to mobile devices. In this article, we highlight the characteristics of mobile search, discuss the state of speech-based mobile search, and present opportunities for exploiting multimodal interaction to optimize the efficiency of mobile search.

Multimodal interactive spaces: MagicTV and magicMAP

Through the growing popularity of voice-enabled search, multimodal applications are finally starting to get into the hands of consumers. However, these applications are principally for mobile platforms and generally involve highly-moded interaction where the user has to click or hold a button in order to speak. Significant technical challenges remain in bringing multimodal interaction to other environments such as smart living rooms and classrooms, where users speech and gesture is directed toward large displays or interactive kiosks and the microphone and other sensors are ‘always on’. In this demonstration, we present a framework combining low cost hardware and open source software that lowers the barrier of entry for exploration of multimodal interaction in smart environments. Specifically, we will demonstrate the combination of infrared tracking, face detection, and open microphone speech recognition for media search (magicTV) and map navigation (magicMap).

OpenGesture: a low-cost authoring framework for gesture and speech based application development and learning analytics

In this paper, we present an application framework for enabling education practitioners and researchers to develop interactive, multi-modal applications. These applications can be designed using typical HTML programming, and will enable a larger audience to make applications that incorporate speech recognition, gesture recognition and engagement detection. The application framework uses open-source software and inexpensive hardware that supports both multi-touch and multi-user capabilities.

Interact: Tightly-coupling Multimodal Dialog with an Interactive Virtual Assistant

Interact is a mobile virtual assistant that uses multimodal dialog to enable an interactive concierge experience over multiple application domains including hotel, restaurants, events, and TV search. Interact demonstrates how multi- modal interaction combined with conversational dialog en- ables a richer and more natural user experience. This demonstration will highlight incremental recognition and under- standing, multimodal speech and gesture input, context track- ing over multiple simultaneous domains, and the use of multimodal interface techniques to enable disambiguation of erors and online personalization.

Book ReviewMultimodality in Language and Speech Systems Björn Granström, David House, and Inger Karlsson (editors) (Royal Institute of Technology, Stockholm) Dordrecht: Kluwer Academic Publishers (Text, speech and language technology series, edited by Nancy Ide and Jean Véronis, volume 19), 2002, ix+241 pp; hardbound, ISBN 0-4020-0635-7,

We are a species of storytellers. We tell stories in many different ways. In ancient legends from Gilgamesh to the Odyssey to the Pentateuch, we set down out pre-literate lore. In proverbs, aphorisms and epigrams, we condense our wisdom. We write poems, romances and detective novels. We write historical romances and “how-to” books that explain how to find love, quit booze, get a better job or fix up an old car. When threatened, we growl ominously, whimper pathetically or compose legal defence briefs. Sometimes, especially when we are ambitiously seeking the truth, we try science.

82.00, £56.00, €89.00

We are a species of storytellers. We tell stories in many different ways. In ancient legends from Gilgamesh to the Odyssey to the Pentateuch, we set down out pre-literate lore. In proverbs, aphorisms and epigrams, we condense our wisdom. We write poems, romances and detective novels. We write historical romances and “how-to” books that explain how to find love, quit booze, get a better job or fix up an old car. When threatened, we growl ominously, whimper pathetically or compose legal defence briefs. Sometimes, especially when we are ambitiously seeking the truth, we try science.