Masayuki Nakazawa

RWC multimodal database for interactions by integration of spoken language and visual information

The paper describes the design policy and prototype data collection of RWC (Real World Computing Program) multimodal database. The database is intended for research and development on the integration of spoken language and visual information for human computer interactions. The interactions are supposed to use image recognition, image synthesis, speech recognition, and speech synthesis. Visual information also includes non-verbal communication such as interactions using hand gestures and facial expressions between human and a human-like CG (computer graphics) agent with a face and hands. Based on the experiments of interactions with these modes, specifications of the database are discussed from the viewpoint of controlling the variability and cost for the collection.

Multimodal agent interface based on dynamical dialogue model: MAICO: multimodal agent interface for communication

In this paper, we describe a multimodal interface prototype system based on Dynamical Dialogue Model. This system not only integrates information of speech and gestures, but also controls the response timing in order to realize a smooth interaction between user and computer. Our approach consists of human-human dialogue analysis, and computational modeling of dialogue.

An active microphone method for speech recognition systems

We propose a method to recognize human speech without the restriction that the speaker should be close to the microphone for the multiple modal interface. This method has the following features: 1) to control the microphone directivity for improvement of S/N ratio by using the speakers position estimated by the result of gesture recognition, 2) to compose the impulse response of the transfer characteristic between speakers and microphones into reference data of the recognition system. We describe the active microphone method in detail, and show its effectivity through several experiments. Then, we discuss the results.