Wolfgang Minker

Affective Dialogue Systems

The monitoring of emotional user states can help to assess the progress of human-machine-communication. If we look at specific databases, however, we are faced with several problems: users behave differently, even within one and the same setting, and some phenomena are sparse; thus it is not possible to model and classify them reliably. We exemplify these difficulties on the basis of SympaFly, a database with dialogues between users and a fully automatic speech dialogue telephone system for flight reservation and booking, and discuss possible remedies.

Evaluation and usability of multimodal spoken language dialogue systems

With the technical advances and market growth in the field, the issues of evaluation and usability of spoken language dialogue systems, unimodal as well as multimodal, are as crucial as ever. This paper discusses those issues by reviewing a series of European and US projects which have produced major results on evaluation and usability. Whereas significant progress has been made on unimodal spoken language dialogue systems evaluation and usability, the emergence of, among others, multimodal, mobile, and domain-oriented systems continues to pose entirely new challenges to research in evaluation and usability.

Endowing Spoken Language Dialogue Systems with Emotional Intelligence

While most dialogue systems restrict themselves to the adjustment of the propositional contents, our work concentrates on the generation of stylistic va- riations in order to improve the users perception of the interaction. To accomplish this goal, our approach integrates a social theory of politeness with a cognitive theory of emotions. We propose a hierarchical selection process for politeness behaviors in order to enable the refinement of decisions in case additional context information becomes available.

Stochastic versus rule-based speech understanding for information retrieval

Abstract In this paper we report our experience at LIMSI-CNRS in developing and porting a stochastic component for natural language understanding to different tasks and human languages. The domains in which we test this component are the American ATIS (Air Travel Information Services) and the French MASK (Multimodal-Multimedia Automated Service Kiosk) applications. The study demonstrates that for limited applications, a stochastic method outperforms a well-tuned rule-based component. In addition we show that the human effort can be limited to the task of data labeling, which is much simpler than the design, maintenance and extension of the grammar rules. Since a stochastic method automatically learns the semantic formalism through an analysis of these data, it is comparatively flexible and robust.

Perception and Interactive Technologies

Head Pose and Eye Gaze Tracking.- Guiding Eye Movements for Better Communication and Augmented Vision.- Detection of Head Pose and Gaze Direction for Human-Computer Interaction.- Modelling and Simulation of Perception.- Modelling and Simulation of Spontaneous Perception Switching with Ambiguous Visual Stimuli in Augmented Vision Systems.- Neural Network Architecture for Modeling the Joint Visual Perception of Orientation, Motion, and Depth.- Integrating Information from Multiple Channels.- AutoSelect: What You Want Is What You Get: Real-Time Processing of Visual Attention and Affect.- Emotion Recognition Using Physiological and Speech Signal in Short-Term Observation.- Visual and Auditory Displays Driven by Perceptive Principles.- Visual Attention in Auditory Display.- A Perceptually Optimized Scheme for Visualizing Gene Expression Ratios with Confidence Values.- Spoken Dialogue Systems.- Combining Speech User Interfaces of Different Applications.- Learning and Forgetting of Speech Commands in Automotive Environments.- Help Strategies for Speech Dialogue Systems in Automotive Environments.- Multimodal and Situated Dialogue Systems.- Information Fusion for Visual Reference Resolution in Dynamic Situated Dialogue.- Speech and 2D Deictic Gesture Reference to Virtual Scenes.- Combining Modality Theory and Context Models.- Integration of Perceptive Technologies and Animation.- Visual Interaction in Natural Human-Machine Dialogue.- Multimodal Sensing, Interpretation and Copying of Movements by a Virtual Agent.- Poster Session.- Perception of Dynamic Facial Expressions of Emotion.- Multi-level Face Tracking for Estimating Human Head Orientation in Video Sequences.- The Effect of Prosodic Features on the Interpretation of Synthesised Backchannels.- Unsupervised Learning of Spatio-temporal Primitives of Emotional Gait.- System Demonstrations.- Talking with Higgins: Research Challenges in a Spoken Dialogue System.- Location-Based Interaction with Children for Edutainment.- An Immersive Game - Augsburg Cityrun.- Gaze-Contingent Spatio-temporal Filtering in a Head-Mounted Display.- A Single-Camera Remote Eye Tracker.- Miniature 3D TOF Camera for Real-Time Imaging.

A stochastic case frame approach for natural language understanding

A stochastically based approach for the semantic analysis component of a natural spoken language system for the ARPA Air Travel Information Services (ATIS) task has been developed. The semantic analyzer of the spoken language system already in use at LIMSI makes use of a rule-based case grammar. In this work, the system of rules for the semantic analysis is replaced with a relatively simple first-order hidden Markov model. The performances of the two approaches can be compared because they use identical semantic representations, despite their rather different methods for meaning extraction. We use an evaluation methodology that assesses performance at different semantic levels, including the database response comparison used in the ARPA ATIS paradigm.

Bandwidth Extension of Speech Signals

Bandwidth Extension of Speech Signals describes the theory and methods for quality enhancement of clean speech signals and distorted speech signals such as those that have undergone a band limitation, for instance, in a telephone network. Problems and the respective solutions are discussed for the different approaches. The different approaches are evaluated and a real-time implementation of the most promising approach is presented. The book includes topics related to speech coding, pattern- / speech recognition, speech enhancement, statistics and digital signal processing in general.

Perception in Multimodal Dialogue Systems

Invited Keynote.- Whence and Whither: The Automatic Recognition of Emotions in Speech (Invited Keynote).- Multimodal and Spoken Dialogue Systems.- A Generic Spoken Dialogue Manager Applied to an Interactive 2D Game.- Adaptive Dialogue Management in the NIMITEK Prototype System.- Adaptive Search Results Personalized by a Fuzzy Recommendation Approach.- Factors Influencing Modality Choice in Multimodal Applications.- Codebook Design for Speech Guided Car Infotainment Systems.- Evaluating Text Normalization for Speech-Based Media Selection.- Classification of Spoken Utterances and Sound.- A Two Phases Statistical Approach for Dialog Management.- Detecting Problematic Dialogs with Automated Agents.- Call Classification with Hundreds of Classes and Hundred Thousands of Training Utterances ... ... and No Target Domain Data.- Hard vs. Fuzzy Clustering for Speech Utterance Categorization.- Static and Dynamic Modelling for the Recognition of Non-verbal Vocalisations in Conversational Speech.- Recognition of Eye Gaze, Head Pose, Mimics and Lip Movements.- Writing with Your Eye: A Dwell Time Free Writing System Adapted to the Nature of Human Eye Gaze.- Unsupervised Learning of Head Pose through Spike-Timing Dependent Plasticity.- Spoken Word Recognition from Side of Face Using Infrared Lip Movement Sensor.- Neurobiologically Inspired, Multimodal Intention Recognition for Technical Communication Systems (NIMITEK).- Speech Recognition.- Deploying DSR Technology on Todays Mobile Phones: A Feasibility Study.- Real-Time Recognition of Isolated Vowels.- Improving Robustness in Jacobian Adaptation for Noisy Speech Recognition.- Comparing Linear Feature Space Transformations for Correlated Features.- Vocal Emotion Recognition and Annotation.- EmoVoice - A Framework for Online Recognition of Emotions from Voice.- Real-Time Emotion Recognition Using Echo State Networks.- Emotion Classification of Audio Signals Using Ensemble of Support Vector Machines.- On the Influence of Phonetic Content Variation for Acoustic Emotion Recognition.- On the Use of Kappa Coefficients to Measure the Reliability of the Annotation of Non-acted Emotions.- Annotation of Emotion in Dialogue: The Emotion in Cooperation Project.- Human-Like Social Dialogue.- Potential Benefits of Human-Like Dialogue Behaviour in the Call Routing Domain.- Human-Likeness in Utterance Generation: Effects of Variability.- Designing Socially Aware Conversational Agents.- A Prototype for Future Spoken Dialog Systems Using an Embodied Conversational Agent.- Innovative Interfaces in MonAMI: The Reminder.- Evaluation Methods.- Evaluation Methods for Multimodal Systems: A Comparison of Standardized Usability Questionnaires.- Subjective Evaluation Method for Speech-Based Uni- and Multimodal Applications.- Weighting the Coefficients in PARADISE Models to Increase Their Generalizability.- EXPROS: A Toolkit for Exploratory Experimentation with Prosody in Customized Diphone Voices.- Automatic Evaluation Tool for Multimodal Dialogue Systems.- Towards a Perception-Based Evaluation Model for Spoken Dialogue Systems.

Spoken Multimodal Human-Computer Dialogue in Mobile Environments

Contents and Contributors: PREFACE CONTRIBUTING AUTHORS INTRODUCTION Part I: Issues in Multimodal Spoken Dialogue Systems and Components ALEXANDER I . RUDNICKY / Multimodal Dialogue Systems SADAOKI FURUI / Speech Recognition Technology in Multimodal/Ubiquitous Computing Environments SATOSHI TAMURA, KOJI IWANO, SADAOKI FURUI / A Robust Multimodal Speech Recognition Method using Optical Flow Analysis KLAUS MACHEREY, HERMANN NEY / Feature Functions fro Tree-Based Dialogue Course Management DIRK BUEHLER, WOLFGANG MINKER / A Reasoning Component for Information-Seeking and Planning Dialogues JONAS BESKOW, JENS EDLUND, MAGNUS NORDSTRAND / A Model for Multimodal Dialogue System Output Applied to an Animated Talking Head Part II: System Architecture and Example Implementations ANDREAS KELLNER / Overview of System Architecture KOUICHI KATSURADA, HIROBUMI YAMADA, YUSAKU NAKAMURA, SATOSHI KOBAYASHI, TSUNEO NITTA / XISL: A Modality-Independent MMI Description Language GEORG NIKLFELD, MICHAEL PUCHER, ROBERT FINAN, WOLFGANG ECKHART / A Path to Multimodal Data Services for Telecommunications ROBERTO PIERACCINI, BOB CARPENTER, ERIC WOUDENBERG, SASHA CASKEY, STEPHEN SPRINGER, JONATHAN BLOOM, MICHAEL PHILIPS / Multimodal Spoken Dialogue with Wireless Devices DIRK BUEHLER, WOLFGANG MINKER / The SmartKom Mobile Car Prototype System for Flexible Human-Machine Communication DAN BOHUS, ALEXANDER I. RUDNICKY / LARRI: A Language-Based Maintenance and Repair Assistant Part III: Evaluation and Usability LAILA DYBKJAER, NIELS OLE BERNSEN, WOLFGANG MINKER / Overview of Evaluation and Usability STEVE WITHAKER, MARILYN WALKER / Evaluating Dialogue Strategies in Multimodal Dialogue Systems NIELS OLE BERNSEN, LAILA DYBKJAER / Enhancing the Usability of Multimodal VirtualCo-drivers WOLFGANG MINKER, UDO HAIBER, PAUL HEISTERKAMP, SVEN SCHEIBLE / Design, Implementation and Evaluation of the SENECA Spoken Language Dialogue System SABINE GELDOF, ROBERT DALE / Segmenting Route Descriptions for Mobile Devices JANIENKE STURM, BERT CRANEN, JACQUES TERKEN, ILSE BAKX / Effects of Prolonged Use on the Usability of a Mutimodal Form-Filling Interface ANTHONY JAMESON, KERSTIN KLOECKNER / User Multitasking with Mobile Multimodal Systems SHARON OVIATT, COURTNEY DARVES, RACHEL COULSTON, MATT WESSON / Speech Convergence with Animated Personas INDEX

Emotion recognition and adaptation in spoken dialogue systems

The involvement of emotional states in intelligent spoken human-computer interfaces has evolved to a recent field of research. In this article we describe the enhancements and optimizations of a speech-based emotion recognizer jointly operating with automatic speech recognition. We argue that the knowledge about the textual content of an utterance can improve the recognition of the emotional content. Having outlined the experimental setup we present results and demonstrate the capability of a post-processing algorithm combining multiple speech-emotion recognizers. For the dialogue management we propose a stochastic approach comprising a dialogue model and an emotional model interfering with each other in a combined dialogue-emotion model. These models are trained from dialogue corpora and being assigned different weighting factors they determine the course of the dialogue.