Jacques Siroux

Multimodal references in GEORAL Tactile

The paper specifically presents how linguistic (oral) and tactile references are dealt with in the GEORAL system which has already been described in other papers. In this system, users can formulate their queries and provide their responses using the oral (linguistic) modality and the tactile modality separately or together. We describe the referential phenomena which occur in such a context and we point out why the oral modality has to be the basis of the processing of the references and why robustness problems have to be dealt with. We then provide details about the three steps of the reference processing (linguistic analysis, processing of the tactile events and merging process) as well as the modeling of the communicative acts used in the system (as planning operators).

Knowledge and data flow architecture for reference processing in multimodal dialog systems

This paper is concerned with the part of the system dedicated to the processing of the users designation activities for multimodal search of information. We highlight the necessity of using specific knowledge for multimodal input processing. We propose and describe knowledge modeling as well as the associated processing architecture. Knowledge modeling is concerned with the natural language and the visual context; it is adapted to the kind of application and allows several types of filtering of the inputs. Part of this knowledge is dynamically updated to take into account the interaction history. In the proposed architecture, each input modality is processed first by using the modeled knowledge, producing intermediate structures. Next a fusion of these structures allows the determination of the referent aimed at by using dynamic knowledge. The steps of this last process take into account the possible combinations of modalities as well as the clues carried by each modality (linguistic clues, gesture type). The development of this part of our system is mainly complete and tested.

Modeling and Processing of Oral and Tactile Activities in the GEORAL System

This paper presents the processing of tactile and oral multimodal events that occur in inputs of the GEORAL tactile system. First we shall describe the principal results of observations on the behaviour of users in a multimodal interaction environment. These observations, combined with consideration of the technical possibilities and limitations of speech recognition, lead us to proposing a system architecture. The chapter ends with details of the modeling of communicative acts, based on plan operators, and the principal stages of the merging of tactile and oral acts.

A model for multimodal representation and processing for reference resolution

We present a model for dealing with designation activities of a user in multimodal systems. This model associates both a well defined language to each modality (NL, gesture, visual) and a mediator one. It takes into account several semantic features of modalities. Functions link objects from each modality to another, and allow reasoning and referent identification. Processing algorithms related to each language are developed.

Le dialogue homme-machine en langue naturelle: un défi?

RésuméLa mise en œuvre d’un dialogue homme-machine dans un système d’interrogation de base de données ou de base de connaissances est un problème difficile. Les auteurs présentent ici trois approches de ce problème : la modélisation du dialogue, illustrée par un travail linguistique et par des réalisations informatiques ; les principes de coopération dans le dialogue et leurs algorithmes associés ; et, enfin, l’approche intelligence artificielle fondée sur les actes de langage et la génération de plans. Pour terminer, ils abordent l’état actuel des possibilités d’utilisation des technologies vocales dans le cadre du dialogue homme-machine et les problèmes restant àrésoudre.AbstractThe implementation of a natural language manmachine dialogue in a data base or knowledge base query system is a difficult task. We present hereafter three different approaches to this problem : dialog modeling, exemplified through a linguistic scheme and several computational implementations ; the cooperation principles in a dialog and their associated algorithms ; and, lastly, the artificial intelligence approach based on speech acts theory and the techniques for generating and recognizing plans. We conclude by assessing the possible use of current speech processing techniques and by listing still unsolved problems.

Modeling and Using Salience in Multimodal Interaction Systems

We are interested in input to human-machine multimodal interaction systems for geographical information search. In our context of study, the system offers to the user the ability of using speech, gesture and visual modes. The system displays a map on the screen, the user ask the system about sites (hotels, campsites, ...) by specifying a place of search. Referenced places are objects in the visual context like cities, road, river, etc. The system should determine the designated object to complete the understanding process of users request. In this context, we aim to improve the reference resolution process while taking into account ambiguous designations. In this paper, we focus on the modeling of visual context. In this modeling we take into account the notion of salience, its role in the designation and in the processing methods.

Compréhension automatique de la parole et intelligence artificielle

RésuméL’article présente la discipline de la compréhension automatique de la parole et ses rapports avec l’intelligence artificielle. Après une introduction destinée à situer ce qu’il semble pour le moment raisonnable d’espérer d’une machine dans un tel domaine, on donne un bref historique des techniques utilisées en reconnaissance de la parole. On présente ensuite les divers endroits où l’intelligence artificielle peut être utile pour ce problème et on détaille deux importantes applications de ses techniques á la compréhension de la parole: le d’ecodage acoustico-phonétique et la modélisation du dialogue oral.AbstractIn this article, we introduce the problem of automatic speech understanding and its relationship with artificial intelligence. We first present what is now believed to be reasonable to ask computers in such a domain, and then give a short history of speech recognition. Then we indicate the various places where artificial intelligence may be useful for designing a speech understanding system and we give more details about its application to two particular problems: the acousticophonetic decoding, and the model designing of oral dialogue.