Gabriel Amores

A MULTIMODAL ARCHITECTURE FOR HOME CONTROL BY DISABLED USERS

This paper describes the architecture of MIMUS, a multimodal, multilingual interaction system which allows users to control the devices in their houses, interacting by voice and clicks. MIMUS design relies on Wizard of Oz experiments and is targeted at disabled users in the in-home scenario. MIMUS consists of a set of OAA collaborative agents with a clear distinction between knowledge resources, dialogue management and domain specific functionalities. The MIMUS system includes contributions to the scientific community in multimodal fusion, knowledge management and presentation strategies.

Multimodal interaction analysis in a smart house

This is a large extension to a previous paper presented in LREC 2006 [6]. It describes the motivation, collection and format of the MIMUS corpus, as well as an in-depth and issue-focused analysis of the data. MIMUS [8] is the result of multimodal WoZ experiments conducted at the University of Seville as part of the TALK project. The main objective of the MIMUS corpus was to gather information about different users and their performance, preferences and usage of a multimodal multilingual natural dialogue system in the Smart Home scenario in Spanish. The focus group is composed by wheel-chair-bound users, because of their special motivation to use this kind of technology, along with their specific needs. Throughout this article, the WoZ platform, experiments, methodology, annotation schemes and tools, and all relevant data will be discussed, as well as the results of the in-depth analysis of these data. The corpus compresses a set of three related experiments. Due to the limited scope of this article, only some results related to the first two experiments (1A and 1B) will be discussed. This article will focus on subjects preferences, multimodal behavioural patterns and willingness to use this kind of technology.

Integrating aggregation strategies in an in-home domain dialogue system

This paper presents the integration of a natural language generation system onto an In-Home Domain Dialogue System to achieve fluent, nonredundant verbal descriptions of the state of the environment. Three important contributions are brought together in this integration: an in-depth study of aggregation strategies preferred by users in the In-Home Domain, a fully operational dialogue system, and a natural language generation system capable of implementing the required aggregation strategies. The integration is validated by means of acceptance tests with human evaluators. In this paper we show how the aggregation strategies remove redundancies and provide a description that is assigned higher scores by human evaluators than prior descriptions.

Modeling Multimodal Multitasking in a Smart House

This paper belongs to an ongoing series of papers presented in different conferences illustrating the results obtained from the analysis of the MIMUS corpus. This corpus is the result of a number of WoZ experiments conducted at the University of Seville as part of the TALK Project. The main objective of the MIMUS corpus was to gather information about different users and their performance, preferences and usage of a multimodal multilingual natural dialogue system in the Smart Home scenario. The focus group is composed by wheel-chair- bound users. In previous papers the corpus and all relevant information related to it has been analyzed in depth. In this paper, we will focus on multimodal multitasking during the experiments, that is, modeling how users may perform more than one task in parallel. These results may help us envision the importance of discriminating complementary vs. independent simultaneous events in multimodal systems. This gains more relevance when we take into account the likelihood of the co- occurrence of these events, and the fact that humans tend to multitask when they are sufficiently comfortable with the tools they are handling.