Luis Alberto Pineda

A model for multimodal reference resolution

An important aspect of the interpretation of multimodal messages is the ability to identify when the same object in the world is the referent of symbols in different modalities. To understand the caption of a picture, for instance, one needs to identify the graphical symbols that are referred to by names and pronouns in the natural language text. One way to think of this problem is in terms of the notion of anaphora; however, unlike linguistic anaphoric inference, in which antecedents for pronouns are selected from a linguistic context, in the interpretation of the textual part of multimodal messages the antecedents are selected from a graphical context. Under this view, resolving multimodal references is like resolving anaphora across modalities. Another way to see the same problem is to look at pronouns in texts about drawings as deictic. In this second view, the context of interpretation of a natural language term is defined as a set of expressions of a graphical language with well-defined syntax and semantics. Natural language and graphical terms are thought of as standing in a relation of translation similar to the translation relation that holds between natural languages. In this paper a theory based on this second view is presented. In this theory, the relations between multimodal representation and spatial deixis, on the one hand, and multimodal reasoning and deictic inference, on the other, are discussed. An integrated model of anaphoric and deictic resolution in the context of the interpretation of multimodal discourse is also advanced.

Why does public transport not arrive on time? The pervasiveness of equal headway instability

Background The equal headway instability phenomenon is pervasive in public transport systems. This instability is characterized by an aggregation of vehicles that causes inefficient service. While equal headway instability is common, it has not been studied independently of a particular scenario. However, the phenomenon is apparent in many transport systems and can be modeled and rectified in abstraction. Methodology We present a multi-agent simulation where a default method with no restrictions always leads to unstable headways. We discuss two methods that attempt to achieve equal headways, called minimum and maximum. Since one parameter of the methods depends on the passenger density, adaptive versions—where the relevant parameter is adjusted automatically—are also put forward. Our results show that the adaptive maximum method improves significantly over the default method. The model and simulation give insights of the interplay between transport design and passenger behavior. Finally, we provide technological and social suggestions for engineers and passengers to help achieve equal headways and thus reduce delays. Conclusions The equal headway instability phenomenon can be avoided with the suggested technological and social measures.

The Corpus DIMEx100: transcription and evaluation

In this paper the transcription and evaluation of the corpus DIMEx100 for Mexican Spanish is presented. First we describe the corpus and explain the linguistic and computational motivation for its design and collection process; then, the phonetic antecedents and the alphabet adopted for the transcription task are presented; the corpus has been transcribed at three different granularity levels, which are also specified in detail. The corpus statistics for each transcription level are also presented. A set of phonetic rules describing phonetic context observed empirically in spontaneous conversation is also validated with the transcription. The corpus has been used for the construction of acoustic models and a phonetic dictionary for the construction of a speech recognition system. Initial performance results suggest that the data can be used to train good quality acoustic models.

DIMEx100: A New Phonetic and Speech Corpus for Mexican Spanish

In this paper the phonetic and speech corpus DIMEx100 for Mexican Spanish is presented. We discuss both the linguistic motivation and the computational tools employed for the design, collection and transcription of the corpus. The phonetic transcription methodology is based on recent empirical studies proposing a new basic set of allophones and phonological rules for the dialect of the central part of Mexico. These phonological rules have been implemented in a visualization tool that provides the expected phonetic representation of a text, and also a default temporal alignment between the spoken corpus and its phonetic representation. The tools are also used to compute the properties of the corpus and compare these figures with previous work.

The DIME Project

In this paper a general description and current state of the project Dialogos Multimodales Inteligentes en Espanol (DIME) -Intelligent Multimodal Dialogs in Spanish- is presented. The purpose of the project is to develop a multimodal conversational agent with spoken input and output facilities in Spanish in a design oriented domain: kitchen design. In this paper, the state of the project, current results, an overview of the prototype system and future work are presented.

GRAFLOG: understanding drawings through natural language

This paper describes an experimental interactive graphics interface, GRAFLOG, in which drawings receive linguistic interpretations. It is possible to emulate linguistic interaction in situations where graphics is thought to be necessary. The paper presents examples of such a kind of dialogue and the architecture of the implementation. The paper explains how representations of drawings can be constructed by treating graphical symbols as “objects”, and how a parallel linguistic interpretation for these symbols can be constructed. It highlights the relevance of “deictic expressions” and “spatial prepositions” in building the interface mechanisms between these two kinds of representations. Lastly, it shows how a reasoning component is constructed for making deductions from premises that are found in both the graphical and linguistic domains. Using GRAFLOG, it is possible to represent knowledge through words and pictures. GRAFLOG is implemented, using an object oriented programming style, in PROLOG and GKS.

Reference, Synthesis and Constraint Satisfaction*

In this paper we discuss two kinds of constraint satisfaction problems that arise in the context of geometric modelling, In particular in the modification of 2‐D wire‐frame diagrams that are subject to an arbitrary number of geometrical and topological constraints. We argue that problems in this domain can be classified in two categories that we shall call problems of reference and problems of synthesis. Since Sutherlands Sketchpad program [16], a large number of systems have addressed constraint satisfaction in terms of the representation of constraints sets as equation systems, which in turn are solved by numerical methods like local propagation, relaxation and Gaussian elimination. Here, we present an alternative framework. We argue that conceptualising constraint satisfaction as symbolic rather than “numerical” problems helps to clarify the notion of “constraint”, simplify solution methods, and to explain the intuitive inferential processes underlying the modification of drawings in the course of interactive drafting sessions. The theory presented in this paper has been tested with an experimental computer program called Graflog [5, 8, 9, 10, 11, 12]. The program has been implemented during the last four years, and has evolved through several stages. The current version is implemented in terms of two Unix‐processes connected by Unix‐pipes. The first is a “C” program running X windows, and handles the external aspects of the interaction. The second is a Prolog program supporting the representational structures and interpreters of the system.

Dialogue model specification and interpretation for intelligent multimodal HCI

A methodology and programming environment for the specification and interpretation of dialogue models for grounded multimodal interaction is presented. This conceptual framework permits the declarative specification of complex interactive systems with multimodal input and output, including speech, computer vision and motor behavior. We first introduce the present notion of dialogue model with its motivation on the structure of conversation. Then, the specification and interpretation of dialogue models is presented and discussed. We also present a cognitive architecture for the construction of intelligent Human-Computer Interaction (HCI) applications within this conceptual framework. The paper concludes with references to working systems, demos and work in progress built within the present framework.

Development of a tour-guide robot using dialogue models and a cognitive architecture

In this paper, we present the development of a tour-guide robot that conducts a poster session through spoken Spanish. The robot is able to navigate around its environment, visually identify informational posters, and explain sections of the posters that users request via pointing gestures. We specify the task by means of dialogue models. A dialogue model defines conversational situations, expectations and robot actions. Dialogue models are integrated into a novel cognitive architecture that allow us to coordinate both human-robot interaction and robot capabilities in a flexible and simple manner. Our robot also incorporates a confidence score on visual outcomes, the history of the conversation and error prevention strategies. Our initial evaluation of the dialogue structure shows the reliability of the overall approach, and the suitability of our dialogue model and architecture to represent complex human-robot interactions, with promising results.

SitLog: A Programming Language for Service Robot Tasks

In this paper we present SitLog: a declarative situation-oriented logical language for programming situated service robot tasks. The formalism is task and domain independent, and can be used in a wide variety of settings. SitLog can also be seen as a behaviour engineering specification and interpretation formalism to support action selection by autonomous agents during the execution of complex tasks. The language combines the recursive transition network formalism, extended with functions to express dynamic and contextualized task structures, with a functional language to express control and content information. The SitLog interpreter is written in Prolog and SitLogs programs follow closely the Prolog notation, permitting the declarative specification and direct interpretation of complex applications in a modular and compact form. We discuss the structure and representation of service robot tasks in practical settings and how these can be expressed in SitLog. The present framework has been tested in the service robot Golem-II+ using the specification and programming of the typical tasks which require completion in the RoboCup@Home Competition.