Maria Chiara Caschera

Multimodal interaction systems: information and time features

Multimodal interaction systems combine visual information (involving images, text, sketches and so on) with voice, gestures and other modalities to provide flexible and powerful dialogue approaches, enabling users to choose one or more of the multiple interaction modalities. They break down the barriers in adopting mobile devices for value-added services and the use of integrated multiple input modes enables users to benefit from the natural approach used in human communication. This paper deals with the main features of multimodal interaction and systems, starting from the definition of visual language given in Bottoni et al. (1995) and extending it to multimodality. Modal/multimodal message, interpretation and materialisation functions and multimodal sentence are defined. This paper introduces and formally defines the different classes of cooperation between different modes, introducing the time relationships among the involved modalities and the relationships between chunks of information connected with these modalities.

InteSe: An Integrated Model for Resolving Ambiguities in Multimodal Sentences

The pervasiveness of ambiguity in communication processes suggests addressing the problem of semantic and syntactic ambiguities in multimodal interaction languages. This paper presents an integrated model based on layered, hierarchical, and hidden Markov models for dealing with the complex process of multimodal ambiguity resolution. The proposed model consists of different levels, from the terminals of a multimodal language (terminal elements) to the level of multimodal sentences. A software module implemented the model that has been evaluated in terms of accuracy and robustness. The experimental results show good levels of accuracy and robustness compared with other existing approaches.

Multidimensional Visualization System for Travel Social Networks

In this paper we provide a multidimensional visualization system for travel social network. This system allows to analyse the structure and the social aggregation of travel networks providing different perspectives according to three dimensions: the space that defines locations connected to the members; the time that defines temporal evolution about tourist interests of the members; and coordinates involving classes of interests that define typologies of tourism chosen by members. Furthermore the system offers location-based tourist services providing a spatial visualization of the services connected to the specific place and sharing information about specific place. Finally in order to evaluate the proposed method, interviews to tourists and stakeholders were carried out after testing of the system.

Ambiguities in Sketch-Based Interfaces

Sketch-based interaction is an intuitive, simple communication method. However, it has several critical aspects, due to difficulties during the interpretation step by the computer side. These mainly derive from the semantic gap between the users communicative intention and how he/she is able to convey it. The interpretation must also consider the deletion and over-tracing actions. This introduces further ambiguities that make it difficult to interpret the users communicative intention. To deal with these problems, this paper provides a classification of ambiguities for sketch-based interfaces, based on experimental observations from a group of users. Ambiguities are caused by inaccuracy, approximation of the represented reality, users indecision and deletion or retracing of parts of the sketch. This paper presents some useful situations to give an overview on ambiguities by considering information strictly related to the users behaviour, considering spatial and temporal information in the drawing process and the users intention

From Modal to Multimodal Ambiguities: a Classification Approach

This paper deals with classifying ambiguities for Multimodal Languages. It evolves the classifications and the methods of the literature on ambiguities for Natural Language and Visual Language, empirically defining an original classification of ambiguities for multimodal interaction using a linguistic perspective. This classification distinguishes between Semantic and Syntactic multimodal ambiguities and their subclasses, which are intercepted using a rule-based method implemented in a software module. The experimental results have achieved an accuracy of the obtained classification compared to the expected one, which are defined by the human judgment, of 94.6% for the semantic ambiguities classes, and 92.1% for the syntactic ambiguities classes.

Studying network dynamics in digital ecosystems

One of the main fields in which the research on Digital Ecosystems has been fruitfully applied is the networking field, with the aim of discovering the dynamics of relationships among the entities of the ecosystems. Following this research direction, this paper addresses the problem of predicting social dynamics of a network in order to emphasize the relationships and the potentials for collaboration and transmission of knowledge, as well as the nature and intensity of the inner sub-networks. To do this, a multi-layer Hidden Markov Model has been applied, which allows predicting the evolution of the interests and intensity of the overall network, based on the most probable evolution of each sub-network (e.g., if it increases, decreases, appears, etc.). This model has been tested using data from a large, realistic network and the prediction accuracy rate has been evaluated.

An approach for managing ambiguities in multimodal interaction

Multimodal systems support people with different needs and different features during the interaction process making it more easy and natural. However naturalness can usually produce ambiguous interpretation. This paper discusses ambiguities in multimodal interaction proposing in particular two different examples of ambiguities: a) multimodal ambiguities produced combining un-ambiguous modal information containing contrasting concepts in multimodal dialogue, b) ambiguities arising by the propagation at multimodal level of a modal input ambiguity. Moreover, the paper proposes a formal representation of multimodal inputs using the hybrid approach that combines the attribute multi-set grammar with the linear logic. This paper starts from discussing the main features of a multimodal system and the classes of cooperation between modalities. The attribute multi-set grammar combined with the linear logic enables to highlight the multidimensionality of multimodality and allows to formally describe ambiguities that can occur.

ME: Multimodal Environment Based on Web Services Architecture

Information, documents and knowledge for each person and for public and private organizations are fundamental in each activity, and they may be the products or services they provide or supply. The daily activities and decision-making processes are usually based on many different pieces of information, which could be handled on PDAs and mobile devices in general, or stored on laptop computers using a lot of different forms, such as spreadsheets, e-mail messages, Web information obtained as the result of a Google search or a query, and so on. Simulating and managing services and information in catastrophic events and emergencies does not represent an exception. This paper describes the Web Services architecture of the Multimodal collaborative knowledge oriented Environment (ME), a platform designed to manage data, information and services for catastrophic events such as earthquakes, floods and dangerous natural phenomena.

Multimodal Interaction in Gaming

Gaming environments are applications that have the great potential to increase people engagement in a participatory and collaborative way. Players interact with games under various situations, where the content, the form, and the modalities will be manipulated to fit the player’s behaviours. This paper provides a multimodal environment for gaming by using a grammar-based approach for supporting the interaction process in the application scenario of scope card game, instantiating grammar by the elements and the rules of the game. Moreover, the paper focuses on the correct interpretation of the player’s input during the game by the use of a HMM-based approach.

An advanced multimodal platform for educational social networks

Multimediality and multimodality have demonstrated significant potential supporting students in learning activities; providing them with multimodal interaction can be a crucial issue for improving accessibility to the multimedia contents in learning environments. In fact, involving multiple input and output modalities enables a broader spectrum of users with different ages, skill levels and abilities to access these contents. In this paper, we present AMPLE, an advanced multimodal platform for e-learning that provides users with a multimodal access to an educational social network. The platform is based on a distributed architecture that integrates social networking technologies and multimodal facilities, enabling a collaborative learning experience to learners.