Francesco Cutugno

An extensible architecture for robust multimodal human-robot communication

Human safety and effective human-robot communication are main concerns in HRI applications. In order to achieve such goals, a system should be very robust, allowing little chance for misunderstanding the users commands. Moreover, the system should permit natural interaction reducing the time and the effort needed to achieve tasks. The main purpose of this work is to develop a general framework for flexible and multimodal human-robot communication. The proposed architecture should be easy to modify and expand, adding or modifying input channels and changing the multimodal fusion strategies. In this paper, we introduce our general approach and provide a case study with two modalities (gesture and speech).

A dialogue system for multimodal human-robot interaction

This paper presents a POMDP-based dialogue system for multimodal human-robot interaction (HRI). Our aim is to exploit a dialogical paradigm to allow a natural and robust interaction between the human and the robot. The proposed dialogue system should improve the robustness and the flexibility of the overall interactive system, including multimodal fusion, interpretation, and decision-making. The dialogue is represented as a Partially Observable Markov Decision Process (POMDPs) to cast the inherent communication ambiguity and noise into the dialogue model. POMDPs have been used in spoken dialogue systems, mainly for tourist information services, but their application to multimodal human-robot interaction is novel. This paper presents the proposed model for dialogue representation and the methodology used to compute a dialogue strategy. The whole architecture has been integrated on a mobile robot platform and has bee n tested in a human-robot interaction scenario to assess the overall performances with respect to baseline controllers.

Multimodal framework for mobile interaction

In recent years multimodal interaction is becoming of great interest thanks to the increasing availability of mobile devices. In this view, many applications making use of speech, gestures on the touch screen and other interaction modalities are presently becoming to appear on the different app-markets. Multimodality requires procedures to integrate different events to be interpreted as a single intention of the user. There is no agreement on how this integration must be realized as well as a shared approach, able to abstract a set of basic functions to be used in any possible multimodal application, is still missing. Designing and implementing multimodal systems is still a difficult task. In response to this situation, the goal of our research is to explore how a simple framework can be used to support the design of multimodal user interfaces. In this paper we propose a framework that aims to help the design of simple multimodal commands in the mobile environment (more specifically in Android applications). The proposed system is based on the standard licensed by the W3C consortium for the Multimodal Interaction [8] [9] and on the definition of a set of CARE [2] properties; moreover the system makes use of some features available in the SMUIML language [3]. We will finally present a case study implementing a mobile GIS application based on the proposed framework.

Caruso: Interactive headphones for a dynamic 3D audio application in the cultural heritage context

One of the most important qualities of the physical environment in which humans live is spatial dimensionality. Talking about 3D, we usually think of 3D video, even if it is not the only existing channel of natural interaction. In this paper, we present an interaction system based on spatialized sounds. We developed an application in the cultural heritage context; a personal guide, in 3D sound, attracting the tourists attention toward monuments or buildings, offering soundscapes of augmented reality. The designed system interacts with smart headphones, remotely takes the orientation of the listeners head and properly generates an audio output, which also takes into account the listeners position and orientation in the environment. Thus, an innovative headphones using an inertial measurement unit for determination of the orientation of a users head have been designed and developed in open-ear mode, in order to locate the user in the real context.

CoWME: a general framework to evaluate cognitive workload during multimodal interaction

Evaluating human machine interaction in the case of multimodal systems is often a difficult task involving the monitoring of multiple sources, data fusion and results interpretation. While subtasks are highly dependent on the specific goal of the application and on the available interaction modalities, it is possible to formalize this workflow into a standard process and to consider a generic measure to estimate the ease of use of a specific application. In this work, we present CoWME, a modular software architecture describing multimodal human machine interaction evaluation, from data collection to final evaluation, in a formal way, in terms of cognitive workload. Communication protocols between modules are described in XML while data fusion is delegated to a configurable rule engine. An interface module is introduced between the monitoring modules and the rule engine to collect and summarize data streams for cognitive workload evaluation. We present a deployment example showing how this architecture is deployed by monitoring an interactive session with an Android application taking into account stressed speech detection, mydriasis and touch analysis.

E.Y.E. C. U.: an Emotional eYe trackEr for Cultural heritage sUpport

Enjoying a painting, a sculpture or, more in general, a piece of art and, at the same time, to receive all the information you need about it: in this paper, we present E.Y.E. C. U. (read “I see you”), a modular eye tracking system which supports art galleries fruition without diverting visitors attention. Every time a visitor lingers on a painting detail, a hidden camera detects her gaze and the framework beams, in real time, the related illustrative contents on the wall region around it, deeply implementing the augmented reality meaning. E.Y.E. C. U. enhances the gaze detection functionalities with an emotional analysis module: as pupil is well known to reflect the emotional arousal, we monitor its size, in order to detect radius variations. Once the visitor has completed her visit, the system summarizes the observed details and the emotional reactions in a report.

ICT Solutions for the OR.C.HE.S.T.R.A. Project: From Personalized Selection to Enhanced Fruition of Cultural Heritage Data

This work discusses the ICT solutions designed and developed within the OR.C.HE.S.T.R.A. Project. The mission of such an industrial and experimental project (Organization of Cultural Heritage and Smart Tourism and Real-time Accessibility) consists in developing some technological solutions for tourists and inhabitants aimed at appraising the cultural heritage of the historic centre of Naples. The project is based on a Social Innovation approach where services are created engaging all the possible actors in an ecosystem oriented to Smart Culture and tourism (companies, research groups and final users). Thus, in this work some innovative solutions in the cultural heritage domain are promoted and described in order to improve at the same time both the cultural knowledge to offer to different kinds of users (for instance tourists, citizens and researchers) and its learning and its preservation and protection as well. More in detail, we describe how our developed system is able to assist users before visiting the city, by suggesting them the most interesting places to see according to their preferences, and during the visit as well, in order to make the trip more interactive and enjoyable.

AutoMyDe: A Detector for Pupil Dilation in Cognitive Load Measurement

Pupil dilation is known to reflect the emotional arousal. Pleasure, effort and fear are examples of stimuli inducing the nervous system to cause dilation mydriasis. The work proposes a tool to automatically quantify the mydriasis in order to evaluate mental effort in HCI. The system uses a feature-based approach and monitors the pupil behavior during a given task. As mydriasis is entailed by various reasons, our system distinguishes the cause-effect relationships by synchronizing monitoring and test, dividing the monitoring in fixed intervals and retrieving a survey of the mydriatic events for each determined period of time. We present a case of study analyzing users resolving arithmetical tasks, viewing pictures and using a mobile application. In each scenario, tests intend to impose gradually increasing reactions to the users. The paper will present different techniques for pupil dilation measurements and related results of mental effort evaluation.

A Framework for Distributed Interaction in Intelligent Environments

Ubiquitous computing is extending its applications to an increasing number of domains. “Monolithic” approaches use centralised systems, controlling devices and users’ requests. A different solution can be found in works proposing “distributed” intelligent devices that communicate, without a central reasoner, creating little communities to support the user. If the former approach uses all the available sensors being more easily context-aware, the latter is scalable and naturally supports multiple users.

A Framework for Interaction Design in Intelligent Environments

Ubiquitous computing is extending its applications to an increasing number of new domains. The implementation of most of the newly appeared scenarios needs to follow guidelines coming from an expert of Interaction Design (IxD). In this paper we present a work-in-progress framework to conduct human-computer and computer-computer interactions and a tool for IxD applying it, as case studies, both in home and museum scenarios. Our framework uses multi-modal interaction and smart-devices, the main computational effort is dedicated to the design of a distributed chain of control monitoring sensors, devices and humans in the environment. The framework aims at merging concepts from the Intelligent Environments with those coming from Natural User Interaction theories. Moreover, the system includes designing procedures to support developers and users to produce new application just working on higher level layers separated from the infrastructural technologies. Preliminary results, coming from tests in Wizard of Oz, highlight potentialities of our framework.