Lea Landucci

Introducing tangerine: a tangible interactive natural environment

In this paper we describe TANGerINE, a tangible tabletop environment in which users can interact with digital contents manipulating tangible smart objects. Such objects provide continuous data about their status through the embedded wireless sensors, while an overhead computer vision module tracks their position and orientation. Merging sensing data, the system is able to detect a richer language of gestures and manipulations both on the tabletop and in its surroundings, enabling for a more expressive interaction language across different contexts.

wikiTable: finger driven interaction for collaborative knowledge-building workspaces

We present an interactive workspace featuring visionbased gesture recognition that allows multiple users to collaborate in the creation of a concept map. The workspace integrates web-based and face-to-face scenarios in knowledge-building activities like brainstorming or problem solving sessions. A wiki serves as the repository for knowledge elements, which are presented to co-located users in form of a concept map visualized on a table. The computer vision module tracks multiple hands and fingers on the table surface using fingertip detection and tracking algorithm. The concept map and the wiki are synchronized in real-time, providing notifications to both co-located and distributed users and allowing a community shared awareness that enhances and enriches the knowledge building experience.

Personalized multimedia content delivery on an interactive table by passive observation of museum visitors

The amount of multimedia data collected in museum databases is growing fast, while the capacity of museums to display information to visitors is acutely limited by physical space. Museums must seek the perfect balance of information given on individual pieces in order to provide sufficient information to aid visitor understanding while maintaining sparse usage of the walls and guaranteeing high appreciation of the exhibit. Moreover, museums often target the interests of average visitors instead of the entire spectrum of different interests each individual visitor might have. Finally, visiting a museum should not be an experience contained in the physical space of the museum but a door opened onto a broader context of related artworks, authors, artistic trends, etc. In this paper we describe the MNEMOSYNE system that attempts to address these issues through a new multimedia museum experience. Based on passive observation, the system builds a profile of the artworks of interest for each visitor. These profiles of interest are then used to drive an interactive table that personalizes multimedia content delivery. The natural user interface on the interactive table uses the visitor’s profile, an ontology of museum content and a recommendation system to personalize exploration of multimedia content. At the end of their visit, the visitor can take home a personalized summary of their visit on a custom mobile application. In this article we describe in detail each component of our approach as well as the first field trials of our prototype system built and deployed at our permanent exhibition space at LeMurate (http://www.lemurate.comune.fi.it/lemurate/) in Florence together with the first results of the evaluation process during the official installation in the National Museum of Bargello (http://www.uffizi.firenze.it/musei/?m=bargello).

Natural interaction on tabletops

We present two different Computer Vision based systems that enable multiple users to concurrently manipulate graphic objects presented over tabletop displays. The two solutions have different hardware layouts and use two different algorithms for gesture analysis and recognition. The first one is a media-handling application that can be used by co-located and remote users. The second is a knowledge-building application where users can manipulate the contents of a wiki as a visual concept map. The performance of both systems is evaluated and expounded. A conceptual framework is introduced, providing the fundamental guidelines for the design of natural interaction languages on tabletops.

MNEMOSYNE: Enhancing the Museum Experience through Interactive Media and Visual Profiling

MNEMOSYNE is a three year project whose primary goal is to deliver a personalized, interactive multimedia experience to museum visitors through the novel application of personalization driven by computer vision-based profiling. A combination of passive, wall-mounted cameras and sensors carried by guests acquiring active and passive imagery will be used to create a general profile of a museum visitor’s interests in order to customize the presentation at interactive tabletop surfaces placed in the museum environment. In this article we discuss the general context in which MNEMOSYNE is defined, as well as the main technical directions the project will follow over the next three years. Some very preliminary results are given for the vision-based techniques to be used for visual profiling of museum visitors.

Passive Profiling and Natural Interaction Metaphors for Personalized Multimedia Museum Experiences

Museums must balance the amount of information given on individual pieces or exhibitions in order to provide sufficient information to aid visitor understanding. At the same time they must avoid cluttering the environment and reducing the enjoyment of the exhibit. Moreover, each visitor has different interests and each might prefer more (or less) information on different artworks depending on their individual profile of interest. Finally, visiting a museum should not be a closed experience but a door opened onto a broader context of related artworks, authors, artistic trends, etc. In this paper we describe the MNEMOSYNE system that attempts to provide such a museum experience. Based on passive observation of visitors, the system builds a profile of the artworks of interest for each visitor. These profiles of interest are then used to personalize content delivery on an interactive table. The natural user interface on the interactive table uses the visitors profile, a museum content ontology and a recommendation system to personalize the users exploration of available multimedia content. At the end of their visit, the visitor can take home a personalized summary of their visit on a custom mobile application. In this article we describe each component of our approach as well as the first field trials of our prototype system built and deployed at our permanent exhibition space at Le Murate in the city of Florence.

Multi-User Natural Interaction System based on Real-Time Hand Tracking and Gesture Recognition

We present a computer vision based system that enables multiple people to interact naturally with a large display table using their own bare-hand gestures. The display presents and supports a particular multimedia application that can be used at the same time even by remote users. Finally we describe two different applications designed for didactic and entertainment scenarios

Imaging Novecento. A Mobile App for Automatic Recognition of Artworks and Transfer of Artistic Styles

Imaging Novecento is a native mobile application that can be used to get insights on artworks in the “Museo Novecento” in Florence, IT. The App provides smart paradigms of interaction to ease the learning of the Italian art history of the 20\(^{th}\) century. Imaging Novecento exploits automatic approaches and gamification techniques with recreational and educational purposes. Its main goal is to reduce the cognitive effort of users versus the complexity and the numerosity of artworks present in the museum. To achieve this the App provides automatic artwork recognition. It also uses gaming, in terms of a playful user interface which features state-of-the-art algorithms for artistic style transfer. Automated processes are exploited as a mean to attract visitors, approaching them to even lesser known aspects of the history of art.

Onna Project: A Natural Interaction Installation and Mobile Solution for Cultural Heritage

This paper describes the design and development of Onna—Past, Present and Future, a multimedia installation for the museum of Onna (L’Aquila, Italy), a space dedicated to the memory of the town affected by the earthquake in April 2009. Our work consists of a natural interaction system, a user profiling system and a mobile application integrated together in order to provide visitors of the museum a multi-modal experience of the events related to the disaster. Tourists and citizens of Onna are for first invited to visit the interactive installation in which multimedia assets about history, architecture and life of the town are presented. User activity is recorded by a profiling system in order to extract a profile of interest for each visitor. In a second stage of the experience users leave the museum and visit the town. They can start the application on their smartphone, which connects to the indoor profiling system via an Internet connection and then shows suggested personalized and geo-located in-depth information based on the actions performed during the session with the interactive exhibit in the museum.

Interactive multi-user video retrieval systems

In this paper we present two interactive multi-user systems for video search and browsing. The first is composed by web applications which allows multiuser interaction in a distributed environment; such applications are based on the Rich Internet Application paradigm, designed to obtain the levels of responsiveness and interactivity typical of a desktop application. The second system implements a multi-user collaborative application within a single location, exploiting multi-touch devices. Both systems use the same backend, based on a service oriented architecture (SOA) that provides services for automatic and manual annotation, and an ontology-based video search and browsing engine. Ontology-based browsing let users to inspect the content of video collections; user queries are expanded through ontology reasoning. User-centered field trials of the systems, conducted to assess the user experience and satisfaction, have shown that the approach followed to design these interfaces is extremely appreciated by professional archivists and people working on multimedia.