Gianpaolo D'Amico

Personalized multimedia content delivery on an interactive table by passive observation of museum visitors

The amount of multimedia data collected in museum databases is growing fast, while the capacity of museums to display information to visitors is acutely limited by physical space. Museums must seek the perfect balance of information given on individual pieces in order to provide sufficient information to aid visitor understanding while maintaining sparse usage of the walls and guaranteeing high appreciation of the exhibit. Moreover, museums often target the interests of average visitors instead of the entire spectrum of different interests each individual visitor might have. Finally, visiting a museum should not be an experience contained in the physical space of the museum but a door opened onto a broader context of related artworks, authors, artistic trends, etc. In this paper we describe the MNEMOSYNE system that attempts to address these issues through a new multimedia museum experience. Based on passive observation, the system builds a profile of the artworks of interest for each visitor. These profiles of interest are then used to drive an interactive table that personalizes multimedia content delivery. The natural user interface on the interactive table uses the visitor’s profile, an ontology of museum content and a recommendation system to personalize exploration of multimedia content. At the end of their visit, the visitor can take home a personalized summary of their visit on a custom mobile application. In this article we describe in detail each component of our approach as well as the first field trials of our prototype system built and deployed at our permanent exhibition space at LeMurate (http://www.lemurate.comune.fi.it/lemurate/) in Florence together with the first results of the evaluation process during the official installation in the National Museum of Bargello (http://www.uffizi.firenze.it/musei/?m=bargello).

Sirio, orione and pan: an integrated web system for ontology-based video search and annotation

In this technical demonstration we show an integrated web system for video search and annotation based on ontologies. The system is composed by three components: the Orione ontology-based search engine, the Sirio\footnote{Sirio was the hound of Orione. It was a dog so swift that no prey could escape it.} search interface, and the Pan web-based video annotation tool. The system is currently being developed within the EU IM3I project. The goal of the system is to provide an integrated environment for video annotation and retrieval of videos, for both technical and non-technical users. In fact, the search engine has different interfaces that permit different query modalities: free-text, natural language, graphical composition of concepts using Boolean and temporal relations and query by visual example. In addition, the ontology structure is exploited to encode semantic relations between concepts permitting, for example, to expand queries to synonyms and concept specializations. The annotation tool can be used to create ground-truth annotations to train automatic annotations systems, or to complement the results of automatic annotation, e.g. adding geolocalized information.

3D content-based retrieval with spin images

Along with images and videos, 3D models have recently gained increasing attention for a number of reasons: advancements in 3D hardware and software technologies; their ever decreasing prices and increasing availability; affordable 3D authoring tools; the establishment of open standards for 3D data interchange. The ever increasing availability of 3D models demands tools to support their effective and efficient management. Among these tools, those enabling content-based retrieval play a key role. We present a novel approach to 3D content-based retrieval that is based on spin images. Spin images are used to derive a view-independent description of both database and query objects: a set of spin images is first created for each object; then, a descriptor is evaluated for each spin image in the set; clustering is performed on the set of image-based descriptors of each object to achieve a compact representation of the object, thus allowing for efficient indexing and matching. Experimental results are presented for a test database of about 300 models. These results indicate that spin images can be successfully exploited for content-based retrieval of 3D objects

Arneb: a rich internet application for ground truth annotation of videos

In this technical demonstration we show the current version of Arneb, a web-based system for manual annotation of videos, developed within the EU VidiVideo project. This tool has been developed with the aim of creating ground truth annotations, that can be used for training and evaluating automatic video annotation systems. Annotations can be exported to MPEG-7 and OWL ontologies. The system has been developed according to the Rich Internet Application paradigm, allowing collaborative web-based annotation.

Passive Profiling and Natural Interaction Metaphors for Personalized Multimedia Museum Experiences

Museums must balance the amount of information given on individual pieces or exhibitions in order to provide sufficient information to aid visitor understanding. At the same time they must avoid cluttering the environment and reducing the enjoyment of the exhibit. Moreover, each visitor has different interests and each might prefer more (or less) information on different artworks depending on their individual profile of interest. Finally, visiting a museum should not be a closed experience but a door opened onto a broader context of related artworks, authors, artistic trends, etc. In this paper we describe the MNEMOSYNE system that attempts to provide such a museum experience. Based on passive observation of visitors, the system builds a profile of the artworks of interest for each visitor. These profiles of interest are then used to personalize content delivery on an interactive table. The natural user interface on the interactive table uses the visitors profile, a museum content ontology and a recommendation system to personalize the users exploration of available multimedia content. At the end of their visit, the visitor can take home a personalized summary of their visit on a custom mobile application. In this article we describe each component of our approach as well as the first field trials of our prototype system built and deployed at our permanent exhibition space at Le Murate in the city of Florence.

Web-based semantic browsing of video collections using multimedia ontologies

In this technical demonstration we present a novel web-based tool that allows a user friendly semantic browsing of video collections, based on ontologies, concepts, concept relations and concept clouds. The system is developed as a Rich Internet Application (RIA) to achieve a fast responsiveness and ease of use that can not be obtained by other web application paradigms, and uses streaming to access and inspect the videos. Users can also use the tool to browse the content of social and media sharing sites like YouTube, Flickr and Twitter, accessing these external resources through the ontologies used in the system. The tool has won the second prize in the Adobe YouGC contest, in the RIA category.

Shape representation by spatial partitioning for content based retrieval applications

Shape representations for classification or retrieval purposes, have been extensively investigated, but only a few methods have tried to represent shapes as extended entities without reducing them to their boundary profiles or to synthetic geometric descriptors. An original solution for shape representation is proposed; it relies on a modelling technique originally developed to express directional spatial relationships between extended spatial entities. The representation selects a discrete set of points with respect to which a relationship matrix is computed, accounting for the spatial distribution of shape pixels. This is accomplished at different levels of resolution by a tree based structural representation. Properties of the representation and a measure of shape similarity are discussed. The efficiency and effectiveness of the proposed solution have also been assessed in the context of content based retrieval applications, through an experimental evaluation using a shape collection

Sirio: an ontology-based web search engine for videos

In this technical demonstration we show a web video search engine based on ontologies, the Sirio system, that has been developed within the EU VidiVideo project. The goal of the system is to provide a search engine for videos for both technical and non-technical users. In fact, the system has different interfaces that permit different query modalities: free-text, natural language, graphical composition of concepts using boolean and temporal relations and query by visual example. In addition, the ontology structure is exploited to encode semantic relations between concepts permitting, for example, to expand queries to synonyms and concept specializations.

Interactive Visual Representations of Complex Information Structures

One of the most challenging issues in managing the large and diverse data available on the World Wide Web is the design of interactive systems to organize and represent information, according to standard usability guidelines. In this paper we define a framework to collect and represent information from different web resources like search engines, real-time networks and multimedia distributed databases. A prototype system has been developed, following the Rich Internet Application paradigm, to allow end-users to visualize, browse and analyze documents and their relationships in a graph-based user interface. Different visual paradigms have been implemented and their effectiveness has been measured in usability tests with real users.

Natural Interactive System for Hemispatial Neglect Rehabilitation

Experimentation of natural interaction principles and advanced interactive solutions for medical treatment of the neglect syndrome. Novel tools are provided for improving rehabilitation and evaluation of patients and overcome the conventional pen-and-paper approach. This project is the result of a multidisciplinary collaboration among Media Integration and Communication Center, the Faculty of Psychology of the University of Florence and Montedomini A.S.P., a public agency for disabled elders and healthcare services in Italy.