Bruno Tellez

Associating textual features with visual ones to improve affective image classification

Many images carry a strong emotional semantic. These last years, some investigations have been driven to automatically identify induced emotions that may arise in viewers when looking at images, based on low-level image properties. Since these features can only catch the image atmosphere, they may fail when the emotional semantic is carried by objects. Therefore additional information is needed, and we propose in this paper to make use of textual information describing the image, such as tags. Thus, we have developed two textual features to catch the text emotional meaning: one is based on the semantic distance matrix between the text and an emotional dictionary, and the other one carries the valence and arousal meanings of words. Experiments have been driven on two datasets to evaluate visual and textual features and their fusion. The results have shown that our textual features can improve the classification accuracy of affective images.

Multimodal recognition of visual concepts using histograms of textual concepts and selective weighted late fusion scheme

The text associated with images provides valuable semantic meanings about image content that can hardly be described by low-level visual features. In this paper, we propose a novel multimodal approach to automatically predict the visual concepts of images through an effective fusion of textual features along with visual ones. In contrast to the classical Bag-of-Words approach which simply relies on term frequencies, we propose a novel textual descriptor, namely the Histogram of Textual Concepts (HTC), which accounts for the relatedness of semantic concepts in accumulating the contributions of words from the image caption toward a dictionary. In addition to the popular SIFT-like features, we also evaluate a set of mid-level visual features, aiming at characterizing the harmony, dynamism and aesthetic quality of visual content, in relationship with affective concepts. Finally, a novel selective weighted late fusion (SWLF) scheme is proposed to automatically select and weight the scores from the best features according to the concept to be classified. This scheme proves particularly useful for the image annotation task with a multi-label scenario. Extensive experiments were carried out on the MIR FLICKR image collection within the ImageCLEF 2011 photo annotation challenge. Our best model, which is a late fusion of textual and visual features, achieved a MiAP (Mean interpolated Average Precision) of 43.69% and ranked 2nd out of 79 runs. We also provide comprehensive analysis of the experimental results and give some insights for future improvements.

ARCAMA-3D --- a context-aware augmented reality mobile platform for environmental discovery

In this paper, we present ARCAMA-3D, a platform for 3D map-based visualization on mobile devices powered by augmented reality. The platform offers context-aware interactions related to the concept of ubiquitous computing. The general purpose of the project is to enable users to navigate in an area with their mobile devices and interactively discover their surroundings. The system integrates real-time sensing technologies (GPS and other embedded sensors) and exploits users context and preferences in order to provide her with the necessary information. In return, the user consults the information (text, photo, audio or video files, etc.) that is published on the 3D model with the help of augmented reality. The innovative aspect of our approach lies in a light-weight 3D visualization system which is superimposed on the real scene. This approach facilitates the discovery of surroundings without preventing the visualization of real entities. We also alleviate the cognitive load of the user by avoiding the presentation of excessive information.

Updating urban database with aerial photographs: A common structuring methodology

Abstract Updated geographical information systems and especially cadastre 2 are difficult to obtain in real time. The goal of our work is to use aerial photos to automatically update a cadastre database. The first problem concerns the difficulties of managing aerial photographs in a raster format. This leads us to propose a methodology that permits one to manipulate in the same way, raster and vector data. That implies the same process (based on Delaunay triangulation) and the same model (including topology). Aerial photoprocessing and cadastre handling provide information about objects and relations between them. They are described in the structural map that will be enriched with derived spatial relations. This global common description for cadastre and photographs is the central point for comparison and updating.

An architecture for surroundings discovery by linking 3D models and LOD cloud

The Linked Open Data (LOD) cloud contains vast spatio-temporal information that can be exploited by the location-based mobile applications and presented using Augmented Reality (AR). While AR shows to be well suited for searching and browsing location-based information, most approaches focus on domain specific scenarios and there is no generic data model for the information search and discovery that could be re-used in various applications. The geo-referenced data, however, is already available on the LOD cloud and can be exploited to provide location-based information to the user. With this approach, in this paper, we present a generic architecture for the surroundings discovery mobile applications. The 3D models that we publish on the LOD cloud represent the real world objects that belong to different temporal intervals. These geometric models allow the user to mentally construct the referential relationship between virtual and real-world objects and the mobile applciation developer to create experiences based on different concepts on the cloud. This way, the LOD cloud, which is a growing structured source of semantic data, becomes the main source of information for the architecture, and facilitates the knowledge discovery. We also extend our mobile location-based application, ARCAMA-3D (Augmented Reality for Context Aware Mobile Applications with 3D) that we have developed previously, by using this architecture.

Query Understanding in Content-Based Image Retrieval Context

Queries in content-based image retrieval are usually ill-defined since the variability of query attributes is not explicitly specified and depends on user tolerance. In this paper, we aim at determining the query variability from user interaction and integrating it into the search process by adapting the calculated similarity to a perceived one. We examine in-depth the query understanding in the case of object retrieval. Our method is tested on the system THIS [14], results are promising.