Eric Bruno

Information Fusion in Multimedia Information Retrieval

In retrieval, indexing and classification of multimedia data an efficient information fusion of the different modalities is essential for the systems overall performance. Since information fusion, its influence factors and performance improvement boundaries have been lively discussed in the last years in different research communities, we will review their latest findings. They most importantly point out that exploiting the features and modalitys dependencies will yield to maximal performance. In data analysis and fusion tests with annotated image collections this is undermined.

Multiview clustering: a late fusion approach using latent models

Multi-view clustering is an important problem in information retrieval due to the abundance of data offering many perspectives and generating multi-view representations. We investigate in this short note a late fusion approach for multi-view clustering based on the latent modeling of cluster-cluster relationships. We derive a probabilistic multi-view clustering model outperforming an early-fusion approach based on multi-view feature correlation analysis.

Information-theoretic temporal segmentation of video and applications: multiscale keyframes selection and shot boundaries detection

The first step in the analysis of video content is the partitioning of a long video sequence into short homogeneous temporal segments. The homogeneity property ensures that the segments are taken by a single camera and represent a continuous action in time and space. These segments can then be used as atomic temporal components for higher level analysis like browsing, classification, indexing and retrieval. The novelty of our approach is to use color information to partition the video into segments dynamically homogeneous using a criterion inspired by compact coding theory. We perform an information-based segmentation using a Minimum Message Length (MML) criterion and minimization by a Dynamic Programming Algorithm (DPA). We show that our method is efficient and robust to detect all types of transitions in a generic manner. A specific detector for each type of transition of interest therefore becomes unnecessary. We illustrate our technique by two applications: a multiscale keyframe selection and a generic shot boundaries detection.

Adaptive Multimedia Retrieval: User, Context, and Feedback

Ontology-Based Retrieval and Annotation.- A Method for Processing the Natural Language Query in Ontology-Based Image Retrieval System.- SAFIRE: Towards Standardized Semantic Rich Image Annotation.- Ontology-Supported Video Modeling and Retrieval.- Ranking and Similarity Measurements.- Learning to Retrieve Images from Text Queries with a Discriminative Model.- A General Principled Method for Image Similarity Validation.- Rank-Test Similarity Measure Between Video Segments for Local Descriptors.- Music Information Retrieval.- Can Humans Benefit from Music Information Retrieval?.- Visual Modelling.- A New Approach to Probabilistic Image Modeling with Multidimensional Hidden Markov Models.- 3D Face Recognition by Modeling the Arrangement of Concave and Convex Regions.- Fuzzy Semantic Action and Color Characterization of Animation Movies in the Video Indexing Task Context.- Retrieval of Document Images Based on Page Layout Similarity.- Adaptive Retrieval.- Multimedia Content Adaptation Within the CAIN Framework Via Constraints Satisfaction and Optimization.- Aspects of Adaptivity in P2P Information Retrieval.- Interactive Museum Guide: Accurate Retrieval of Object Descriptions.- Structuring Multimedia.- Semantic Image Retrieval Using Region-Based Relevance Feedback.- Image Retrieval with Segmentation-Based Query.- Fast Structuring of Large Television Streams Using Program Guides.- User Integration and Profiling.- Variation of Relevance Assessments for Medical Image Retrieval.- An Efficient Collaborative Information Retrieval System by Incorporating the User Profile.- The Potential of User Feedback Through the Iterative Refining of Queries in an Image Retrieval System.

Robust motion estimation using spatial Gabor-like filters

This paper presents a new algorithm for motion estimation. It combines Gabor-like filter decomposition and robust least-squares estimation in a multiresolution framework. The spatial Gabor-like filter bank, based on recursive implementation, provides a fast multichannel decomposition of frame sequence. Then, applying the brightness constancy constraint on each channel between two consecutive frames, an over-determined system of velocity equations at each pixel is obtained. In order to be robust to outliers, this over-determined system is solved using a robust least-squares technique. A multiresolution framework is used in order to manage large and small displacements. Two kinds of recursive filter implementation have been tested: whereas third order filtering is very similar to a real Gabor filter, first order recursive filters are fastest and can be implemented with very large scale integration (VLSI) analog circuit. Performances of our algorithm for the two filter implementations are tested on synthetic and real sequences, and are compared with other techniques.

A robust multiscale B-spline function decomposition for estimating motion transparency

Motion transparency phenomena in image sequences are frequent, but classical methods of motion estimation are unable to deal with them. This paper describes a method for estimating optical flow by a generalization of the brightness constancy assumption to additive transparencies. The brightness constancy assumption is obtained by setting constant velocity fields during three images of a sequence. Thus, by a Taylor development to its second order, we reach an extension of the optical flow constraint equation. Since the equation is nonlinear, the Levenberg-Marquardt algorithm is used. In order to suppress the unavoidable aperture problem, a global model based on B-spline basis functions is applied with the aim of constraining optical flows. This description of motion allows us to work on a coarse to fine estimation of artificial image sequences that shows good convergence properties. It is also applied to natural image sequences.

TagCaptcha: annotating images with CAPTCHAs

Image retrieval has long been plagued by limitations on automatic methods because they cannot reliably extract semantic data from low-level features. The result is that users must formulate awkward and inefficient queries in terms these systems can understand. Humans, on the other hand, have the ability to quickly and accurately summarise visual data. This dichotomy, named the semantic gap, is a fundamental problem in image retrieval. We aim to narrow the semantic gap in a typical retrieval scenario by motivating users to provide semantic image annotations. We propose a system of collecting image annotations based on the need for human verification on the web. Similar in principle to work by von Ahn et al. [2, 3], the idea is to exploit the requirement of users to pass tests in order to incrementally annotate images.

Can feature information interaction help for information fusion in multimedia problems

This article presents the information-theoretic based feature information interaction, a measure that can describe complex feature dependencies in multivariate settings. According to the theoretical development, feature interactions are more accurate than current, bivariate dependence measures due to their stable and unambiguous definition. In experiments with artificial and real data we compare first the empirical dependency estimates of correlation, mutual information and 3-way feature interaction. Then, we present feature selection and classification experiments that show superior performance of interactions over bivariate dependence measures for the artificial data, for real world data this goal is not achieved yet.

Learning user queries in multimodal dissimilarity spaces

Different strategies to learn user semantic queries from dissimilarity representations of audio-visual content are presented. When dealing with large corpora of videos documents, using a feature representation requires the on-line computation of distances between all documents and a query. Hence, a dissimilarity representation may be preferred because its offline computation speeds up the retrieval process. We show how distances related to visual and audio video features can directly be used to learn complex concepts from a set of positive and negative examples provided by the user. Based on the idea of dissimilarity spaces, we derive three algorithms to fuse modalities and therefore to enhance the precision of retrieval results. The evaluation of our technique is performed on artificial data and on the annotated TRECVID corpus.

Combining multimodal preferences for multimedia information retrieval

Representing and fusing multimedia information is a key issue to discover semantics in multimedia. In this paper we address more specifically the problem of multimedia content retrieval by first defining a novel preference-based representation particularly adapted to the fusion problem, and then, by investigating the RankBoost algorithm to combine those preferences and a learn multimodal retrieval model. The approach has been tested on annotated images and on the complete TRECVID 2005 corpus and compared with SVM-based fusion strategies. The results show that our approach equals SVM performance but, contrary to SVM, is parameter free and faster.