Juan Manuel Barrios

Sketch based Image Retrieval using Learned KeyShapes (LKS).

Sketch based image retrieval is a particular case of the image retrieval problem, in which a query is not a regular example image. Instead, the query is a hand-drawn sketch representing what the user is looking for. This kind of problem has a lot of applications, in particular when an example image is not available. For instance, in searching for design pieces in digital catalogs. The natural ambiguity of sketches as well as the poor skills of drawing make the problem very challenging, which is reflected in the low performance achieved by current methods. In this work, we present a novel method for describing sketches based on detecting mid-level patterns called learned keyshapes. Our experiments were performed in two datasets, one with 1326 images and the other with approximately 15k images. Our results show an increase of effectiveness around 17% on the smaller dataset and 98% on the larger one, which represent new state-of-the-art performance in the sketch based image retrieval domain. We also show that our method allows us to achieve good performance even when we use around 20% of the sketch content.

Competitive content-based video copy detection using global descriptors

Content-Based Video Copy Detection (CBVCD) consists of detecting whether or not a video document is a copy of some known original and to retrieve the original video. CBVCD systems rely on two different tasks: Feature Extraction task, that calculates many representative descriptors for a video sequence, and Similarity Search task, that is the algorithm for finding videos in an indexed collection that match a query video. This work details a CBVCD approach based on a combination of global descriptors, an automatic weighting algorithm, a pivot-based index structure, an approximate similarity search, and a voting algorithm for copy localization. This approach is analyzed using MUSCLE-VCD-2007 corpus, and it was tested at the TRECVID 2010 evaluation together with other state-of-the-art CBVCD systems. The results show that this approach enables global descriptors to achieve competitive results and even outperforms systems based on combination of local descriptors and audio information. This approach has a potential of achieving even higher effectiveness due to its seamless ability of combining descriptors from different sources at the similarity search level.

Visual-semantic graphs: using queries to reduce the semantic gap in web image retrieval

We explore the application of a graph representation to model similarity relationships that exist among images found on the Web. The resulting similarity-induced graph allows us to model in a unified way different types of content-based similarities, as well as semantic relationships. Content-based similarities include different image descriptors, and semantic similarities can include relevance user feedback from search engines. The goal of our representation is to provide an experimental framework for combining apparently unrelated metrics into a unique graph structure, which allows us to enhance the results of Web image retrieval. We evaluate our approach by re-ranking Web image search results.

Text-Based and Content-Based Image Retrieval on Flickr: DEMO

We present an image retrieval system based on a combined search of text and content. The idea is to use the text present in title, description, and tags of the images for improving the results obtained with a standard content-based search. The system contains two different user interfaces: a sidebar for the browser designed for end users, where the user must enter the Flickr URL that is visiting and the system retrieves similar images from the collection, and an advanced search designed for experienced users, where the distance functions and weights can be customized.

Improving 3D similarity search by enhancing and combining 3D descriptors

Effective content-based retrieval in 3D model databases is an important problem that has attracted much research attention over the last years. Many individual methods proposed to date rely on calculating global 3D model descriptors based on image, surface, volumetric, or structural model properties. Descriptors such as these are then input for determining the degree of similarity between models. Traditionally, the ability of individual descriptors to perform effective 3D search is decided by benchmarking. However, in practice the data set on which 3D retrieval is to be applied may differ from the characteristics of the respective benchmark. Therefore, statically determining the descriptor to use based on a fixed benchmark may lead to suboptimal results. We propose a generic strategy to improve the retrieval effectiveness in 3D retrieval systems consisting of multiple model descriptors. The specific contribution of this paper is two-fold. First, we propose to adaptively combine multiple descriptors by forming weighted descriptor combinations, where the weight of each descriptor is decided at query time. Second, we enhance the set of global model descriptors to be combined by including partial descriptors of the same kind in the combinations. Partial descriptors are obtained by applying a given descriptor extractor on the set of parts of a model, obtained by a simple model partitioning scheme. Thereby, more model information is exposed to the 3D descriptors, leading to a more complete object description. We give a systematic discussion of the descriptor combination space involving static and query-adaptive weighting schemes, and based on descriptors of different type and focus (model global vs. partial). The combination of both global and partial model descriptors is shown to deliver improved retrieval precision, compared to policies using single descriptors or fixed-weight combinations. The resulting scheme is generic and can accommodate a large class of global 3D model descriptors.

Analyzing and dynamically indexing the query set

Abstract Most of the current metric indexes focus on indexing the collection of reference. In this work we study the problem of indexing the query set by exploiting some property that query objects may have. Thereafter, we present the Snake Table, which is an index structure designed for supporting streams of k-NN searches within a content-based similarity search framework. The index is created and updated in the online phase while resolving the queries, thus it does not need a preprocessing step. This index is intended to be used when the stream of query objects fits a snake distribution, that is, when the distance between two consecutive query objects is small. In particular, this kind of distribution is present in content-based video retrieval systems, image classification based on local descriptors, rotation-invariant shape matching, and others. We show that the Snake Table improves the efficiency of k-NN searches in these systems, avoiding the building of a static index in the offline phase.

Automatic weight selection for multi-metric distances

Content-Based Multimedia Information Retrieval retrieves multimedia documents based on their content (colors, edges, textures, etc.). The content of a whole multimedia document is represented by a global descriptor. The similarity of two multimedia documents can be defined as the distance between their descriptors. A multi-metric function that combines distances from many descriptors usually outperforms the effectiveness of any single descriptor. In this case, a different weight is assigned to each descriptor representing its relative importance in the combination. Usually, these sets of weights are fixed manually or by performing many effectiveness evaluations. In this work, we present three novel techniques for weighting multi-metrics: á-normalization, which is a generalization of the normalization by maximum distance that uses the histogram of distances, MID-weighting which selects weights that maximize intrinsic dimensionality, and MID-á-weighting that combines the two previous techniques. These techniques enable the selection of a set of weights with satisfactory effectiveness without performing any effectiveness evaluation. Thus, they are suitable when a ground truth does not exist or when it is expensive to perform an evaluation. We tested their effectiveness on a content-based copy detection corpus, and we analyzed the behavior of effectiveness and efficiency in a multi-metric space. We conclude that MID-á-weighting outperforms the widely used maximum distance normalization, and that it can be used as an automatic weight selection for further manual adjustment.

Multimodal fusion for video copy detection

Content-based video copy detection algorithms (CBCD) focus on detecting video segments that are identical or transformed versions of segments in a known video. In recent years some systems have proposed the combination of orthogonal modalities (e.g. derived from audio and video) to improve detection performance, although not always achieving consistent results. In this paper we propose a fusion algorithm that is able to combine as many modalities as available at the decision level. The algorithm is based on the weighted sum of the normalized scores, which are modified depending on how well they rank in each modality. This leads to a virtually parameter-free fusion algorithm. We performed several tests using 2010 TRECVID VCD datasets and obtain up to 46% relative improvement in min-NDCR while also improving the F1 metric on the fused results in comparison to just using the best single modality.

Content-based video copy detection

Content-Based Video Copy Detection consists in retrieving all the modified versions of an original document in a video collection. It relies on two tasks: content description, for extracting one or many fingerprints from a video document, and similarity search, for determining the set of extracted fingerprints that make a close match. For the similarity search task, a copy detection system usually relies on a metric distance for measuring the degree of similarity between fingerprints. The metric properties represent a tradeoff between efficiency and effectiveness for a similarity search. A metric distance allows the use of well studied indexing structures. However, the metric properties restrict the similarity model that can be used for comparing two objects. For the present thesis, the main focus will be on researching similarity models for video sequences that do not necessarily comply the metric properties. In particular, we plan to research multi-metric and non-metric similarity measures for fulfilling effective and efficient detection. The issues involved in video copy detection (visual transformations, local and global fingerprints, temporal dimension, and approximated searches) make this problem a relevant topic for researching new similarity models.

P-VCD: A pivot-based approach for Content-Based Video Copy Detection

Content-Based Video Copy Detection (CBVCD) consists of detecting and retrieving videos that are copies of known original videos. CBVCD systems rely on two different tasks: Feature Extraction task, that calculates many representative descriptors for a video sequence, and Similarity Search task, that is the algorithm for finding videos in an indexed collection that match a query video. This paper describes P-VCD, which is a novel approach for CBVCD based on global descriptors, weighted combinations of distances, a pivot-based index structure, an approximate similarity search, and a voting algorithm for copy localization. P-VCD was tested at the TRECVID 2010 evaluation, where it was the best positioned CBVCD system for Balanced and No False Alarms profiles considering visual-only runs (and above the median considering all runs). P-VCD shows that by using approximate similarity searches one can obtain good effectiveness, and that global descriptors can achieve competitive results with TRECVID transformations.