Rodrigo Tripodi Calumby

Multimodal retrieval with relevance feedback based on genetic programming

This paper presents a framework for multimodal retrieval with relevance feedback based on genetic programming. In this supervised learning-to-rank framework, genetic programming is used for the discovery of effective combination functions of (multimodal) similarity measures using the information obtained throughout the user relevance feedback iterations. With these new functions, several similarity measures, including those extracted from different modalities (e.g., text, and content), are combined into one single measure that properly encodes the user preferences. This framework was instantiated for multimodal image retrieval using visual and textual features and was validated using two image collections, one from the Washington University and another from the ImageCLEF Photographic Retrieval Task. For this image retrieval instance several multimodal relevance feedback techniques were implemented and evaluated. The proposed approach has produced statistically significant better results for multimodal retrieval over single modality approaches and superior effectiveness when compared to the best submissions of the ImageCLEF Photographic Retrieval Task 2008.

A rank aggregation framework for video multimodal geocoding

This paper proposes a rank aggregation framework for video multimodal geocoding. Textual and visual descriptions associated with videos are used to define ranked lists. These ranked lists are later combined, and the resulting ranked list is used to define appropriate locations for videos. An architecture that implements the proposed framework is designed. In this architecture, there are specific modules for each modality (e.g, textual and visual) that can be developed and evolved independently. Another component is a data fusion module responsible for combining seamlessly the ranked lists defined for each modality. We have validated the proposed framework in the context of the MediaEval 2012 Placing Task, whose objective is to automatically assign geographical coordinates to videos. Obtained results show how our multimodal approach improves the geocoding results when compared to methods that rely on a single modality (either textual or visual descriptors). We also show that the proposed multimodal approach yields comparable results to the best submissions to the Placing Task in 2012 using no extra information besides the available development/training data. Another contribution of this work is related to the proposal of a new effectiveness evaluation measure. The proposed measure is based on distance scores that summarize how effective a designed/tested approach is, considering its overall result for a test dataset.

Unsupervised Distance Learning By Reciprocal kNN Distance for Image Retrieval

This paper presents a novel unsupervised learning approach that takes into account the intrinsic dataset structure, which is represented in terms of the reciprocal neighborhood references found in different ranked lists. The proposed Reciprocal kNN Distance defines a more effective distance between two images, and is used to improve the effectiveness of image retrieval systems. Several experiments were conducted for different image retrieval tasks involving shape, color, and texture descriptors. The proposed approach is also evaluated on multimodal retrieval tasks, considering visual and textual descriptors. Experimental results demonstrate the effectiveness of proposed approach. The Reciprocal kNN Distance yields better results in terms of effectiveness than various state-of-the-art algorithms.

A Genetic Programming approach for coffee crop recognition

This work presents a new approach for automatic recognition of coffee crops in RSIs. The method applies an approach based on Genetic Programming (GP) to combine texture and spectral information encoded by image descriptors. Experiments show that the proposed method yields slightly better results than the traditional MaxVer approach.

Using contextual spaces for image re-ranking and rank aggregation

This article presents two novel re-ranking approaches that take into account contextual information defined by the K-Nearest Neighbours (KNN) of a query image for improving the effectiveness of CBIR systems. The main contributions of this article are the definition of the concept of contextual spaces for encoding contextual information of images; the definition of two new re-ranking algorithms that exploit contextual information encoded in contextual spaces; and the evaluation of the proposed algorithms in several CBIR tasks related to the combination of image descriptors; combination of visual and textual descriptors; and combination of post-processing (re-ranking) methods. We conducted a large evaluation protocol involving visual descriptors (considering shape, color, and texture) and textual descriptors, various datasets, and comparisons with other post-processing methods. Experimental results demonstrate the effectiveness of our approaches.

On interactive learning-to-rank for IR: Overview, recent advances, challenges, and directions

Abstract With the amount and variety of information available on digital repositories, answering complex user needs and personalizing information access became a hard task. Putting the user in the retrieval loop has emerged as a reasonable alternative to enhance search effectiveness and consequently the user experience. Due to the great advances on machine learning techniques, optimizing search engines according to user preferences has attracted great attention from the research and industry communities. Interactively learning-to-rank has greatly evolved over the last decade but it still faces great theoretical and practical obstacles. This paper describes basic concepts and reviews state-of-the-art methods on the several research fields that complementarily support the creation of interactive information retrieval (IIR) systems. By revisiting ground concepts and gathering recent advances, this article also intends to foster new research activities on IIR by highlighting great challenges and promising directions. The aggregated knowledge provided here is intended to work as a comprehensive introduction to those interested in IIR development, while also providing important insights on the vast opportunities of novel research.

Multimedia multimodal geocoding

This work is developed in the context of the placing task of the MediaEval 2011 initiative. The objective is to geocode (or geotag) a set of videos, i.e., automatically assign geographical coordinates to them. This paper presents an architecture for multimodal geocoding that exploits both visual and textual descriptions associated with videos. This work also describes our efforts regarding the implementation of this architecture to demonstrate its applicability. Conducted experiments show how our multimodal approach enhances the results compared to relying on a single modality.

Diversity-driven learning for multimodal image retrieval with relevance feedback

We introduce a new genetic programming approach for enhancing the user search experience based on relevance feedback over results produced by a multimodal image retrieval technique with explicit diversity promotion. We have studied maximal marginal relevance re-ranking methods for result diversification and their impacts on the overall retrieval effectiveness. We show that the learning process using diverse results may improve user experience in terms of both the number of relevant items retrieved and subtopic coverage.

Diversity-based interactive learning meets multimodality

Abstract In interactive retrieval tasks, one of the main objectives is to maximize the user information gain throughout search sessions. Retrieving many relevant items is quite important, but it does not necessarily completely satisfy the user needs. When only relevant near-duplicate items are retrieved, the amount of different concepts users are able to extract from the target collection is very limited. Therefore, broadening the number of concepts present in a result set may improve the overall search experience. Diversifying concepts present in the retrieved set is one possibility for increasing the information gain in a single search iteration, maximizing the likelihood of including at least some relevant items for each possible intent of ambiguous or underspecified queries. Relevance feedback approaches may also take advantage of diverse results to improve internal machine learning models. In this context, this work proposes and analyses several multimodal image retrieval approaches built over a learning framework for relevance feedback on diversified results. Our experimental analysis shows that different retrieval modalities are positively impacted by diversity, but achieve best retrieval effectiveness with diversification applied at different moments of a search session. Moreover, the best results are achieved with a query-by-example approach using multimodal information obtained from feedback. In summary, we demonstrate that learning with diversity is an effective alternative for boosting multimodal interactive learning approaches.

Semi-supervised Learning for Relevance Feedback on Image Retrieval Tasks

Relevance feedback approaches have been established as an important tool for interactive search, enabling users to express their needs. However, in view of the growth of multimedia collections available, the user efforts required by these methods tend to increase as well, demanding approaches for reducing the need of user interactions. In this context, this paper proposes a semi-supervised learning algorithm for relevance feedback to be used in image retrieval tasks. The proposed semi-supervised algorithm aims at using both supervised and unsupervised approaches simultaneously. While a supervised step is performed using the information collected from the user feedback, an unsupervised step exploits the intrinsic dataset structure, which is represented in terms of ranked lists of images. Several experiments were conducted for different image retrieval tasks involving shape, color, and texture descriptors and different datasets. The proposed approach was also evaluated on multimodal retrieval tasks, considering visual and textual descriptors. Experimental results demonstrate the effectiveness of the proposed approach.