Philippe Mulhem

A comparative study of diversity methods for hybrid text and image retrieval approaches

This article compares eight different diversity methods: 3 based on visual information, 1 based on date information, 3 adapted to each topic based on location and visual information; finally, for completeness, 1 based on random permutation. To compare the effectiveness of these methods, we apply them on 26 runs obtained with varied methods from different research teams and based on different modalities. We then discuss the results of the more than 200 obtained runs. The results show that query-adapted methods are more effcient than nonadapted method, that visual only runs are more difficult to diversify than text only and text-image runs, and finally that only few methods maximize both the precision and the cluster recall at 20 documents.

Integrating Perceptual Signal Features within a Multi-facetted Conceptual Model for Automatic Image Retrieval

The majority of the content-based image retrieval (CBIR) systems are restricted to the representation of signal aspects, e.g. color, texture...without explicitly considering the semantic content of images. According to these approaches a sun, for example, is represented by an orange or yellow circle, but not by the term “sun”. The signal-oriented solutions are fully automatic, and thus easily usable on substantial amounts of data, but they do not fill the existing gap between the extracted low-level features and semantic descriptions. This obviously penalizes qualitative and quantitative performances in terms of recall and precision, and therefore users’ satisfaction. Another class of methods, which were tested within the framework of the Fermi-GC project, consisted in modeling the content of images following a sharp process of human-assisted indexing. This approach, based on an elaborate model of representation (the conceptual graph formalism) provides satisfactory results during the retrieval phase but is not easily usable on large collections of images because of the necessary human intervention required for indexing. The contribution of this paper is twofold: in order to achieve more efficiency as far as user interaction is concerned, we propose to highlight a bond between these two classes of image retrieval systems and integrate signal and semantic features within a unified conceptual framework. Then, as opposed to state-of-the-art relevance feedback systems dealing with this integration, we propose a representation formalism supporting this integration which allows us to specify a rich query language combining both semantic and signal characterizations. We will validate our approach through quantitative (recall-precision curves) evaluations.

A conceptual image retrieval architecture combining keyword-based querying with transparent and penetrable query-by-example

Performance of state-of-the-art image retrieval systems is strongly limited due to the difficulty of accurately relating semantics conveyed by images to low-level extracted features. Moreover, dealing with the problem of combining modalities for querying is of huge importance in forthcoming retrieval methodologies and is the only solution for achieving significant retrieval performance on image documents. This paper presents an architecture addressing both of these issues which is based on an expressive formalism handling high-level image descriptions. First, it features a multi-facetted conceptual framework which integrates semantics and signal characterizations and operates on image objects (abstractions of visual entities within a physical image) in an attempt to perform indexing and querying operations beyond trivial low-level processes and region-based frameworks. Then, it features a query-by-example framework based on high-level image descriptions instead of their extracted low-level features and operate both on semantics and signal features. The flexibility of this module and the rich query language it offers, consisting of both boolean and quantification operators, lead to optimized user interaction and increased retrieval performance. Experimental results on a test collection of 2500 images show that our approach gives better results in terms of recall and precision measures than state-of-the-art frameworks which couple loosely keyword-based query modules and relevance feedback processes operating on low-level features.

A Generic Framework for Structured Document Access

Current electronic document retrieval systems, i.e., database systems or information retrieval systems, do not handle enough the richness related to the document structures. Based on a classical model of structured documents, our approach intends to integrate propagation mechanisms related to document structure. These machanisms, called “scopes of attributes”, have two goals: they exhibit information that is usually kept implicit in documents, and they address dependencies between the values of the attributes used for the retrieval of the documents. The model of documents, the propagation and retrieval mechanisms compose a generic framework for the retrieval of structured documents.