Julien Ah-Pine

Semantic combination of textual and visual information in multimedia retrieval

The goal of this paper is to introduce a set of techniques we call semantic combination in order to efficiently fuse text and image retrieval systems in the context of multimedia information access. These techniques emerge from the observation that image and textual queries are expressed at different semantic levels and that a single image query is often ambiguous. Overall, the semantic combination techniques overcome a conceptual barrier rather than a technical one: these methods can be seen as a combination of late fusion and image reranking. Albeit simple, this approach has not been used yet. We assess the proposed techniques against late and cross-media fusion using 4 different ImageCLEF datasets. Compared to late fusion, performances significantly increase on two datasets and remain similar on the two other ones.

Crossing textual and visual content in different application scenarios

This paper deals with multimedia information access. We propose two new approaches for hybrid text-image information processing that can be straightforwardly generalized to the more general multimodal scenario. Both approaches fall in the trans-media pseudo-relevance feedback category. Our first method proposes using a mixture model of the aggregate components, considering them as a single relevance concept. In our second approach, we define trans-media similarities as an aggregation of monomodal similarities between the elements of the aggregate and the new multimodal object. We also introduce the monomodal similarity measures for text and images that serve as basic components for both proposed trans-media similarities. We show how one can frame a large variety of problem in order to address them with the proposed techniques: image annotation or captioning, text illustration and multimedia retrieval and clustering. Finally, we present how these methods can be integrated in two applications: a travel blog assistant system and a tool for browsing the Wikipedia taking into account the multimedia nature of its content.

Unsupervised Visual and Textual Information Fusion in CBMIR Using Graph-Based Methods

Multimedia collections are more than ever growing in size and diversity. Effective multimedia retrieval systems are thus critical to access these datasets from the end-user perspective and in a scalable way. We are interested in repositories of image/text multimedia objects and we study multimodal information fusion techniques in the context of content-based multimedia information retrieval. We focus on graph-based methods, which have proven to provide state-of-the-art performances. We particularly examine two such methods: cross-media similarities and random-walk-based scores. From a theoretical viewpoint, we propose a unifying graph-based framework, which encompasses the two aforementioned approaches. Our proposal allows us to highlight the core features one should consider when using a graph-based technique for the combination of visual and textual information. We compare cross-media and random-walk-based results using three different real-world datasets. From a practical standpoint, our extended empirical analyses allow us to provide insights and guidelines about the use of graph-based methods for multimodal information fusion in content-based multimedia information retrieval.

Data Fusion in Information Retrieval Using Consensus Aggregation Operators

In this paper, we address the problem of unsupervised rank aggregation in the context of meta-searching in information retrieval field. The first goal of this paper is to apply aggregation operators that are defined in information fusion domain to the particular issue mentioned beforehand. Triangular norms, conorms and quasi-arithmetic means, are such kind of operators. Then, the second goal of this work is to introduce a new aggregation function, its logical foundations and its combinatorial properties. Particularly, this operator allows to take into account the relationships between experts in a flexible way. Finally, we test these different aggregation operators on the LETOR dataset. The results of our experiments show that this kind of aggregation functions can lead to better results than baseline methods such as CombSUM and CombMNZ approaches.

Clique-Based Clustering for Improving Named Entity Recognition Systems

We propose a system which builds, in a semi-supervised manner, a resource that aims at helping a NER system to annotate corpus-specific named entities. This system is based on a distributional approach which uses syntactic dependencies for measuring similarities between named entities. The specificity of the presented method however, is to combine a clique-based approach and a clustering technique that amounts to a soft clustering method. Our experiments show that the resource constructed by using this clique-based clustering system allows to improve different NER systems.

On data fusion in information retrieval using different aggregation operators

This paper is concerned with the problem of unsupervised rank aggregation in the context of metasearch in information retrieval. In such tasks, we are given many partial ordered lists of retrieved items provided by many search engines and we want to define a way for aggregating those lists in order to find out a consensus. One classical approach consists in aggregating, for each retrieved item, the scores given by the different search engines. Then, we use the resulting aggregated scores distribution in order to infer a consensus ordered list. In this paper we investigate whether aggregation operators defined in the fields of multi-sensor fusion and multicriteria decision making are of interest for metasearch problems or not. Moreover, another purpose of this paper is to introduce a new aggregation operator, its foundations and its properties. We finally test all these aggregation operators for metasearch tasks using the Letor 2.0 dataset. Our results show that among the studied aggregation functions, the ones which are more compensatory outperform the baseline methods CombSUM and CombMNZ.

Identification of a 2-Additive Bi-Capacity by Using Mathematical Programming

In some multi-criteria decision making problems, it is more convenient to express the decision maker preferences in bipolar scales. In such cases, the bipolar Choquet integral with respect to bi-capacities was introduced. In this paper, we address the problem of eliciting a bipolar Choquet integral with respect to a 2-additive bi-capacity. We assume that we are given a set of examples with i their scores distribution in regard to several criteria and ii their overall scores. We propose two types of optimization problems that allow identifying the parameters of a 2-additive bi-capacity such that the inferred bipolar Choquet integral is consistent with the given examples as much as possible. Furthermore, since the elicitation process we study has many relationships with problems in statistical machine learning, we also present the links between our models and concepts developed in the latter field.

Overview of the Relational Analysis approach in Data-Mining and Multi-criteria Decision Making

In this chapter we introduce a general framework called the Relational Analysis approachand its related contributions and applications in the fields of data analysis, data mining andmulti-criteria decision making. This approach was initiated by J.F. Marcotorchino and P.Michaud at the end of the 70’s and has generated many research activities. However, theaspects of this framework that we would like to focus on are of a theoretical kind. Indeed, weaimed at recalling the background and the basics of this framework, the unifying results andthe modeling contributions that it has allowed to achieve. Besides, the main tasks that we areinterested in are the ranking aggregation problem, the clustering problem and the block seri-ation problem. Those problems are combinatorial ones and the computational considerationsof such tasks in the context of the RA methodology will not be covered. However, amongthe list of references that we give throughout this chapter, they are numerous articles that theinterested reader could consult to this end.In order to introduce the Relational Analysis approach (denoted “RA” in the rest of the doc-ument), let us first introduce several problems that one could encounter in the data anal-ysis field. To this end, let us consider a data table concerning a set of

Leveraging Image, Text and Cross–media Similarities for Diversity–focused Multimedia Retrieval

This chapter summarizes the different cross–modal information retrieval techniques Xerox Research Centre implemented during three years of participation in ImageCLEF Photo tasks. The main challenge remained constant: how to optimally couple visual and textual similarities, when they capture things at different semantic levels and when one of the media (the textual one) gives, most of the time, much better retrieval performance. Some core components turned out to be very effective all over the years: the visual similarity metrics based on Fisher Vector representation of images and the cross–media similarity principle based on relevance models. However, other components were introduced to solve additional issues: We tried different query– and document–enrichment methods by exploiting auxiliary resources such as Flickr or open–source thesauri, or by doing some statistical ‘semantic smoothing’. We also implemented some clustering mechanisms in order to promote diversity in the top results and to provide faster access to relevant information. This chapter describes, analyses and assesses each of these components, namely: the monomodal similarity measures, the different cross–media similarities, the query and document enrichment, and finally the mechanisms to ensure diversity in what is proposed to the user. To conclude, we discuss the numerous lessons we have learnt over the years by trying to solve this very challenging task.

Comparison of several combinations of multimodal and diversity seeking methods for multimedia retrieval

The aim of this paper is to analyze the technologies designed and used in the context of XRCEs participation in the Photo Retrieval Task of ImageCLEF 2009 [1]. We evaluate and compare different mono and multimedia retrieval methods and two distinct diversity-seeking strategies as well. Our analysis allows us to better understand which combinations of basic approaches are the best ones. It appears that taking advantage of the multimodal nature of the data by means of our cross-modal similarities technique and leveraging different text representations of the topics in the goal of covering distinct related subtopics, allow us to tackle the Photo Retrieval Task effectively.