Philippe Mulhem

Home photo content modeling for personalized event-based retrieval

Rapid advances in sensor, storage, processor, and communication technologies let consumers store large digital photo collections. Consumers need effective tools to organize and access photos in a semantically meaningful way. We address the semantic gap between feature-based indexes computed automatically and human query and retrieval preferences.

Pivot vector space approach for audio-video mixing

An audio-mixing artist usually adds the musical accompaniment to video. Employing such artists is expensive and not feasible for a home video presentation. Our automatic audio-video mixing technique is suited for home videos. It uses a pivot vector space mapping method that matches video shots with music segments based on aesthetic cinematographic heuristics.

Symbolic photograph content-based retrieval

Photograph retrieval systems face the difficulty to deal with the different ways to apprehend the content of images. We consider and demonstrate here the use of multiple index representations of photographs to achieve effective retrieval. The use of multiple indexes allows integration of the complementary strengths of different indexing and retrieval models. The proposed representation supports multiple labels for regions and attributes, and handles inferences and relationships. We define links between indexing levels and the related query modes. The experiment conducted on 2400 home photographs shows the behavior of the multiple indexing levels during retrieval.

Home photo retrieval: time matters

Temporal information has been regarded as a key vehicle for sorting and grouping home photos into albums associated with events. While time-based browsing might be adequate for relatively small photo collection, query and retrieval would be very useful to find relevant photos of an event in large collection. In this paper, we propose the use of temporal events for organizing and representing home photos using structured document formalism and hence a new way to retrieve photos of an event using both image content and temporal context. We describe a hierarchical model of temporal events and the algorithm to construct it from a collection of home photos. In particular, we compute metadata of a node from the metadata of its children recursively to facilitate content-based and context-based matching between a query and an event. With semantic content representation extracted using Visual Keywords and Extended Conceptual Graphs, we demonstrate the effectiveness of photo retrieval on 2400 time-stamped heterogeneous home photos with very promising results.

Interactive information retrieval systems: from user centered interface design to software design

This article is concerned with the design and implementation of Information Retrieval Systems (IRS). We show how theories and models from the domain of Human Computer Interaction (HCI) can be applied to the design of IRS. We first study the user’s tasks by modelling the mental activities of the user while accomplishing a task. Adopting a system perspective, we consider the processing tasks of an IRS and organize them in a design space. We then build upon the design space to consider the implications of such data processing and levels of abstraction on software design. Finally we present PAC-Amodeus, a software architecture model and illustrate the applicability of the approach with the implementation of an IRS: the TIAPRI system.

A model for weighting image objects in home photographs

The paper presents a contribution to image indexing consisting in a weighting model for visible objects -- or image objects -- in home photographs. To improve its effectiveness this weighting model has been designed according to human perception criteria about what is estimated as important in photographs. Four basic hypotheses related to human perception are presented, and their validity is estimated as compared to actual observations from a user study. Finally a formal definition of this weighting model is presented and its consistence with the user study is evaluated.

Metadata handling: A video perspective

This article addresses the problem of processing the annotations of preexisting video productions to enable reuse and repurposing of metadata. We introduce the concept of automatic content-based editing of preexisting semantic home video metadata. We propose a formal representation and implementation techniques for reusing and repurposing semantic video metadata in concordance with the actual video editing operations. A novel representation for metadata editing is proposed and an implementation framework for editing the metadata in accordance with the video editing operations is demonstrated. Conflict resolution and regularization operations are defined and implemented in the context of the video metadata editing operations.

A relational vector space model using an advanced weighting scheme for image retrieval

In this paper, we lay out a relational approach for indexing and retrieving photographs from a collection. The increase of digital image acquisition devices, combined with the growth of the World Wide Web, requires the development of information retrieval (IR) models and systems that provide fast access to images searched by users in databases. The aim of our work is to develop an IR model suited to images, integrating rich semantics for representing this visual data and user queries, which can also be applied to large corpora. Our proposal merges the vector space model of IR - widely tested in textual IR - with the conceptual graph (CG) formalism, based on the use of star graphs (i.e. elementary CGs made up of a single relation connected to some concepts representing image objects). A novel weighting scheme for star graphs, based on image objects size, position, and image heterogeneity is outlined. We show that integrating relations into the vector space model through star graphs increases the systems precision, and that the results are comparable to those from graph projection systems, and also that they shorten processing time for user queries.

A Method for Photograph Indexing Using Speech Annotation

We explore the feasibility of using speech input to perform the task of indexing a large volume of digital photographs. As a natural medium for image communication, speech can be used to complement existing contentbased techniques thereby promoting the reliability and use-ability of image retrieval systems. We introduce a methodology for image indexing using speech annotation technique. Speech recognition tools, like Dragon NaturallySpeaking can be adapted to perform the main role of speech-to-text transcription. The use of structured speech as opposed to free form speech in a limited system can further boost the transcription accuracy. We also introduce the idea of using N-best lists from the speech recognition output to improve the recognition performance. The transcribed text is used to populate the metadata of the corresponding photograph. A photo query strategy is implemented to affirm the performance of proposed technique for photo indexing and retrieval.

SmartAlbum: a multi-modal photo annotation system

This demonstration presents a novel application (called SmartAlbum) for photo indexing and retrieval that unifies two different image indexing approaches. The system uses two modalities to extract information about a digital photograph; i.e. content-based and speech annotation for image description. The result is a powerful image retrieval tool that has capabilities beyond what current single-mode retrieval systems can offer. We show on a corpus of 1200 images the interest of our approach.