Mark J. Huiskes

The MIR flickr retrieval evaluation

In most well known image retrieval test sets, the imagery typically cannot be freely distributed or is not representative of a large community of users. In this paper we present a collection for the MIR community comprising 25000 images from the Flickr website which are redistributable for research purposes and represent a real community of users both in the image content and image tags. We have extracted the tags and EXIF image metadata, and also make all of these publicly available. In addition we discuss several challenges for benchmarking retrieval and classification methods.

New trends and ideas in visual concept detection: the MIR flickr retrieval evaluation initiative

The MIR Flickr collection consists of 25000 high-quality photographic images of thousands of Flickr users, made available under the Creative Commons license. The database includes all the original user tags and EXIF metadata. Additionally, detailed and accurate annotations are provided for topics corresponding to the most prominent visual concepts in the user tag data. The rich metadata allow for a wide variety of image retrieval benchmarking scenarios. In this paper, we provide an overview of the various strategies that were devised for automatic visual concept detection using the MIR Flickr collection. In particular we discuss results from various experiments in combining social data and low-level content-based descriptors to improve the accuracy of visual concept classifiers. Additionally, we present retrieval results obtained by relevance feedback methods, demonstrating (i) how their performance can be enhanced using features based on visual concept classifiers, and (ii) how their performance, based on small samples, can be measured relative to their large sample classifier counterparts. Additionally, we identify a number of promising trends and ideas in visual concept detection. To keep the MIR Flickr collection up-to-date on these developments, we have formulated two new initiatives to extend the original image collection. First, the collection will be extended to one million Creative Commons Flickr images. Second, a number of state-of-the-art content-based descriptors will be made available for the entire collection.

Performance evaluation of relevance feedback methods

In this paper we review the evaluation of relevance feedback methods for content-based image retrieval systems. We start out by presenting an overview of current common practice, and argue that the evaluation of relevance feedback methods differs from evaluating CBIR systems as a whole. Specifically, we identify the challenging issues that are particular to the evaluation of retrieval employing relevance feedback. Next, we propose three guidelines to move toward more effective evaluation benchmarks. We focus particularly on assessing feedback methods more directly in terms of their goal of identifying the relevant target class with a small number of samples, and show how to compensate for query targets of varying difficulty by measuring efficiency at generalization.

Large scale image copy detection evaluation

In this paper we provide a comparative study of content-based copy detection methods, which include research literature methods based on salient point matching (SURF), discrete cosine and wavelet transforms, color histograms, biologically motivated visual matching and other methods. In our evaluation we focus on large-scale applications, especially on performance in the context of search engines for web images. We assess the scalability of the tested methods by investigating the detection accuracy relative to descriptor size, description time per image and matching time per image. For testing, original images altered by a diverse set of realistic transformations are embedded in a collection of one million web images.

A framework for robust feature selection for real-time fashion style recommendation

In this paper, we present the Smart Mirror system for fashion recommendation. The system uses intelligent vision technology to recognize clothing styles and supports real-time fashion recommendation. An important design challenge is to achieve sufficiently high style recognition accuracy while simultaneously offering robustness to input variations occurring in practice. We propose a framework for the selection of features that offer robust performance by assessing various evaluation measures under realistic deviations of optimal input data. The process is applied to a variety of low level features for clothing style description, including color histograms, local binary pattern (LBP) features and histogram of oriented gradient (HOG) features. We conclude the paper with an illustration of our results for web camera data and with a number of recommendations on how to move forward towards automatic fashion style perception.

Using an artificial imagination for texture retrieval

Our goal is to determine if artificially imagined or synthesized images can be beneficial to interactive visual search. We present a novel approach for using artificially imagined images in relevance feedback. Since the search engine constructs the synthetic images itself, any feedback given by the user on these images allows it to obtain a better understanding of what the user is looking for than it would from feedback on database images alone. We evaluated and compared our image synthesis approach with a normal Rocchio-based system on a well-known texture database with real users.

An evaluation of content-based duplicate image detection methods for web search

The world wide web is filled with billions of images and duplicates of images can frequently be found on many websites. These duplicates can be exact copies or differ slightly in their visual content. In this paper we provide a comparative study on how well content-based duplicate image detection methods are able to detect the duplicates of a query image. We conduct a survey to better understand in which ways such images on the internet differ from each other and use these observations to form a realistic and challenging duplicate image detection scenario. The methods we evaluate in our study are representative techniques from the research literature. In our evaluation, we target the performance of each method in relation to their descriptor size, description time and matching time, to assess their feasibility of application to large image collections (> 1 million).

Image searching and browsing by active aspect-based relevance learning

Aspect-based relevance learning is a relevance feedback scheme based on a natural model of relevance in terms of image aspects. In this paper we propose a number of active learning and interaction strategies, capitalizing on the transparency of the aspect-based framework. Additionally, we demonstrate that, relative to other schemes, aspect-based relevance learning upholds its retrieval performance well under feedback consisting mainly of example images that are only partially relevant.

Aspect-based relevance learning for image retrieval

We analyze the special structure of the relevance feedback learning problem, focusing particularly on the effects of image selection by partial relevance on the clustering behavior of feedback examples. We propose a scheme, aspect-based relevance learning, which guarantees that feedback on feature values is accepted only once evidential support that the feedback was intended by the user is sufficiently strong. The scheme additionally allows for natural simulation of the relevance feedback process. By means of simulation we analyze retrieval performance, search regularity and sensitivity to feature errors.

Deep exploration for experiential image retrieval

Experiential image retrieval systems aim to provide the user with a natural and intuitive search experience. The goal is to empower the user to navigate large collections based on his own needs and preferences, while simultaneously providing him with an accurate sense of what the database has to offer. In this paper we integrate a new browsing mechanism called deep exploration with the proven technique of retrieval by relevance feedback. In our approach, relevance feedback focuses the search on relevant regions, while deep exploration facilitates transparent navigation to promising regions of feature space that would normally remain unreachable. Optimal feature weights are determined automatically based on the evidential support for the relevance of each single feature. To achieve efficient refinement of the search space, images are ranked and presented to the user based on their likelihood of being useful for further exploration.