Michael S. Lew

Content-based multimedia information retrieval: State of the art and challenges

Extending beyond the boundaries of science, art, and culture, content-based multimedia information retrieval provides new paradigms and methods for searching through the myriad variety of media all over the world. This survey reviews 100+ recent articles on content-based multimedia information retrieval and discusses their role in current research directions which include browsing and search paradigms, user studies, affective computing, learning, semantic queries, new features and media types, high performance indexing, and evaluation techniques. Based on the current state of the art, we discuss the major challenges for the future.

The MIR flickr retrieval evaluation

In most well known image retrieval test sets, the imagery typically cannot be freely distributed or is not representative of a large community of users. In this paper we present a collection for the MIR community comprising 25000 images from the Flickr website which are redistributable for research purposes and represent a real community of users both in the image content and image tags. We have extracted the tags and EXIF image metadata, and also make all of these publicly available. In addition we discuss several challenges for benchmarking retrieval and classification methods.

Image and Video Retrieval

We have witnessed a decade of exploding research interest in multimedia content analysis. The goal of content analysis has been to derive automatic methods for high-level description and annotation. In this paper we will summarize the main research topics in this area and state some assumptions that we have been using all along. We will also postulate the main future trends including usage of long term memory, context, dynamic processing, evolvable generalized detectors and user aspects.

New trends and ideas in visual concept detection: the MIR flickr retrieval evaluation initiative

The MIR Flickr collection consists of 25000 high-quality photographic images of thousands of Flickr users, made available under the Creative Commons license. The database includes all the original user tags and EXIF metadata. Additionally, detailed and accurate annotations are provided for topics corresponding to the most prominent visual concepts in the user tag data. The rich metadata allow for a wide variety of image retrieval benchmarking scenarios. In this paper, we provide an overview of the various strategies that were devised for automatic visual concept detection using the MIR Flickr collection. In particular we discuss results from various experiments in combining social data and low-level content-based descriptors to improve the accuracy of visual concept classifiers. Additionally, we present retrieval results obtained by relevance feedback methods, demonstrating (i) how their performance can be enhanced using features based on visual concept classifiers, and (ii) how their performance, based on small samples, can be measured relative to their large sample classifier counterparts. Additionally, we identify a number of promising trends and ideas in visual concept detection. To keep the MIR Flickr collection up-to-date on these developments, we have formulated two new initiatives to extend the original image collection. First, the collection will be extended to one million Creative Commons Flickr images. Second, a number of state-of-the-art content-based descriptors will be made available for the entire collection.

Principles of Visual Information Retrieval

This text introduces the basic concepts and techniques in VIR. In doing so, it develops a foundation for further research and study. Divided into two parts, the first part describes the fundamental principles. A chapter is devoted to each of the main features of VIR, such as colour, texture and shape-based search. There is coverage of search techniques for time-based image sequences or videos, and an overview of how to combine all the basic features described and integrate them into the search process. The second part looks at advanced topics such as multimedia query. This book is essential reading for researchers in VIR, and final-year undergraduate and postgraduate students on courses such as Multimedia Information Retrieval, Multimedia Databases, and others.

Authentic facial expression analysis

It is argued that for the computer to be able to interact with humans, it needs to havve the communication skills o humans. One of these skills is the ability to understand the emotional state of the person. The most expressive way humans display emotions is through facial expressions. In most facial expression systems and databases, the emotion data was collected by asking the subjects to perform a series of facial expressions. However, these directed or deliberate facial action tasks typically differ in appearance and timing from the authentic facial expressions induced through events in the normal environment of the subject. In this paper, we present our effort in creating an authentic facial expression database based on spontaneous emotions derived from the environment. Furthermore, we test and compare a wide range of classifiers from the machine learning literature that can be used for facial expression classification.

Deep learning for visual understanding

Deep learning algorithms are a subset of the machine learning algorithms, which aim at discovering multiple levels of distributed representations. Recently, numerous deep learning algorithms have been proposed to solve traditional artificial intelligence problems. This work aims to review the state-of-the-art in deep learning algorithms in computer vision by highlighting the contributions and challenges from over 210 recent research papers. It first gives an overview of various deep learning approaches and their recent developments, and then briefly describes their applications in diverse vision tasks, such as image classification, object detection, image retrieval, semantic segmentation and human pose estimation. Finally, the paper summarizes the future trends and challenges in designing and training deep neural networks.

Next-generation Web searches for visual content

Major search engines such as Hotbot (http://www.hotbot.com) help us find text on the Web, but typically have few or no capabilities for finding visual media. Yet many Web users, such as magazine editors or professional Web site designers, need to find images using just a few global features. My colleagues and I developed a prototype system called ImageScape (http://skynet.liacs.nl) to find visual media over intranets and the Web. The system integrates technologies such as vector quantization-based compression of the image database and k-d trees for fast searching over high-dimensional spaces.

Image retrieval using wavelet-based salient points

Content-based image retrieval (CBIR) has become one of the most active research areas in the past few years. Most of the attention from the research has been focused on indexing techniques based on global feature distributions. However, these global distributions have limited discriminating power because they are unable to capture local image information. The use of interest points in content-based image retrieval allow image index to represent local properties of the image. Classic corner detectors can be used for this purpose. However, they have drawbacks when applied to various natural images for image retrieval, because visual features need not be corners and corners may gather in small regions. In this paper, we present a salient point detector. The detector is based on wavelet transform to detect global variations as well as local ones. The wavelet-based salient points are evaluated for image retrieval with a retrieval system using color and texture features. The results show that salient points with Gabor feature perform better than the other point detectors from the literature and the randomly chosen points. Significant improvements are achieved in terms of retrieval accuracy, computational complexity when compared to the global feature approaches.

The distributed ASCI Supercomputer project

The Distributed ASCI Supercomputer (DAS) is a homogeneous wide-area distributed system consisting of four cluster computers at different locations. DAS has been used for research on communication software, parallel languages and programming systems, schedulers, parallel applications, and distributed applications. The paper gives a preview of the most interesting research results obtained so far in the DAS project.