Charlie K. Dagli

Active Learning for Interactive Multimedia Retrieval

As the first decade of the 21st century comes to a close, growth in multimedia delivery infrastructure and public demand for applications built on this backbone are converging like never before. The push towards reaching truly interactive multimedia technologies becomes stronger as our media consumption paradigms continue to change. In this paper, we profile a technology leading the way in this revolution: active learning. Active learning is a strategy that helps alleviate challenges inherent in multimedia information retrieval through user interaction. We show how active learning is ideally suited for the multimedia information retrieval problem by giving an overview of the paradigm and component technologies used with special attention given to the application scenarios in which these technologies are useful. Finally, we give insight into the future of this growing field and how it fits into the larger context of multimedia information retrieval.

Content-Based Image Retrieval by Feature Adaptation and Relevance Feedback

The paper proposes an adaptive retrieval approach based on the concept of relevance-feedback, which establishes a link between high-level concepts and low-level features, using the users feedback not only to assign proper weights to the features, but also to dynamically select them within a large collection of parameters. The target is to identify a set of relevant features according to a user query while at the same time maintaining a small sized feature vector to attain better matching and lower complexity. To this end, the image description is modified during each retrieval by removing the least significant features and better specifying the most significant ones. The feature adaptation is based on a hierarchical approach. The weights are then adjusted based on previously retrieved relevant and irrelevant images without further user-feedback. The algorithm is not fixed to a given feature set. It can be used with different hierarchical feature sets, provided that the hierarchical structure is defined a priori. Results achieved on different image databases and two completely different feature sets show that the proposed algorithm outperforms previously proposed methods. Further, it is experimentally demonstrated that it approaches the results obtained by state-of-the-art feature-selection techniques having complete knowledge of the data set.

Leveraging active learning for relevance feedback using an information theoretic diversity measure

Interactively learning from a small sample of unlabeled examples is an enormously challenging task. Relevance feedback and more recently active learning are two standard techniques that have received much attention towards solving this interactive learning problem. How to best utilize the users effort for labeling, however, remains unanswered. It has been shown in the past that labeling a diverse set of points is helpful, however, the notion of diversity has either been dependent on the learner used, or computationally expensive. In this paper, we intend to address these issues by proposing a fundamentally motivated, information-theoretic view of diversity and its use in a fast, non-degenerate active learning-based relevance feedback setting. Comparative testing and results are reported and thoughts for future work are presented.

Evaluating group-based relevance feedback for content-based image retrieval

New relevance feedback algorithms have been developed for content-based image retrieval (CBIR) that allow the user to achieve more flexible query. In conjunction with the new user interface, called group-oriented user interface, the users interest can be expressed with multiple groups of positive and negative image examples. This provides users with greater flexibility as compared with previous systems that consider image query as one or two-class problems. In this paper, we analyze our new algorithm qualitatively and quantitatively. For comparison with previous approaches, the systems are tested on both toy problems and real image retrieval tasks. From the results of our experiments, we suggest when and how our algorithm has advantages over the previous methods.

A framework for grid-based image retrieval

In this paper, we present a grid-based framework for image retrieval. In order to represent the intricate composition of images, the grid-based approach partitions each image into blocks from which a feature representation is derived from the local low-level content. Since the background often dominates the subject in the foreground, a special query selection method was developed. It combines the salient region-of-interest/query-by-example paradigm with coarse segmentation to remove the irrelevant background regions. The proposed search method looks for similar features across all block positions and at several scales. Existing local grid-based methods are constrained by searching for objects in the same position as the query object. Using this framework, the spatial constraint can be eliminated, and steps toward scale invariance can be taken. Promising results show that the grid-based method performs better than global search.

Utilizing Information Theoretic Diversity for SVM Active Learn

Incrementally learning from a large number of unlabeled examples continues to be an active area of research in pattern recognition. Active learning has made great strides in recent years to address this problem, taking advantage of SVMs to develop robust learning systems. Recently, diversity sampling for SVM active learning has garnered much attention. In this work we propose a fundamentally motivated view of diversity for SVM active learning based on an information-theoretic diversity measure. Comparative testing on a database from the small-sample learning problem of image retrieval is done and thoughts for future work are presented

Combining diversity-based active learning with discriminant analysis in image retrieval

Small-sample learning in image retrieval is a pertinent and interesting problem. Relevance feedback is an active area of research that seeks to find algorithms that are robust with only a small number of examples. Much work has been done in both the machine learning and pattern recognition communities to develop algorithms that learn a high-level semantic concept in a low-level image feature space. In this paper we seek to leverage techniques from both these communities to explore a hybrid relevance feedback system which combines the insight gained from discriminant analysis and active learning. Our technique uses a diversity-based pool-query technique along with biased discriminant analysis to improve the query refinement process. Comparative results are observed and thoughts for future work are presented.

Image Interpretation Using Large Corpus: Wikipedia

Image is a powerful medium for expressing ones ideas and rightly confirms the adage, “One picture is worth a thousand words.” In this work, we explore the application of world knowledge in the form of Wikipedia to achieve this objective-literally. In the first part, we disambiguate and rank semantic concepts associated with ambiguous keywords by exploiting link structure of articles in Wikipedia. In the second part, we explore an image representation in terms of keywords which reflect the semantic content of an image. Our approach is inspired by the desire to augment low-level image representation with massive amounts of world knowledge, to facilitate computer vision tasks like image retrieval based on this information. We represent an image as a weighted mixture of a predetermined set of concrete concepts whose definition has been agreed upon by a wide variety of audience. To achieve this objective, we use concepts defined by Wikipedia articles, e.g., sky, building, or automobile. An important advantage of our approach is availability of vast amounts of highly organized human knowledge in Wikipedia. Wikipedia evolves rapidly steadily increasing its breadth and depth over time.

Diverse Active Ranking for Multimedia Search

Interactively learning from a small sample of unlabeled examples is an enormously challenging task, one that often arises in vision applications. Relevance feedback and more recently active learning are two standard techniques that have received much attention towards solving this interactive learning problem. How to best utilize the users effort for labeling, however, remains unanswered. It has been shown in the past that labeling a diverse set of points is helpful, however, the notion of diversity has either been dependent on the learner used, or computationally expensive. In this paper, we intend to address these issues in the bipartite ranking setting. First, we introduce a scheme for picking the query set which will be labeled by an oracle so that it will aid us in learning the ranker in as few active learning rounds as possible. Secondly, we propose a fundamentally motivated, information theoretic view of diversity and its use in a fast, non-degenerate active learning-based relevance feedback setting. Finally, we report comparative testing and results in a real-time image retrieval setting.

Multi-Modal Person-Profiles from Broadcast News Video

The need to analyze and index large amounts of video information is becoming more important as the way people consume media continues to change. In recent years, the push to attack multimedia indexing and retrieval applications in a holistic, multi-modal way has garnered great attention. In this work we propose the holistic use of both audio, visual and textual information for the automatic indexing of broadcast news video to create person-profiles. Indexing videos in this matter helps facilitate a unique way to create multimedia person databases automatically, as well as for attacking existing video analysis tasks. We test our algorithm on news data from NBC and present areas for future exploration.