Peter N. Yianilos

The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical experiments

This paper presents the theory, design principles, implementation and performance results of PicHunter, a prototype content-based image retrieval (CBIR) system. In addition, this document presents the rationale, design and results of psychophysical experiments that were conducted to address some key issues that arose during PicHunters development. The PicHunter project makes four primary contributions to research on CBIR. First, PicHunter represents a simple instance of a general Bayesian framework which we describe for using relevance feedback to direct a search. With an explicit model of what users would do, given the target image they want, PicHunter uses Bayess rule to predict the target they want, given their actions. This is done via a probability distribution over possible image targets, rather than by refining a query. Second, an entropy-minimizing display algorithm is described that attempts to maximize the information obtained from a user at each iteration of the search. Third, PicHunter makes use of hidden annotation rather than a possibly inaccurate/inconsistent annotation structure that the user must learn and make queries in. Finally, PicHunter introduces two experimental paradigms to quantitatively evaluate the performance of the system, and psychophysical experiments are presented that support the theoretical claims.

Learning string-edit distance

In many applications, it is necessary to determine the similarity of two strings. A widely-used notion of string similarity is the edit distance: the minimum number of insertions, deletions, and substitutions required to transform one string into the other. In this report, we provide a stochastic model for string-edit distance. Our stochastic model allows us to learn a string-edit distance function from a corpus of examples. We illustrate the utility of our approach by applying it to the difficult problem of learning the pronunciation of words in conversational speech. In this application, we learn a string-edit distance with nearly one-fifth the error rate of the untrained Levenshtein distance. Our approach is applicable to any string classification problem that may be solved using a similarity function against a database of labeled prototypes.

PicHunter: Bayesian relevance feedback for image retrieval

This paper describes PicHunter, an image retrieval system that implements a novel approach to relevance feedback, such that the entire history of user selections contributes to the systems estimate of the users goal image. To accomplish this, PicHunter uses Bayesian learning based on a probabilistic model of a users behavior. The predictions of this model are combined with the selections made during a search to estimate the probability associated with each image. These probabilities are then used to select images for display. Details of our model of a users behavior were tuned using an off-line leaning algorithm. For clarity, our studies were done with the simplest possible user interface but the algorithm can easily be incorporated into systems which support complex queries, including most previously proposed systems. However, even with this constraint and simple image features, PicHunter is able to locate randomly selected targets in a database of 4522 images after displaying an average of only 55 groups of 4 images which is over 10 times better than chance. We therefore expect that the performance of current image database retrieval systems can be improved by incorporation of the techniques described here.

Feature-based face recognition using mixture-distance

We consider the problem of feature-based face recognition in the setting where only a single example of each face is available for training. The mixture-distance technique we introduce achieves a recognition rate of 95% on a database of 685 people in which each face is represented by 30 measured distances. This is currently the best recorded recognition rate for a feature-based system applied to a database of this size. By comparison, nearest neighbor search using Euclidean distance yields 84%. In our work a novel distance function is constructed based on local second order statistics as estimated by modeling the training data as a mixture of normal densities. We report on the results from mixtures of several sizes. We demonstrate that a flat mixture of mixtures performs as well as the best model and therefore represents an effective solution to the model selection problem. A mixture perspective is also taken for individual Gaussians to choose between first order (variance) and second order (covariance) models. Here an approximation to flat combination is proposed and seen to perform well in practice. Our results demonstrate that even in the absence of multiple training examples for each class, it is sometimes possible to infer from a statistical model of training data, a significantly improved distance function for use in pattern recognition.

A prototype implementation of archival Intermemory

An Archival Intermemory solves the problem of highly survivable digital data storage in the spirit of the Internet. In this paper we describe a prototype implementation of Intermemory, including an overall system architecture and implementations of key system components. The result is a working Intermemory that tolerates up to 17 simultaneous node failures, and includes a Web gateway for browser-based access to data. Our work demonstrates the basic feasibility of Intermemory and represents signi cant progress towards a deployable system.

Towards an archival Intermemory

We propose a self-organizing archival Intermemory. That is, a noncommercial subscriber-provided distributed information storage service built on the existing Internet. Given an assumption of continued growth in the memorys total size, a subscribers participation for only a finite time can nevertheless ensure archival preservation of the subscribers data. Information disperses through the network over time and memories become more difficult to erase as they age. The probability of losing an old memory given random node failures is vanishingly small-and an adversary would have to corrupt hundreds of thousands of nodes to destroy a very old memory. This paper presents a framework for the design of an Intermemory, and considers certain aspects of the design in greater detail. In particular, the aspects of addressing, space efficiency, and redundant coding are discussed.

Hidden annotation in content based image retrieval

The Bayesian relevance feedback approach introduced with the PicHunter system (Cox, Miller, Omohundro and Yianilos, Int. Conf. on Pattern Recognition, pp. 361-369, 1996) is extended to include hidden semantic attributes. The general approach is motivated and experimental results are presented that demonstrate significant reductions in search times (28-32%) using these annotations

Psychophysical studies of the performance of an image database retrieval system

We describe psychophysical experiments conducted to study PicHunter, a content-based image retrieval (CBIR) system. Experiment 1 studies the importance of using (a) semantic information, (2) memory of earlier input and (3) relative, rather than absolute, judgements of image similarity. The target testing paradigm is used in which a user must search for an image identical to a target. We find that the best performance comes from a version of PicHunter that uses only semantic cues, with memory and relative similarity judgements. Second best is use of both pictorial and semantic cues, with memory and relative similarity judgements. Most reports of CBIR systems provide only qualitative measures of performance based on how similar retrieved images are to a target. Experiment 2 puts PicHunter into this context with a more rigorous test. We first establish a baseline for our database by measuring the time required to find an image that is similar to a target when the images are presented in random order. Although PicHunters performance is measurably better than this, the test is weak because even random presentation of images yields reasonably short search times. This casts doubt on the strength of results given in other reports where no baseline is established.

Psychophysical experiments on the PicHunter image retrieval system

Psychophysical experiments were conducted on PicHunter, a content-based image retrieval (CBIR) experimental prototype with the following properties: (1) Based on a model of how users respond, it uses Bayess rule to predict what target users want, given their actions. (2) It possesses an extremely simple user interface. (3) It employs an entropy-based scheme to improve convergence. (4) It introduces a paradigm for assessing the performance of CBIR systems. Experiments 1-3 studied human judgment of image similarity to obtain data for the model. Experiment 4 studied the importance of using: (a) semantic information, (b) memory of earlier input and (c) relative and absolute judgments of similarity. Experiment 5 tested an approach that we propose for comparing performances of CBIR systems objectively. Finally, experiment 6 evaluated the most informative display-updating scheme that is based on entropy minimization, and confirmed earlier simulation results. These experiments represent one of the first attempts to quantify CBIR performance based on psychophysical studies, and they provide valuable data for improving CBIR algorithms. Even though they were designed with PicHunter in mind, their results can be applied to any CBIR system and, more generally, to any system that involves judgment of image similarity by humans

Towards EM-style algorithms for a posteriori optimization of normal mixtures

Expectation maximization (EM) provides a simple and elegant approach to the problem of optimizing the parameters of a normal mixture on an unlabeled dataset. To accomplish this, EM iteratively reweights the elements of the dataset until a locally optimal normal mixture is obtained. This paper explores the intriguing question of whether such an EM-style algorithm exists for the related and apparently more difficult problem of finding a normal mixture that maximizes the a posteriori class probabilities of a labeled dataset. We expose a fundamental degeneracy in the relationship between a normal mixture and the a posteriori class probability functions that it induces, and use this degeneracy to prove that reweighting a dataset can almost always give rise to a normal mixture exhibiting any desired class function behavior. This establishes that EM-style approaches are sufficiently expressive for a posteriori optimization problems and opens the way to the design of new algorithms for them.