Garrison W. Cottrell

Behavior recognition via sparse spatio-temporal features

A common trend in object recognition is to detect and leverage the use of sparse, informative feature points. The use of such features makes the problem more manageable while providing increased robustness to noise and pose variation. In this work we develop an extension of these ideas to the spatio-temporal case. For this purpose, we show that the direct 3D counterparts to commonly used 2D interest point detectors are inadequate, and we propose an alternative. Anchoring off of these interest points, we devise a recognition algorithm based on spatio-temporally windowed data. We present recognition results on a variety of datasets including both human and rodent behavior.

SUN: A Bayesian framework for saliency using natural statistics

We propose a definition of saliency by considering what the visual system is trying to optimize when directing attention. The resulting model is a Bayesian framework from which bottom-up saliency emerges naturally as the self-information of visual features, and overall saliency (incorporating top-down information with bottom-up saliency) emerges as the pointwise mutual information between the features and the target when searching for a target. An implementation of our framework demonstrates that our models bottom-up saliency maps perform as well as or better than existing algorithms in predicting peoples fixations in free viewing. Unlike existing saliency measures, which depend on the statistics of the particular image being viewed, our measure of saliency is derived from natural image statistics, obtained in advance from a collection of natural images. For this reason, we call our model SUN (Saliency Using Natural statistics). A measure of saliency based on natural image statistics, rather than based on a single test image, provides a straightforward explanation for many search asymmetries observed in humans; the statistics of a single test image lead to predictions that are not consistent with these asymmetries. In our model, saliency is computed locally, which is consistent with the neuroanatomy of the early visual system and results in an efficient algorithm with few free parameters.

Early lateralization and orientation tuning for face, word, and object processing in the visual cortex.

Event-related potential (ERP) studies of the human brain have shown that object categories can be reliably distinguished as early as 130-170 ms on the surface of occipito-temporal cortex, peaking at the level of the N170 component. Consistent with this finding, neuropsychological and neuroimaging studies suggest major functional distinctions within the human object recognition system, particularly in hemispheric advantage, between the processing of words (left), faces (right), and objects (bilateral). Given these observations, our aim was to (1) characterize the differential response properties of the N170 to pictures of faces, objects, and words across hemispheres; and (2) test whether an effect of inversion for highly familiar and monooriented nonface stimuli such as printed words can be observed at the level of the N170. Scalp EEG (53 channels) was recorded in 15 subjects performing an orientation decision task with pictures of faces, words, and cars presented upright or inverted. All three categories elicited at the same latency a robust N170 component associated with a positive counterpart at centro-frontal sites (vertex-positive potential, VPP). While there were minor amplitude differences at the level of the occipital medial P1 between linguistic and nonlinguistic categories, scalp topographies and source analyses indicated strong hemispheric and orientation effects starting at the level of the N170, which was right lateralized for faces, smaller and bilateral for cars, and as large for printed words in the left hemisphere as for faces. The entire N170/VPP complex was accounted for by two dipolar sources located in the lateral inferior occipital cortex/posterior fusiform gyrus. These two locations were roughly equivalent across conditions but differed in strength and lateralization. Inversion delayed the N170 (and VPP) response for all categories, with an increasing delay for cars, words, and faces, respectively, as suggested by source modeling analysis. Such results show that early processes in object recognition respond to category-specific visual information, and are associated with strong lateralization and orientation bias.

Transmitting and Decoding Facial Expressions

This article examines the human face as a transmitter of expression signals and the brain as a decoder of these expression signals. If the face has evolved to optimize transmission of such signals, the basic facial expressions should have minimal overlap in their information. If the brain has evolved to optimize categorization of expressions, it should be efficient with the information available from the transmitter for the task. In this article, we characterize the information underlying the recognition of the six basic facial expression signals and evaluate how efficiently each expression is decoded by the underlying brain structures.

Connectionist models of face processing : a survey

Abstract Connectionist models of face recognition, identification, and categorization have appeared recently in several disciplines, including psychology, computer science, and engineering. We present a review of these models with the goal of complementing a recent survey by Samal and Iyengar [Pattern Recognition25, 65–77 (1992)] of nonconnectionist approaches to the problem of the automatic face recognition. We concentrate on models that use linear autoassociative networks, nonlinear autoassociative (or compression) and/or heteroassociative backpropagation networks. One advantage of these models over some nonconnectionist approaches is that analyzable features emerge naturally from image-based codes, and hence the problem of feature selection and segmentation from faces can be avoided.

Fusion Via a Linear Combination of Scores

We present a thorough analysis of the capabilities of the linear combination (LC) model for fusion of information retrieval systems. The LC model combines the results lists of multiple IR systems by scoring each document using a weighted sum of the scores from each of the component systems. We first present both empirical and analytical justification for the hypotheses that such a model should only be used when the systems involved have high performance, a large overlap of relevant documents, and a small overlap of nonrelevant documents. The empirical approach allows us to very accurately predict the performance of a combined system. We also derive a formula for a theoretically optimal weighting scheme for combining 2 systems. We introduce d—the difference between the average score on relevant documents and the average score on nonrelevant documents—as a performance measure which not only allows mathematical reasoning about system performance, but also allows the selection of weights which generalize well to new documents. We describe a number of experiments involving large numbers of different IR systems which support these findings.

Automatic combination of multiple ranked retrieval systems

Retrieval performance can often be improved significantly by using a number of different retrieval algorithms and combining the results, in contrast to using just a single retrieval algorithm. This is because different retrieval algorithms, or retrieval experts, often emphasize different document and query features when determining relevance and therefore retrieve different sets of documents. However, it is unclear how the different experts are to be combined, in general, to yield a superior overall estimate. We propose a method by which the relevance estimates made by different experts can be automatically combined to result in superior retrieval performance. We apply the method to two expert combination tasks. The applications demonstrate that the method can identify high performance combinations of experts and also is a novel means for determining the combined effectiveness of experts.

Two Fixations Suffice in Face Recognition

It is well known that there exist preferred landing positions for eye fixations in visual word recognition. However, the existence of preferred landing positions in face recognition is less well established. It is also unknown how many fixations are required to recognize a face. To investigate these questions, we recorded eye movements during face recognition. During an otherwise standard face-recognition task, subjects were allowed a variable number of fixations before the stimulus was masked. We found that optimal recognition performance is achieved with two fixations; performance does not improve with additional fixations. The distribution of the first fixation is just to the left of the center of the nose, and that of the second fixation is around the center of the nose. Thus, these appear to be the preferred landing positions for face recognition. Furthermore, the fixations made during face learning differ in location from those made during face recognition and are also more variable in duration; this suggests that different strategies are used for face learning and face recognition.

SUN: Top-down saliency using natural statistics

When people try to find particular objects in natural scenes they make extensive use of knowledge about how and where objects tend to appear in a scene. Although many forms of such “top-down” knowledge have been incorporated into saliency map models of visual search, surprisingly, the role of object appearance has been infrequently investigated. Here we present an appearance-based saliency model derived in a Bayesian framework. We compare our approach with both bottom-up saliency algorithms as well as the state-of-the-art Contextual Guidance model of Torralba et al. (2006) at predicting human fixations. Although both top-down approaches use very different types of information, they achieve similar performance; each substantially better than the purely bottom-up models. Our experiments reveal that a simple model of object appearance can predict human fixations quite well, even making the same mistakes as people.

Robust classification of objects, faces, and flowers using natural image statistics

Classification of images in many category datasbets has rapidly improved in recent years. However, systems that perform well on particular datasets typically have one or more limitations such as a failure to generalize across visual tasks (e.g., requiring a face detector or extensive retuning of parameters), insufficient translation invariance, inability to cope with partial views and occlusion, or significant performance degradation as the number of classes is increased. Here we attempt to overcome these challenges using a model that combines sequential visual attention using fixations with sparse coding. The models biologically-inspired filters are acquired using unsupervised learning applied to natural image patches. Using only a single feature type, our approach achieves 78.5% accuracy on Caltech-101 and 75.2% on the 102 Flowers dataset when trained on 30 instances per class and it achieves 92.7% accuracy on the AR Face database with 1 training instance per person. The same features and parameters are used across these datasets to illustrate its robust performance.