Alice J. O'Toole

Connectionist models of face processing : a survey

Abstract Connectionist models of face recognition, identification, and categorization have appeared recently in several disciplines, including psychology, computer science, and engineering. We present a review of these models with the goal of complementing a recent survey by Samal and Iyengar [Pattern Recognition25, 65–77 (1992)] of nonconnectionist approaches to the problem of the automatic face recognition. We concentrate on models that use linear autoassociative networks, nonlinear autoassociative (or compression) and/or heteroassociative backpropagation networks. One advantage of these models over some nonconnectionist approaches is that analyzable features emerge naturally from image-based codes, and hence the problem of feature selection and segmentation from faces can be avoided.

Recognizing moving faces: a psychological and neural synthesis

Information for identifying a human face can be found both in the invariant structure of features and in idiosyncratic movements and gestures. When both kinds of information are available, psychological evidence indicates that: (1) dynamic information contributes more to recognition under non-optimal viewing conditions, e.g. poor illumination, low image resolution, recognition from a distance; (2) dynamic information contributes more as a viewers experience with the face increases; and (3) a structure-from-motion analysis can make a perceptually based contribution to face recognition. A recently proposed distributed neural system for face perception, with minor modifications, can accommodate the psychological findings with moving faces.

FRVT 2006 and ICE 2006 Large-Scale Experimental Results

This paper describes the large-scale experimental results from the Face Recognition Vendor Test (FRVT) 2006 and the Iris Challenge Evaluation (ICE) 2006. The FRVT 2006 looked at recognition from high-resolution still frontal face images and 3D face images, and measured performance for still frontal face images taken under controlled and uncontrolled illumination. The ICE 2006 evaluation reported verification performance for both left and right irises. The images in the ICE 2006 intentionally represent a broader range of quality than the ICE 2006 sensor would normally acquire. This includes images that did not pass the quality control software embedded in the sensor. The FRVT 2006 results from controlled still and 3D images document at least an order-of-magnitude improvement in recognition performance over the FRVT 2002. The FRVT 2006 and the ICE 2006 compared recognition performance from high-resolution still frontal face images, 3D face images, and the single-iris images. On the FRVT 2006 and the ICE 2006 data sets, recognition performance was comparable for high-resolution frontal face, 3D face, and the iris images. In an experiment comparing human and algorithms on matching face identity across changes in illumination on frontal face images, the best performing algorithms were more accurate than humans on unfamiliar faces.

Partially Distributed Representations of Objects and Faces in Ventral Temporal Cortex

Object and face representations in ventral temporal (VT) cortex were investigated by combining object confusability data from a computational model of object classification with neural response confusability data from a functional neuroimaging experiment. A pattern-based classification algorithm learned to categorize individual brain maps according to the object category being viewed by the subject. An identical algorithm learned to classify an image-based, view-dependent representation of the stimuli. High correlations were found between the confusability of object categories and the confusability of brain activity maps. This occurred even with the inclusion of multiple views of objects, and when the object classification model was tested with high spatial frequency line drawings of the stimuli. Consistent with a distributed representation of objects in VT cortex, the data indicate that object categories with shared image-based attributes have shared neural structure.

Structural aspects of face recognition and the other-race effect

The other-race effect was examined in a series of experiments and simulations that looked at the relationships among observer ratings of typicality, familiarity, attractiveness, memorability, and the performance variables ofd’ and criterion. Experiment 1 replicated the other-race effect with our Caucasian and Japanese stimuli for both Caucasian and Asian observers. In Experiment 2, we collected ratings from Caucasian observers on the faces used in the recognition task. A Varimax-rotated principal components analysis on the rating and performance data for the Caucasian faces replicated Vokey and Read’s (1992) finding that typicality is composed of two orthogonal components, dissociable via their independent relationships to: (1) attractiveness and familiarity ratings and (2) memorahility ratings. For Japanese faces, however, we fond that typicality was related only to memorahility. Where performance measures were concerned, two additional principal components dominated by criterion and byd’ emerged for Caucasian faces. For the Japanese faces, however, the performance measures ofd’ and criterion merged into a single component that represented a second component of typicality, one orthogonal to thememorability-dominated component. A measure offace representation quality extracted from an autoassociative neural network trained with a majority of Caucasian faces and a minority of Japanese faces was incorporated into the principal components analysis. For both Caucasian and Japanese faces, the neural network measure related both to memorability ratings and to human accuracy measures. Combined, the human data and simulation results indicate that the memorahility component of typicality may be related to small, local, distinctive features, whereas the attractiveness/familiarity component may be more related to the global, shape-based properties of the face.

Theoretical, Statistical, and Practical Perspectives on Pattern-based Classification Approaches to the Analysis of Functional Neuroimaging Data

The goal of pattern-based classification of functional neuroimaging data is to link individual brain activation patterns to the experimental conditions experienced during the scans. These brain-reading analyses advance functional neuroimaging on three fronts. From a technical standpoint, pattern-based classifiers overcome fatal f laws in the status quo inferential and exploratory multivariate approaches by combining pattern-based analyses with a direct link to experimental variables. In theoretical terms, the results that emerge from pattern-based classifiers can offer insight into the nature of neural representations. This shifts the emphasis in functional neuroimaging studies away from localizing brain activity toward understanding how patterns of brain activity encode information. From a practical point of view, pattern-based classifiers are already well established and understood in many areas of cognitive science. These tools are familiar to many researchers and provide a quantitatively sound and qualitatively satisfying answer to most questions addressed in functional neuroimaging studies. Here, we examine the theoretical, statistical, and practical underpinnings of pattern-based classification approaches to functional neuroimaging analyses. Pattern-based classification analyses are well positioned to become the standard approach to analyzing functional neuroimaging data.

Low-dimensional representation of faces in higher dimensions of the face space

Faces can be represented efficiently as a weighted linear combination of the eigenvectors of a covariance matrix of face images. It has also been shown [ J. Opt. Soc. Am.4, 519– 524 ( 1987)] that identifiable faces can be made by using only a subset of the eigenvectors, i.e., those with the largest eigenvalues. This low-dimensional representation is optimal in that it minimizes the squared error between the representation of the face image and the original face image. The present study demonstrates that, whereas this low-dimensional representation is optimal for identifying the physical categories of face, like sex, it is not optimal for recognizing the faces (i.e., discriminating known from unknown faces). Various low-dimensional representations of the faces in the higher dimensions of the face space (i.e., the eigenvectors with smaller eigenvalues) provide better information for face recognition.

More about the Difference between Men and Women: Evidence from Linear Neural Networks and the Principal-Component Approach

The ability of a statistical/neural network to classify faces by sex by means of a pixel-based representation has not been fully investigated. Simulations with pixel-based codes have provided sex-classification results that are less impressive than those reported for measurement-based codes. In no case, however, have the reported pixel-based simulations been optimized for the task of classifying faces by sex. A series of simulations is described in which four network models were applied to the same pixel-based face code. These simulations involved either a radial basis function network or a perceptron as a classifier, preceded or not by a preprocessing step of eigendecomposition. It is shown that performance comparable to that of the measurement-based models can be achieved with pixel-based input (90%) when the data are preprocessed. The effect of the eigendecomposition preprocessing of the faces is then compared with spatial-frequency analysis of face images and analyzed in terms of the perceptual information it captures. It is shown that such an examination may offer insight into the facial aspects important to the sex-classification process. Finally, the contribution of hair information to the performance of the model is evaluated. It is shown that, although the hair contributes to the sex-classification process, it is not the only important contributor.

Overview of the Multiple Biometrics Grand Challenge

The goal of the Multiple Biometrics Grand Challenge (MBGC) is to improve the performance of face and iris recognition technology from biometric samples acquired under unconstrained conditions. The MBGC is organized into three challenge problems. Each challenge problem relaxes the acquisition constraints in different directions. In the Portal Challenge Problem, the goal is to recognize people from near-infrared (NIR) and high definition (HD) video as they walk through a portal. Iris recognition can be performed from the NIR video and face recognition from the HD video. The availability of NIR and HD modalities allows for the development of fusion algorithms. The Still Face Challenge Problem has two primary goals. The first is to improve recognition performance from frontal and off angle still face images taken under uncontrolled indoor and outdoor lighting. The second is to improve recognition performance on still frontal face images that have been resized and compressed, as is required for electronic passports. In the Video Challenge Problem, the goal is to recognize people from video in unconstrained environments. The video is unconstrained in pose, illumination, and camera angle. All three challenge problems include a large data set, experiment descriptions, ground truth, and scoring code.

Face recognition algorithms and the other‐race effect: computational mechanisms for a developmental contact hypothesis

People recognize faces of their own race more accurately than faces of other races. The “contact” hypothesis suggests that this “other-race effect” occurs as a result of the greater experience we have with own- versus other-race faces. The computational mechanisms that may underlie different versions of the contact hypothesis were explored in this study. We replicated the other-race effect with human participants and evaluated four classes of computational face recognition algorithms for the presence of an other-race effect. Consistent with the predictions of a developmentalcontact hypothesis, “experience-based models” demonstrated an other-race effect only when the representational system was developed through experience that warped the perceptual space in a way that was sensitive to the overall structure of the model’s experience with faces of different races. When the model’s representation relied on a feature set optimized to encode the information in the learned faces, experience-based algorithms recognized minority-race faces more accurately than majority-race faces. The results suggest a developmental learning process that warps the perceptual space to enhance the encoding of distinctions relevant for own-race faces. This feature space limits the quality of face representations for other-race faces.