Kevin Bailly

Facial Action Recognition Combining Heterogeneous Features via Multikernel Learning

This paper presents our response to the first international challenge on facial emotion recognition and analysis. We propose to combine different types of features to automatically detect action units (AUs) in facial images. We use one multikernel support vector machine (SVM) for each AU we want to detect. The first kernel matrix is computed using local Gabor binary pattern histograms and a histogram intersection kernel. The second kernel matrix is computed from active appearance model coefficients and a radial basis function kernel. During the training step, we combine these two types of features using the recently proposed SimpleMKL algorithm. SVM outputs are then averaged to exploit temporal information in the sequence. To evaluate our system, we perform deep experimentation on several key issues: influence of features and kernel function in histogram-based SVM approaches, influence of spatially independent information versus geometric local appearance information and benefits of combining both, sensitivity to training data, and interest of temporal context adaptation. We also compare our results with those of the other participants and try to explain why our method had the best performance during the facial expression recognition and analysis challenge.

Multiple kernel learning SVM and statistical validation for facial landmark detection

In this paper we present a robust and accurate method to detect 17 facial landmarks in expressive face images. We introduce a new multi-resolution framework based on the recent multiple kernel algorithm. Low resolution patches carry the global information of the face and give a coarse but robust detection of the desired landmark. High resolution patches, using local details, refine this location. This process is combined with a bootstrap process and a statistical validation, both improving the system robustness. Combining independent point detection and prior knowledge on the point distribution, the proposed detector is robust to variable lighting conditions and facial expressions. This detector is tested on several databases and the results reported can be compared favorably with the current state of the art point detectors.

Combining AAM coefficients with LGBP histograms in the multi-kernel SVM framework to detect facial action units

This study presents a combination of geometric and appearance features used to automatically detect Action Units in face images. We use one multi-kernel SVM for each Action Unit we want to detect. The first kernel matrix is computed using Local Gabor Binary Pattern (LGBP) histograms and a histogram intersection kernel. The second kernel matrix is computed from AAM coefficients and a RBF kernel. During the training step, we combine these two type s of features using the recent SimpleMKL algorithm. SVM outputs are then filtered to exploit dynamic relationships between Action Units.

Pairwise Conditional Random Forests for Facial Expression Recognition

Facial expression can be seen as the dynamic variation of ones appearance over time. Successful recognition thus involves finding representations of high-dimensional spatiotemporal patterns that can be generalized to unseen facial morphologies and variations of the expression dynamics. In this paper, we propose to learn Random Forests from heterogeneous derivative features (e.g. facial fiducial point movements or texture variations) upon pairs of images. Those forests are conditioned on the expression label of the first frame to reduce the variability of the ongoing expression transitions. When testing on a specific frame of a video, pairs are created between this frame and the previous ones. Predictions for each previous frame are used to draw trees from Pairwise Conditional Random Forests (PCRF) whose pairwise outputs are averaged over time to produce robust estimates. As such, PCRF appears as a natural extension of Random Forests to learn spatio-temporal patterns, that leads to significant improvements over standard Random Forests as well as state-of-the-art approaches on several facial expression benchmarks.

Impact of action unit detection in automatic emotion recognition

In this paper, we investigate the interest of action unit (AU) detection for automatic emotion recognition. We propose and compare two emotion detectors: the first works directly on a high-dimensional feature space and the second projects facial image in the low-dimensional space of AU intensities before recognizing emotion. In both approaches, facial images are coded by local Gabor binary pattern (LGBP) histogram differences. These features reduce the sensitivity to subject identity by computing differences between two LGBP histograms: one computed on an expressive image and the other synthesized and approaching the one we would compute on a neutral face of the same subject. As classifiers, we test support vector machines with different kernels. A new kernel is proposed, the histogram difference intersection kernel that increases classification performances. This kernel is well suited when exploiting the proposed histogram differences. Thorough experiments on three challenging databases (respectively, the Cohn-Kanade, MMI and Bosphorus databases) show the accuracy of our AU and emotion detectors. They lead to significant conclusions on three critical issues: (1) the interest of combining different training databases labeled by different AU coders, (2) the influence of each AU according to its type and detection accuracy on emotion recognition and (3) the sensitivity to identity variations.

2009 Special Issue: Boosting feature selection for Neural Network based regression

The head pose estimation problem is well known to be a challenging task in computer vision and is a useful tool for several applications involving human-computer interaction. This problem can be stated as a regression one where the input is an image and the output is pan and tilt angles. Finding the optimal regression is a hard problem because of the high dimensionality of the input (number of image pixels) and the large variety of morphologies and illumination. We propose a new method combining a boosting strategy for feature selection and a neural network for the regression. Potential features are a very large set of Haar-like wavelets which are well known to be adapted to face image processing. To achieve the feature selection, a new Fuzzy Functional Criterion (FFC) is introduced which is able to evaluate the link between a feature and the output without any estimation of the joint probability density function as in the Mutual Information. The boosting strategy uses this criterion at each step: features are evaluated by the FFC using weights on examples computed from the error produced by the neural network trained at the previous step. Tests are carried out on the commonly used Pointing 04 database and compared with three state-of-the-art methods. We also evaluate the accuracy of the estimation on FacePix, a database with a high angular resolution. Our method is compared positively to a Convolutional Neural Network, which is well known to incorporate feature extraction in its first layers.

Serious games to teach social interactions and emotions to individuals with autism spectrum disorders (ASD)

The use of information communication technologies (ICTs) in therapy offers new perspectives for treating many domains in individuals with autism spectrum disorders (ASD) because they can be used in many different ways and settings and they are attractive to the patients. We reviewed the available literature on serious games that are used to teach social interactions to individuals with ASD. After screening the Medline, Science Direct and ACM Digital Library databases, we found a total of 31 serious games: 16 that targeted emotion recognition or production and 15 that targeted social skills. There was a significant correlation between the number of reports per year and the year of publication. Serious games appeared promising because they can support training on many different skills and they favour interactions in diverse contexts and situations, some of which may resemble real life. However, the currently available serious games exhibit some limitations: (i) most of them are developed for High-Functioning individuals; (ii) their clinical validation has rarely met the evidence-based medicine standards; (iii) the game design is not usually described; and, (iv) in many cases, the clinical validation and playability/game design are not compatible.

Automatic Facial Action Detection Using Histogram Variation Between Emotional States

This article presents an appearance based method to detect automatically facial actions. Our approach focuses on reducing features sensitivity to identity of the subject. We compute from an expressive image a Local Gabor Binary Pattern (LGBP) histogram and synthesize a LGBP histogram approaching the one we would compute on a neutral face. Difference between these two histograms are used as inputs of Support Vector Machine (SVM) binary detectors associated with a new kernel: the Histogram Difference Intersection (HDI) kernel. Experimental results carried out for 16 Action Units (AUs) on the benchmark Cohn-Kanade database can be compared favorably with two state-of-the-art methods.

Dynamic facial expression recognition by joint static and multi-time gap transition classification

Automatic facial expression classification is a challenging problem for developing intelligent human-computer interaction systems. In order to take into account the expression dynamics, existing works usually make the assumption that a specific facial expression is displayed with a pre-segmented evolution, i.e. starting from neutral and finishing on an apex frame. In this paper, we propose a method to train a transition classifier from pairs of images. This transition classifier is applied at multiple time gaps and the output probabilities are fused along with a static estimation. We eventually show that our approach yields state-of-the-art accuracy on popular datasets without exploiting any such prior on the segmentation of the expression.

Locating facial landmarks with binary map cross-correlations

Precise facial landmark localization in still images is a key step for many face analysis applications, such as biometrics or automatic emotion recognition. In this paper, we propose a framework for facial point detection in frontal and near-frontal images. We introduce a new appearance model based on binary map cross-correlations that efficiently uses LBP and LPQ in a localization context. Inclusion of shape-related constraints is performed by a nonparametric voting method using relational properties within triplets of points, designed to correct outliers without losing precision for accurately detected points. We tested our systems performance on the widely used as benchmark BioID database obtaining state-of-the-art results. We also discuss evaluation metrics used to compare facial landmarking systems and which have been mixed up in recent literature.