Anıl Yüce

Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data

We study the relevance of context-learning for handling asynchrony of annotation.We unite audiovisual and physiological data for continuous affect analysis.We propose multi-time resolution features extraction from multimodal data.The use of context-learning allows to include reaction time delay of raters.Fusion of audiovisual and physiological data performs best on arousal and valence. Automatic emotion recognition systems based on supervised machine learning require reliable annotation of affective behaviours to build useful models. Whereas the dimensional approach is getting more and more popular for rating affective behaviours in continuous time domains, e.g., arousal and valence, methodologies to take into account reaction lags of the human raters are still rare. We therefore investigate the relevance of using machine learning algorithms able to integrate contextual information in the modelling, like long short-term memory recurrent neural networks do, to automatically predict emotion from several (asynchronous) raters in continuous time domains, i.e., arousal and valence. Evaluations are performed on the recently proposed RECOLA multimodal database (27 subjects, 5? min of data and six raters for each), which includes audio, video, and physiological (ECG, EDA) data. In fact, studies uniting audiovisual and physiological information are still very rare. Features are extracted with various window sizes for each modality and performance for the automatic emotion prediction is compared for both different architectures of neural networks and fusion approaches (feature-level/decision-level). The results show that: (i) LSTM network can deal with (asynchronous) dependencies found between continuous ratings of emotion with video data, (ii) the prediction of the emotional valence requires longer analysis window than for arousal and (iii) a decision-level fusion leads to better performance than a feature-level fusion. The best performance (concordance correlation coefficient) for the multimodal emotion prediction is 0.804 for arousal and 0.528 for valence.

Detecting emotional stress from facial expressions for driving safety

Monitoring the attentive and emotional status of the driver is critical for the safety and comfort of driving. In this work a real-time non-intrusive monitoring system is developed, which detects the emotional states of the driver by analyzing facial expressions. The system considers two negative basic emotions, anger and disgust, as stress related emotions. We detect an individual emotion in each video frame and the decision on the stress level is made on sequence level. Experimental results show that the developed system operates very well on simulated data even with generic models. An additional pose normalization step reduces the impact of pose mismatch due to camera setup and pose variation, and hence improves the detection accuracy further.

Discriminant multi-label manifold embedding for facial Action Unit detection

This article describes a system for participation in the Facial Expression Recognition and Analysis (FERA2015) sub-challenge for spontaneous action unit occurrence detection. The problem of AU detection is a multi-label classification problem by its nature, which is a fact overseen by most existing work. The correlation information between AUs has the potential of increasing the detection accuracy. We investigate the multi-label AU detection problem by embedding the data on low dimensional manifolds which preserve multi-label correlation. For this, we apply the multi-label Discriminant Laplacian Embedding (DLE) method as an extension to our base system. The system uses SIFT features around a set of facial landmarks that is enhanced with the use of additional non-salient points around transient facial features. Both the base system and the DLE extension show better performance than the challenge baseline results for the two databases in the challenge, and achieve close to 50% as F1-measure on the testing partition in average (9.9% higher than the baseline, in the best case). The DLE extension proves useful for certain AUs, but also shows the need for more analysis to assess the benefits in general.

Distribution-based iterative pairwise classification of emotions in the wild using LGBP-TOP

Automatic facial expression analysis promises to be a game-changer in many application areas. But before this promise can be fulfilled, it has to move from the laboratory into the wild. The Emotion Recognition in the Wild challenge provides an opportunity to develop approaches in this direction. We propose a novel Distribution-based Pairwise Iterative Classification scheme, which outperforms standard multi-class classification on this challenge data. We also verify that the recently proposed dynamic appearance descriptor, Local Gabor Patterns on Three Orthogonal Planes, performs well on this real-world data, indicating that it is robust to the type of facial misalignments that can be expected in such scenarios. Finally, we provide details of ACTC, our affective computing tools on the cloud, which is a new resource for researchers in the field of affective computing.

Improved local binary pattern based action unit detection using morphological and bilateral filters

Automatic facial action unit (AU) detection in videos is the key ingredient to all systems that utilize a subject face for either interaction or analysis purposes. With the ever growing range of possible applications, achieving a high accuracy in the simplest possible manner gains even more importance. In this paper, we present new features obtained by applying local binary patterns to images processed by morphological and bilateral filters. We use as features the variations of these patterns between the expressive and neutral faces, and show that we can gain a considerable amount of accuracy increase by simply applying these fundamental image processing tools and choosing the right way of representing the patterns. We also use these features in conjunction with additional features based on facial point geometrical relations between frames and achieve detection rates higher than methods previously proposed, using a small number of features and basic support vector machine classification.

Multiple Local Curvature Gabor Binary Patterns for Facial Action Recognition

Curvature Gabor features have recently been shown to be powerful facial texture descriptors with applications on face recognition. In this paper we introduce their use in facial action unit (AU) detection within a novel framework that combines multiple Local Curvature Gabor Binary Patterns (LCGBP) on different filter sizes and curvature degrees. The proposed system uses the distances of LCGBP histograms between neutral faces and AU containing faces combined with an AU-specific feature selection and classification process. We achieve 98.6% overall accuracy in our tests with the extended Cohn-Kanade database, which is higher than achieved previously by any state-of-the-art method.