Albert C. Cruz

A psychologically-inspired match-score fusion mode for video-based facial expression recognition

Communication between humans is complex and is not limited to verbal signals; emotions are conveyed with gesture, pose and facial expression. Facial Emotion Recognition and Analysis (FERA), the techniques by which non-verbal communication is quantified, is an exemplar case where humans consistently outperform computer methods. While the field of FERA has seen many advances, no system has been proposed which scales well to very large data sets. The challenge for computer vision is how to automatically and nonheuristically downsample the data while maintaining the maximum representational power that does not sacrifice accuracy. In this paper, we propose a method inspired by human vision and attention theory [2]. Video is segmented into temporal partitions with a dynamic sampling rate based on the frequency of visual information. Regions are homogenized by a match-score fusion technique. The approach is shown to provide classification rates higher than the baseline on the AVEC 2011 video-subchallenge dataset [15].

Vision and Attention Theory Based Sampling for Continuous Facial Emotion Recognition

Affective computing-the emergent field in which computers detect emotions and project appropriate expressions of their own-has reached a bottleneck where algorithms are not able to infer a persons emotions from natural and spontaneous facial expressions captured in video. While the field of emotion recognition has seen many advances in the past decade, a facial emotion recognition approach has not yet been revealed which performs well in unconstrained settings. In this paper, we propose a principled method which addresses the temporal dynamics of facial emotions and expressions in video with a sampling approach inspired from human perceptual psychology. We test the efficacy of the method on the Audio/Visual Emotion Challenge 2011 and 2012, CohnKanade and the MMI Facial Expression Database. The method shows an average improvement of 9.8 percent over the baseline for weighted accuracy on the Audio/Visual Emotion Challenge 2011 video-based frame-level subchallenge testing set.

Facial emotion recognition with anisotropic inhibited Gabor energy histograms

This paper presents a novel image descriptor called Derivative Variation Pattern (DVP) and its application to face and palmprint recognition. DVP captures image variations in both the frequency and the spatial domains. The effects of uncontrolled illumination are compensated in the frequency domain by discarding the illumination affected frequencies. Image pixels are encoded as binary patterns based on the higher-order spatial derivatives computed in the spatial domain. The proposed descriptor was evaluated on the Extended Yale-B and FERET face databases, and the PolyU palmprint database. Experimental results demonstrate the effectiveness of the DVP descriptor in both the face and the palmprint recognition tasks under uncontrolled illuminations. State-of-the-art approaches have yet to deliver a feature representation for facial emotion recognition that can be applied to non-trivial unconstrained, continuous video data sets. Initially, research advanced with the use of Gabor energy filters. However, in recent work more attention has been given to other features. Gabor energy filters lack generalization needed in unconstrained situations. Additionally, they result in an undesirably high feature vector dimensionality. Nontrivial data sets have millions of samples; feature vectors must be as low dimensional as possible. We propose a novel texture feature based on Gabor energy filters that offers generalization with a background texture suppression component and is as compact as possible due to a maximal response representation and local histograms. We improve performance on the non-trivial Audio/Visual Emotion Challenge 2012 grandchallenge data set.

Facial emotion recognition with expression energy

Facial emotion recognition, the inference of an emotion from apparent facial expressions, in unconstrained settings is a typical case where algorithms perform poorly. A property of the AVEC2012 data set is that individuals in testing data are not encountered in training data. In these situations, conventional approaches suffer because models developed from training data cannot properly discriminate unforeseen testing samples. Additional information beyond the feature vectors is required for successful detection of emotions. We propose two similarity metrics that address the problems of a conventional approach: neutral similarity, measuring the intensity of an expression; and temporal similarity, measuring changes in an expression over time. These similarities are taken to be the energy of facial expressions, measured with a SIFT-based warping process. Our method improves correlation by 35.5% over the baseline approach on the frame-level sub-challenge.

A biologically inspired approach for fusing facial expression and appearance for emotion recognition

Facial emotion recognition from video is an exemplar case where both humans and computers underperform. In recent emotion recognition competitions, top approaches were using either geometric relationships that best captured facial dynamics or an accurate registration technique to develop appearance features. These two methods capture two different types of facial information similarly to how the human visual system divides information when perceiving faces. In this paper, we propose a biologically-inspired fusion approach that emulates this process. The efficacy of the approach is tested with the Audio/Visual Emotion Challenge 2011 data set, a non-trivial data set where state-of-the-art approaches perform under chance. The proposed approach increases classification rates by 18.5% on publicly available data.

Quantification of Cinematography Semiotics for Video-based Facial Emotion Recognition in the EmotiW 2015 Grand Challenge

The Emotion Recognition in the Wild challenge poses significant problems to state of the art auditory and visual affect quantification systems. To overcome the challenges, we investigate supplementary meta features based on film semiotics. Movie scenes are often presented and arranged in such a way as to amplify the emotion interpreted by the viewing audience. This technique is referred to as mise en scene in the film industry and involves strict and intentional control of color palette, light source color, and arrangement of actors and objects in the scene. To this end, two algorithms for extracting mise en scene information are proposed. Rule of thirds based motion history histograms detect motion along rule of thirds guidelines. Rule of thirds color layout descriptors compactly describe a scene at rule of thirds intersections. A comprehensive system is proposed that measures expression, emotion, vocalics, syntax, semantics, and film-based meta information. The proposed mise en scene features have a higher classification rate and ROC area than LBP-TOP features on the validation set of the EmotiW 2015 challenge. The complete system improves classification performance over the baseline algorithm by 3.17% on the testing set.

Background suppressing Gabor energy filtering

In the field of facial emotion recognition, early research advanced with the use of Gabor filters. However, these filters lack generalization and result in undesirably large feature vector size. In recent work, more attention has been given to other local appearance features. Two desired characteristics in a facial appearance feature are generalization capability, and the compactness of representation. In this paper, we propose a novel texture feature inspired by Gabor energy filters, called background suppressing Gabor energy filtering. The feature has a generalization component that removes background texture. It has a reduced feature vector size due to maximal representation and soft orientation histograms, and it is a white box representation. We demonstrate improved performance on the non-trivial Audio/Visual Emotion Challenge 2012 grand-challenge dataset by a factor of 7.17 over the Gabor filter on the development set. We also demonstrate applicability of our approach beyond facial emotion recognition which yields improved classification rate over the Gabor filter for four bioimaging datasets by an average of 8.22%.

One shot emotion scores for facial emotion recognition

Facial emotion recognition in unconstrained settings is a difficult task. They key problems are that people express their emotions in ways that are different from other people, and, for large datasets, there are not enough examples of a specific person to model his/her emotion. A model for predicting emotions will not generalize well to predicting the emotions of a person who has not been encountered during the training. We propose a system that addresses these issues by matching a face video to references of emotion. It does not require examples from the person in the video being queried. We compute the matching scores without requiring fine registration. The method is called one-shot emotion score. We improve classification rate of interdataset experiments over a baseline system by 23% when training on MMI and testing on CK+.

Automated spatial analysis of ARK2: Putative link between microtubules and cell polarity

In leaves of A. thaliana, there exists an intricate network of epidermal surface layer cells responsible for anatomical stability and vigor of flexibility to the entire leaf. Rho GTPases direct this organization of cell polarity, but full understanding of the underlying mechanisms demands further inquiry. We conduct two experiments: (1) a novel procedure is proposed that could be used in other life and plant science studies to quantify microtubule orientation, and (2) shape analysis. We hypothesize ARK2 as a putative interactor in cell polarity maintenance through stabilization of microtubule ordering. We are the first to automate pavement cell phenotype analysis for cell polarity and microtubule orientation. Breakthroughs in the signaling network regulating leaf cell polarity and development will lead science into the frontier of genetically modifying leaves to dramatically increase Earths plant biomass; impending food shortages in the 21st century will be well served by such research.

Frequency divergence image: A novel method for action recognition

Action recognition systems have the potential to support clinicians, coaches and physical therapists in identifying important adopted movement patterns which could aid injury detection potential or inform rehabilitation strategies. Currently, motion capture systems, structured light pattern and time-of-flight sensors have utilization limitations that place constraints on their use outside of the laboratory setting. For this reason, we propose a system for human action recognition from video. The method presented in this work has utility with patient populations, such as Parkinsons disease, Alzheimers disease, multiple sclerosis and dementia, outside of laboratory setting to detect the degree of which, and progression of, gait pathology. We developed a novel vision algorithm for template matching—the characterization of the motion in a video sequence. The method, titled Frequency Divergence Image, is a paradigm shift in template matching methods. Template matching methods measure macro-motion, whereas the proposed method detects micro-motion that differs from the flow of the action. We show that micro-cues improve prediction performance of human action on a real-world data set. We demonstrate a 9.15% improvement in classification accuracy over the original Motion History Image formulation when used with a convolutional neural network. Future work will focus on the deployment of the system to identify gait pathology from various patient populations.