Anh Cat Le Ngo

Subtle Expression Recognition Using Optical Strain Weighted Features

Optical strain characterizes the relative amount of displacement by a moving object within a time interval. Its ability to compute any small muscular movements on faces can be advantageous to subtle expression research. This paper proposes a novel optical strain weighted feature extraction scheme for subtle facial micro-expression recognition. Motion information is derived from optical strain magnitudes, which is then pooled spatio-temporally to obtain block-wise weights for the spatial image plane. By simple product with the weights, the resulting feature histograms are intuitively scaled to accommodate the importance of block regions. Experiments conducted on two recent spontaneous micro-expression databases–CASMEII and SMIC, demonstrated promising improvement over the baseline results.

Spontaneous Subtle Expression Recognition: Imbalanced Databases and Solutions

Facial expression analysis has been well studied in recent years; however, these mainly focus on domains of posed or clear facial expressions. Meanwhile, subtle/micro-expressions are rarely analyzed, due to three main difficulties: inter-class similarity (hardly discriminate facial expressions of two subtle emotional states from a person), intra-class dissimilarity (different facial morphology and behaviors of two subjects in one subtle emotion state), and imbalanced sample distribution for each class and subject. This paper aims to solve the last two problems by first employing preprocessing steps: facial registration, cropping and interpolation; and proposes a person-specific AdaBoost classifier with Selective Transfer Machine framework. While preprocessing techniques remove morphological facial differences, the proposed variant of AdaBoost deals with imbalanced characteristics of available subtle expression databases. Performance metrics obtained from experiments on the SMIC and CASME2 spontaneous subtle expression databases confirm that the proposed method improves classification of subtle emotions.

Monogenic Riesz wavelet representation for micro-expression recognition

A monogenic signal is a two-dimensional analytical signal that provides the local information of magnitude, phase, and orientation. While it has been applied on the field of face and expression recognition [1], [2], [3], there are no known usages for subtle facial micro-expressions. In this paper, we propose a feature representation method which succinctly captures these three low-level components at multiple scales. Riesz wavelet transform is employed to obtain multi-scale monogenic wavelets, which are formulated by quaternion representation. Instead of summing up the multi-scale monogenic representations, we consider all monogenic representations across multiple scales as individual features. For classification, two schemes were applied to integrate these multiple feature representations: a fusion-based method which combines the features efficiently and discriminately using the ultra-fast, optimized Multiple Kernel Learning (UFO-MKL) algorithm; and concatenation-based method where the features are combined into a single feature vector and classified by a linear SVM. Experiments carried out on a recent spontaneous micro-expression database demonstrated the capability of the proposed method in outperforming the state-of-the-art monogenic signal approach to solving the micro-expression recognition problem.

Spontaneous subtle expression detection and recognition based on facial strain

Optical strain is an extension of optical flow that is capable of quantifying subtle changes on faces and representing the minute facial motion intensities at the pixel level. This is computationally essential for the relatively new field of spontaneous micro-expression, where subtle expressions can be technically challenging to pinpoint. In this paper, we present a novel method for detecting and recognizing micro-expressions by utilizing facial optical strain magnitudes to construct optical strain features and optical strain weighted features. The two sets of features are then concatenated to form the resultant feature histogram. Experiments were performed on the CASME II and SMIC databases. We demonstrate on both databases, the usefulness of optical strain information and more importantly, that our best approaches are able to outperform the original baseline results for both detection and recognition tasks. A comparison of the proposed method with other existing spatio-temporal feature extraction approaches is also presented. HighlightsThe method proposed is a combination of two optical strain derived features.Optical strain magnitudes were employed to describe fine subtle facial movements.Evaluation was performed in both the detection and recognition tasks.Promising performances were obtained in two micro-expression databases.

Sparsity in Dynamics of Spontaneous Subtle Emotions: Analysis and Application

Subtle emotions are present in diverse real-life situations: in hostile environments, enemies and/or spies maliciouslyconceal their emotions as part of their deception; in life-threatening situations, victims under duress have no choice but to withhold theirreal feelings; in the medical scene, patients with psychological conditions such as depression could either be intentionally orsubconsciously suppressing their anguish from loved ones. Under such circumstances, it is often crucial that these subtle emotions arerecognized before it is too late. These spontaneous subtle emotions are typically expressed through micro-expressions, which are tiny,sudden and short-lived dynamics of facial muscles; thus, such micro-expressions pose a great challenge for visual recognition. Theabrupt but significant dynamics for the recognition task are temporally sparse while the rest, i.e. irrelevant dynamics, are temporallyredundant. In this work, we analyze and enforce sparsity constraints to learn significant temporal and spectral structures whileeliminating irrelevant facial dynamics of micro-expressions, which would ease the challenge in the visual recognition of spontaneoussubtle emotions. The hypothesis is confirmed through experimental results of automatic spontaneous subtle emotion recognition withseveral sparsity levels on CASME II and SMIC, the two well-established and publicly available spontaneous subtle emotion databases.The overall performances of the automatic subtle emotion recognition are boosted when only significant dynamics of the originalsequences are preserved.

Are subtle expressions too sparse to recognize

As subtle emotions are slightly and involuntarily expressed, they need to be recorded by high-speed camera. Though this high frame-per-second rate allows better capture of subtle expressions, it typically generates a lot of redundant frames with rapid varying illumination and noise but without significant motions. The redundancy is analyzed and eliminated by Sparsity-Promoting Dynamic Mode Decomposition (DMDSP), which helps synthesize dynamically condensed sequences. Moreover, DMDSP can also visualize dynamics of subtle expressions in both temporal and spectral domains. As meaningful subtle expressions are temporarily sparse, DMDSP would be able to extract these meaningful dynamics and improve recognition rates of subtle expressions. The hypothesis is evaluated on CASME II, a database of spontaneous subtle facial expressions. Recognition performance measured by F1-score, recall and precision metrics showed a significant leap of improvement when DMDSP is used to preserve a small percentage of meaningful frames in sequences with temporally high sparsity levels.

Visual saliency based on fast nonparametric multidimensional entropy estimation

Bottom-up visual saliency can be computed through information theoretic models but existing methods face significant computational challenges. Whilst nonparametric methods suffer from the curse of dimensionality problem and are computationally expensive, parametric approaches have the difficulty of determining the shape parameters of the distribution models. This paper makes two contributions to information theoretic based visual saliency models. First, we formulate visual saliency as center surround conditional entropy which gives a direct and intuitive interpretation of the center surround mechanism under the information theoretic framework. Second, and more importantly, we introduce a fast nonparametric multidimensional entropy estimation solution to make information theoretic-based saliency models computationally tractable and practicable in realtime applications. We present experimental results on publicly available eye-tracking image databases to demonstrate that the proposed method is competitive to state of the art.

Eulerian emotion magnification for subtle expression recognition

Subtle emotions are expressed through tiny and brief movements of facial muscles, called micro-expressions; thus, recognition of these hidden expressions is as challenging as inspection of microscopic worlds without microscopes. In this paper, we show that through motion magnification, subtle expressions can be realistically exaggerated and become more easily recognisable. We magnify motions of facial expressions in the Eulerian perspective by manipulating their amplitudes or phases. To evaluate effects of exaggerating facial expressions, we use a common framework (LBP-TOP features and SVM classifiers) to perform 5-class subtle emotion recognition on the CASME II corpus, a spontaneous subtle emotion database. According to experimental results, significant improvements in recognition rates of magnified micro-expressions over normal ones are confirmed and measured. Furthermore, we estimate upper bounds of effective magnification factors and empirically corroborate these theoretical calculations with experimental data.

Intrinsic two-dimensional local structures for micro-expression recognition

An elapsed facial emotion involves changes of facial contour due to the motions (such as contraction or stretch) of facial muscles located at the eyes, nose, lips and etc. Thus, the important information such as corners of facial contours that are located in various regions of the face are crucial to the recognition of facial expressions, and even more apparent for micro-expressions. In this paper, we propose the first known notion of employing intrinsic two-dimensional (i2D) local structures to represent these features for micro-expression recognition. To retrieve i2D local structures such as phase and orientation, higher order Riesz transforms are employed by means of monogenic curvature tensors. Experiments performed on micro-expression datasets show the effectiveness of i2D local structures in recognizing micro-expressions.

Automatic apex frame spotting in micro-expression database

Micro-expression usually occurs at high-stakes situations and may provide useful information in the field of behavioral psychology for better interpretion and analysis. Unfortunately, it is technically challenging to detect and recognize micro-expressions due to its brief duration and the subtle facial distortions. Apex frame, which is the instant indicating the most expressive emotional state in a video, is effective to classify the emotion in that particular frame. In this work, we present a novel method to spot the apex frame of a spontaneous micro-expression video sequence. A binary search approach is employed to locate the index of the frame in which the peak facial changes occur. Features from specific facial regions are extracted to better represent and describe the expression details. The defined facial regions are selected based on the action unit and landmark coordinates of the subject, in which case these processes are automated. We consider three distinct feature descriptors to evaluate the reliability of the proposed approach. Improvements of at least 20% are achieved when compared to the baselines.