Claudia Lainscsek

Recognizing facial expression: machine learning and application to spontaneous behavior

We present a systematic comparison of machine learning methods applied to the problem of fully automatic recognition of facial expressions. We report results on a series of experiments comparing recognition engines, including AdaBoost, support vector machines, linear discriminant analysis. We also explored feature selection techniques, including the use of AdaBoost for feature selection prior to classification by SVM or LDA. Best results were obtained by selecting a subset of Gabor filters using AdaBoost followed by classification with support vector machines. The system operates in real-time, and obtained 93% correct generalization to novel subjects for a 7-way forced choice on the Cohn-Kanade expression dataset. The outputs of the classifiers change smoothly as a function of time and thus can be used to measure facial expression dynamics. We applied the system to to fully automated recognition of facial actions (FACS). The present system classifies 17 action units, whether they occur singly or in combination with other actions, with a mean accuracy of 94.8%. We present preliminary results for applying this system to spontaneous facial expressions.

Automatic recognition of facial actions in spontaneous expressions

Spontaneous facial expressions differ from posed expressions in both which muscles are moved, and in the dynamics of the movement. Advances in the field of automatic facial expression measurement will require development and assessment on spontaneous behavior. Here we present preliminary results on a task of facial action detection in spontaneous facial expressions. We employ a user independent fully automatic system for real time recognition of facial actions from the Facial Action Coding System (FACS). The system automatically detects frontal faces in the video stream and coded each frame with respect to 20 Action units. The approach applies machine learning methods such as support vector machines and AdaBoost, to texture-based image representations. The output margin for the learned classifiers predicts action unit intensity. Frame-by-frame intensity measurements will enable investigations into facial expression dynamics which were previously intractable by human coding.

Fully Automatic Facial Action Recognition in Spontaneous Behavior

We present results on a user independent fully automatic system for real time recognition of facial actions from the facial action coding system (FACS). The system automatically detects frontal faces in the video stream and codes each frame with respect to 20 action units. We present preliminary results on a task of facial action detection in spontaneous expressions during discourse. Support vector machines and AdaBoost classifiers are compared. For both classifiers, the output margin predicts action unit intensity

Machine learning methods for fully automatic recognition of facial expressions and facial actions

We present a systematic comparison of machine learning methods applied to the problem of fully automatic recognition of facial expressions. We explored recognition of facial actions from the facial action coding system (FACS), as well as recognition of fall facial expressions. Each video-frame is first scanned in real-time to detect approximately upright frontal faces. The faces found are scaled into image patches of equal size, convolved with a bank of Gabor energy filters, and then passed to a recognition engine that codes facial expressions into 7 dimensions in real time: neutral, anger, disgust, fear, joy, sadness, surprise. We report results on a series of experiments comparing recognition engines, including AdaBoost, support vector machines, linear discriminant analysis, as well as feature selection techniques. Best results were obtained by selecting a subset of Gabor filters using AdaBoost and then training support vector machines on the outputs of the filters selected by AdaBoost. The generalization performance to new subjects for recognition of full facial expressions in a 7-way forced choice was 93% correct, the best performance reported so far on the Cohn-Kanade FACS-coded expression dataset. We also applied the system to fully automated facial action coding. The present system classifies 18 action units, whether they occur singly or in combination with other actions, with a mean agreement rate of 94.5% with human FACS codes in the Cohn-Kanade dataset. The outputs of the classifiers change smoothly as a function of time and thus can be used to measure facial expression dynamics.

Global modeling of the Rössler system from the z-variable

Obtaining a global model from the z-variable of the Rossler system is considered to be difficult because of its spiky structure. In this Letter, a 3D global model from the z-variable is derived in a space spanned by the state variable of the time-series itself and generic functions of the other two state variables. We term this space the Ansatz Space. The procedure consists of two steps. First, models built in the derivative coordinates are obtained. Second, we use the analytical form of the map ϕ between systems in the original state space and in the differential space to find a class of models in the Ansatz Space. We find eight models in this class which we show to be dynamically equivalent to the original Rossler system. The important attribute of this approach is that we do not need to use any prior knowledge of the dynamical system other than the measured time series data in order to obtain global models from a single time series.  2003 Elsevier B.V. All rights reserved.

Finger tapping movements of Parkinson's disease patients automatically rated using nonlinear delay differential equations

Parkinsons disease is a degenerative condition whose severity is assessed by clinical observations of motor behaviors. These are performed by a neurological specialist through subjective ratings of a variety of movements including 10-s bouts of repetitive finger-tapping movements. We present here an algorithmic rating of these movements which may be beneficial for uniformly assessing the progression of the disease. Finger-tapping movements were digitally recorded from Parkinsons patients and controls, obtaining one time series for every 10 s bout. A nonlinear delay differential equation, whose structure was selected using a genetic algorithm, was fitted to each time series and its coefficients were used as a six-dimensional numerical descriptor. The algorithm was applied to time-series from two different groups of Parkinsons patients and controls. The algorithmic scores compared favorably with the unified Parkinsons disease rating scale scores, at least when the latter adequately matched with ratings from the Hoehn and Yahr scale. Moreover, when the two sets of mean scores for all patients are compared, there is a strong (r = 0.785) and significant (p<0.0015) correlation between them.

Electrocardiogram classification using delay differential equations.

Time series analysis with nonlinear delay differential equations (DDEs) reveals nonlinear as well as spectral properties of the underlying dynamical system. Here, global DDE models were used to analyze 5 min data segments of electrocardiographic (ECG) recordings in order to capture distinguishing features for different heart conditions such as normal heart beat, congestive heart failure, and atrial fibrillation. The number of terms and delays in the model as well as the order of nonlinearity of the model have to be selected that are the most discriminative. The DDE model form that best separates the three classes of data was chosen by exhaustive search up to third order polynomials. Such an approach can provide deep insight into the nature of the data since linear terms of a DDE correspond to the main time-scales in the signal and the nonlinear terms in the DDE are related to nonlinear couplings between the harmonic signal parts. The DDEs were able to detect atrial fibrillation with an accuracy of 72%, congestive heart failure with an accuracy of 88%, and normal heart beat with an accuracy of 97% from 5 min of ECG, a much shorter time interval than required to achieve comparable performance with other methods.

Non-linear dynamical analysis of EEG time series distinguishes patients with Parkinson's disease from healthy individuals

The pathophysiology of Parkinson’s disease (PD) is known to involve altered patterns of neuronal firing and synchronization in cortical-basal ganglia circuits. One window into the nature of the aberrant temporal dynamics in the cerebral cortex of PD patients can come from analysis of the patients electroencephalography (EEG). Rather than using spectral-based methods, we used data models based on delay differential equations (DDE) as non-linear time-domain classification tools to analyze EEG recordings from PD patients on and off dopaminergic therapy and healthy individuals. Two sets of 50 1-s segments of 64-channel EEG activity were recorded from nine PD patients on and off medication and nine age-matched controls. The 64 EEG channels were grouped into 10 clusters covering frontal, central, parietal, and occipital brain regions for analysis. DDE models were fitted to individual trials, and model coefficients and error were used as features for classification. The best models were selected using repeated random sub-sampling validation and classification performance was measured using the area under the ROC curve A′. In a companion paper, we show that DDEs can uncover hidden dynamical structure from short segments of simulated time series of known dynamical systems in high noise regimes. Using the same method for finding the best models, we found here that even short segments of EEG data in PD patients and controls contained dynamical structure, and moreover, that PD patients exhibited a greater dynamic range than controls. DDE model output on the means from one set of 50 trials provided nearly complete separation of PD patients off medication from controls: across brain regions, the area under the receiver-operating characteristic curves, A′, varied from 0.95 to 1.0. For distinguishing PD patients on vs. off medication, classification performance A′ ranged from 0.86 to 1.0 across brain regions. Moreover, the generalizability of the model to the second set of 50 trials was excellent, with A′ ranging from 0.81 to 0.94 across brain regions for controls vs. PD off medication, and from 0.62 to 0.82 for PD on medication vs. off. Finally, model features significantly predicted individual patients’ motor severity, as assessed with standard clinical rating scales.

Delay differential analysis of time series

Nonlinear dynamical system analysis based on embedding theory has been used for modeling and prediction, but it also has applications to signal detection and classification of time series. An embedding creates a multidimensional geometrical object from a single time series. Traditionally either delay or derivative embeddings have been used. The delay embedding is composed of delayed versions of the signal, and the derivative embedding is composed of successive derivatives of the signal. The delay embedding has been extended to nonuniform embeddings to take multiple timescales into account. Both embeddings provide information on the underlying dynamical system without having direct access to all the system variables. Delay differential analysis is based on functional embeddings, a combination of the derivative embedding with nonuniform delay embeddings. Small delay differential equation (DDE) models that best represent relevant dynamic features of time series data are selected from a pool of candidate models for detection or classification. We show that the properties of DDEs support spectral analysis in the time domain where nonlinear correlation functions are used to detect frequencies, frequency and phase couplings, and bispectra. These can be efficiently computed with short time windows and are robust to noise. For frequency analysis, this framework is a multivariate extension of discrete Fourier transform (DFT), and for higher-order spectra, it is a linear and multivariate alternative to multidimensional fast Fourier transform of multidimensional correlations. This method can be applied to short or sparse time series and can be extended to cross-trial and cross-channel spectra if multiple short data segments of the same experiment are available. Together, this time-domain toolbox provides higher temporal resolution, increased frequency and phase coupling information, and it allows an easy and straightforward implementation of higher-order spectra across time compared with frequency-based methods such as the DFT and cross-spectral analysis.

A class of Lorenz-like systems

The transformation of a three-dimensional dynamical system to its differential model can be used to identify different nonlinear dynamical systems that share the same time series of one of its variables. This transformation then can be used to find classes of nonlinear dynamical systems with similar dynamical behavior.