Tomas Pfister

Learning from Simulated and Unsupervised Images through Adversarial Training

With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulators output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a self-regularization term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.

Recognising spontaneous facial micro-expressions

Facial micro-expressions are rapid involuntary facial expressions which reveal suppressed affect. To the best knowledge of the authors, there is no previous work that successfully recognises spontaneous facial micro-expressions. In this paper we show how a temporal interpolation model together with the first comprehensive spontaneous micro-expression corpus enable us to accurately recognise these very short expressions. We designed an induced emotion suppression experiment to collect the new corpus using a high-speed camera. The system is the first to recognise spontaneous facial micro-expressions and achieves very promising results that compare favourably with the human micro-expression detection accuracy.

Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework

In this paper we propose the first method known to the authors that successfully differentiates spontaneous from posed facial expressions using a realistic training corpus. We propose a new spatiotemporal local texture descriptor (CLBP-TOP) that outperforms other descriptors. We demonstrate that our temporal interpolation and visual/near-infrared fusion methods improve the differentiation performance. Finally, we propose a new generic facial expression recognition framework that subdivides the facial expression recognition problem into a cascade of smaller tasks that are simpler to tackle. The system is the first to differentiate spontaneous from posed facial expressions with a realistic corpus and achieves promising results.

Real-Time Recognition of Affective States from Nonverbal Features of Speech and Its Application for Public Speaking Skill Analysis

This paper presents a new classification algorithm for real-time inference of affect from nonverbal features of speech and applies it to assessing public speaking skills. The classifier identifies simultaneously occurring affective states by recognizing correlations between emotions and over 6,000 functional-feature combinations. Pairwise classifiers are constructed for nine classes from the Mind Reading emotion corpus, yielding an average cross-validation accuracy of 89 percent for the pairwise machines and 86 percent for the fused machine. The paper also shows a novel application of the classifier for assessing public speaking skills, achieving an average cross-validation accuracy of 81 percent and a leave-one-speaker-out classification accuracy of 61 percent. Optimizing support vector machine coefficients using grid parameter search is shown to improve the accuracy by up to 25 percent. The emotion classifier outperforms previous research on the same emotion corpus and is successfully applied to analyze public speaking skills.

Speech emotion classification and public speaking skill assessment

This paper presents a new classification algorithm for real-time inference of emotions from the non-verbal features of speech. It identifies simultaneously occurring emotional states by recognising correlations between emotions and features such as pitch, loudness and energy. Pairwise classifiers are constructed for nine classes from the Mind Reading emotion corpus, yielding an average cross-validation accuracy of 89% for the pairwise machines and 86% for the fused machine. The paper also shows a novel application of the classifier for assessing public speaking skills, achieving an average cross-validation accuracy of 81%. Optimisation of support vector machine coefficients is shown to improve the accuracy by up to 25%. The classifier outperforms previous research on the same emotion corpus and achieves real-time performance.

Automatic identification of facial clues to lies

Micro-expressions are very short involuntary facial expressions that reveal emotions people try to hide (see Figure 1). They can be used for lie detection and are used by trained officials at US airports to detect suspicious behavior. For example, a terrorist trying to conceal a plan to commit suicide would very likely show a very short expression of intense anguish. However, human recognition accuracy is very low; even highly trained human detectors are notoriously inaccurate, achieving a recognition accuracy of only 47%.1 This performance makes an automatic computer detector very attractive. We have developed the first system for recognizing real, spontaneous facial micro-expressions. We have developed a temporal interpolation method that enables multiple kernel learning and other machine learning algorithms to classify micro-expressions even with a normal 25 frames per second (fps) camera (see Figure 2 for an illustration of the method). Our system achieves very promising results that compare favorably to human microexpression recognition accuracy. We also have the first publicly available database of micro-expressions.2 The major challenges in recognizing micro-expressions are twofold. The first is involuntariness. How can we get human training data for our algorithm when the expressions are involuntary? We cannot rely on actors as they cannot act out involuntary expressions. The second is short duration. The implication of this is there are a very limited number of frames available using normal cameras, making recognition very challenging. To obtain samples of involuntary expressions, we conducted an induced emotion suppression experiment where subjects were recorded as they attempted to suppress their facial expressions while watching 16 emotion-eliciting film clips. They were told that experimenters would be watching their face and that if their facial expression leaked so that the experimenter correctly guessed which clip they were watching, they would be asked to fill in a dull 500-question survey. This induced Figure 1. The bottom image shows a temporal cross-section during the six-frame-long facial micro-expression depicted in the top image. The cross-section is positioned at a given x-coordinate on the upper lip of the subject.