Mark G. Frank

Recognizing facial expression: machine learning and application to spontaneous behavior

We present a systematic comparison of machine learning methods applied to the problem of fully automatic recognition of facial expressions. We report results on a series of experiments comparing recognition engines, including AdaBoost, support vector machines, linear discriminant analysis. We also explored feature selection techniques, including the use of AdaBoost for feature selection prior to classification by SVM or LDA. Best results were obtained by selecting a subset of Gabor filters using AdaBoost followed by classification with support vector machines. The system operates in real-time, and obtained 93% correct generalization to novel subjects for a 7-way forced choice on the Cohn-Kanade expression dataset. The outputs of the classifiers change smoothly as a function of time and thus can be used to measure facial expression dynamics. We applied the system to to fully automated recognition of facial actions (FACS). The present system classifies 17 action units, whether they occur singly or in combination with other actions, with a mean accuracy of 94.8%. We present preliminary results for applying this system to spontaneous facial expressions.

A Few Can Catch a Liar

Research suggests that most people cannot tell from demeanor when others are lying. Such poor performance is typical not only of laypeople but also of most professionals concerned with lying. In this study, three professional groups with special interest or skill in deception, two law-enforcement groups and a select group of clinical psychologists, obtained high accuracy in judging videotapes of people who were lying or telling the truth about their opinions. These findings strengthen earlier evidence that some professional lie catchers are highly accurate, and that behavioral clues to lying are detectable in real time. This study also provides the first evidence that some psychologists can achieve high accuracy in catching lies.

Fully Automatic Facial Action Recognition in Spontaneous Behavior

We present results on a user independent fully automatic system for real time recognition of facial actions from the facial action coding system (FACS). The system automatically detects frontal faces in the video stream and codes each frame with respect to 20 action units. We present preliminary results on a task of facial action detection in spontaneous expressions during discourse. Support vector machines and AdaBoost classifiers are compared. For both classifiers, the output margin predicts action unit intensity

To Catch a Liar: Challenges for Research in Lie Detection Training

Can we train people to detect deception? It is the contention of this article that communication scholars should learn how to train law enforcement professionals on how to detect high stake lies, like those faced by police, judges, customs officials, immigration officials, and so forth. It is proposed that in order to know whether we can train or should bother to train people to detect deception, each training study must meet 6 challenges: (1) relevance, (2) high stakes, (3) proper training, (4) proper testing, (5) generalizability across situations, and (6) generalizability over time. Our quantitative review of the literature suggests that training does significantly raise lie detection accuracy rates. Meta-analytic findings indicate a mean effect size of r = .20 across 20 (11 published studies) paired comparisons of lie detection training versus the control group (i.e., those without some type of training). It should be noted that the majority of the studies that attempt to train lie detectors fall short on many of the above challenges. Current research in lie detection training may actually underestimate the ability to train lie detectors due to the stimulus materials employed in most experiments.

Imaging Facial Physiology for the Detection of Deceit

Previous work has demonstrated the correlation of increased blood perfusion in the orbital muscles and stress levels for human beings. It has also been suggested that this periorbital perfusion can be quantified through the processing of thermal video. The idea has been based on the fact that skin temperature is heavily modulated by superficial blood flow. Proof of this concept was established for two different types of stress inducing experiments: startle experiments and mock-crime polygraph interrogations. However, the polygraph interrogation scenarios were simplistic and highly constrained. In the present paper, we report results derived from a large and realistic mock-crime interrogation experiment. The interrogation is free flowing and no restrictions have been placed on the subjects. Additionally, we propose a new methodology to compute the mean periorbital temperature signal. The present approach addresses the deficiencies of the earlier methodology and is capable of coping with the challenges posed by the realistic setting. Specifically, it features a tandem CONDENSATION tracker to register the periorbital area in the context of a moving face. It operates on the raw temperature signal and tries to improve the information content by suppressing the noise level instead of amplifying the signal as a whole. Finally, a pattern recognition method classifies stressful (Deceptive) from non-stressful (Non-Deceptive) subjects based on a comparative measure between the entire interrogation signal (baseline) and a critical subsection of it (transient response). The successful classification rate is 87.2% for 39 subjects. This is on par with the success rate achieved by highly trained psycho-physiological experts and opens the way for automating lie detection in realistic settings.

The Effect of Rapport in Forensic Interviewing

The psychological literature suggests that establishing rapport between interviewer and subject — whether in clinical, experimental or forensic settings — is likely to enhance the quality of the interaction. Yet there are surprisingly few studies that test this assumption. This article reports a study of the effect of rapport on eyewitness recall of a dramatic videotaped event by creating three interviewer-attitude conditions — “rapport”, “neutral” and “abrupt”. Participants were randomly assigned to the three conditions, and recall was elicited by two methods — free narrative and a semi-structured questionnaire. The results indicate participants in the rapport interview recalled more correct information, and the same amount of incorrect information as participants in the other two conditions. However, prompting via the semi-structured questionnaire yielded additional correct as well as incorrect information for the neutral and abrupt conditions. The results are discussed for their relevance to interviews conducted in forensic settings, and to highlight the need for more specific and improved interview training for police and other justice personnel.

Lie detection and language comprehension.

People are usually no better than chance at detecting lies from a liars demeanour, even when clues to deceit are evident from facial expression and tone of voice. We suspected that people who are unable to understand words (aphasics) may be better at spotting liars, so we tested their performance as lie detectors. We found that aphasics were significantly better at detecting lies about emotion than people with no language impairment, suggesting that loss of language skills may be associated with a superior ability to detect the truth.

Enhancing images of facial expressions.

Facial images can be enhanced by application of an algorithm—the caricature algorithm—that systematically manipulates their distinctiveness (Benson & Perrett, 1991c; Brennan, 1985). In this study, we first produced a composite facial image from natural images of the six facial expressions offear, sadness, surprise, happiness, disgust, andanger shown on a number of different individual faces (Ekman & Friesen, 1975). We then caricatured the composite images with respect to a neutral (resting) expression. Experiment 1 showed that rated strength of the target expression was directly related to the degree of enhancement for all the expressions. Experiment 2, which used a free rating procedure, found that, although caricature enhanced the strength of the target expression (more extreme ratings), it did not necessarily enhance its purity, inasmuch as the attributes of nontarget expressions were also enhanced. Naming of prototypes, of original exemplar images, and of caricatures was explored in Experiment 3 and followed the pattern suggested by the free rating conditions of Experiment 2, with no overall naming advantage to caricatures under these conditions. Overall, the experiments suggested that computational methods of compositing and caricature can be usefully applied to facial images of expression. Their utility in enhancing the distinctiveness of the expression depends on the purity of expression in the source image.

Automatic thermal monitoring system (ATHEMOS) for deception detection

Previous work has demonstrated the correlation of periorbital perfusion and stress levels in human beings. In this paper, we report results on a large and realistic mock-crime interrogation experiment. The interrogation is free flowing and no restrictions have been placed on the subjects. We propose a new methodology to compute the average periorbital temperature signal. The present approach addresses the deficiencies of the earlier methodology and is capable of coping with the challenges posed by the realistic setting. Specifically, it features a tandem condensation tracker to register the periorbital area in the context of a moving face. It operates on the raw temperature signal and tries to improve the information content by suppressing the noise level instead of amplifying the signal as a whole. Finally, a pattern recognition method classifies stressful (deceptive) from non-stressful (non-deceptive) subjects based on a comparative measure between the interrogation signal (baseline) and portions thereof (transient response).

Nonverbal Detection of Deception in Forensic Contexts

Publisher Summary This chapter focuses on different types of lies that eye witnesses can use before the jury, while examining the importance of identifying nonverbal detection of deception in forensic contexts. Lies can occur in many different forms, from outright fabrication, denial, distortion, evasion, and concealment, to even telling the truth falsely. Similar to this type of lie is the concealment lie that involves concealing the truth in situations where it is implicit or explicit that the person should not conceal it. If the situation involves assessing the true feelings of a witness, then the lie catcher should try to observe the presence or absence of as many of these reliable clues to the emotion as possible. Low levels of fear may help liars get away with their deceptions by maintaining their alertness. In its moderate and high levels, fear can produce behavioral signs that can be noticed by the skilled lie catcher.