Dvijesh Shastri

Imaging Facial Physiology for the Detection of Deceit

Previous work has demonstrated the correlation of increased blood perfusion in the orbital muscles and stress levels for human beings. It has also been suggested that this periorbital perfusion can be quantified through the processing of thermal video. The idea has been based on the fact that skin temperature is heavily modulated by superficial blood flow. Proof of this concept was established for two different types of stress inducing experiments: startle experiments and mock-crime polygraph interrogations. However, the polygraph interrogation scenarios were simplistic and highly constrained. In the present paper, we report results derived from a large and realistic mock-crime interrogation experiment. The interrogation is free flowing and no restrictions have been placed on the subjects. Additionally, we propose a new methodology to compute the mean periorbital temperature signal. The present approach addresses the deficiencies of the earlier methodology and is capable of coping with the challenges posed by the realistic setting. Specifically, it features a tandem CONDENSATION tracker to register the periorbital area in the context of a moving face. It operates on the raw temperature signal and tries to improve the information content by suppressing the noise level instead of amplifying the signal as a whole. Finally, a pattern recognition method classifies stressful (Deceptive) from non-stressful (Non-Deceptive) subjects based on a comparative measure between the entire interrogation signal (baseline) and a critical subsection of it (transient response). The successful classification rate is 87.2% for 39 subjects. This is on par with the success rate achieved by highly trained psycho-physiological experts and opens the way for automating lie detection in realistic settings.

Imaging Facial Signs of Neurophysiological Responses

In the present paper, we introduce an integrated framework for detecting peripheral sympathetic responses through purely imaging means. The measurements are performed on three facial areas of sympathetic importance, that is, periorbital, supraorbital, and maxillary. To the best of our knowledge, this is the first time that the sympathetic importance of the maxillary area is analyzed. Because the imaging measurements are thermal in nature and are composed of multiple components of variable frequency (i.e., blood flow, sweat gland activation, and breathing), we chose wavelets as the image analysis framework. The measurements also carry substantial noise due to imperfections in tissue tracking and segmentation. The image analysis is grounded on galvanic skin response (GSR) signals, which are still considered the golden standard in peripheral neurophysiological and psychophysiological studies. The experimental results show that monitoring of the facial channels yields similar detecting power to GSRs. However, detailed quantification of the responses, although feasible in GSR through appropriate modeling, is quite difficult in the facial channels for the moment. Further improvements in facial tissue tracking and segmentation are bound to overcome this limitation. This paper opens a new research area that leads to unobtrusive screening technologies in neurophysiology and psychophysiology.

Fast by Nature - How Stress Patterns Define Human Experience and Performance in Dexterous Tasks

In the present study we quantify stress by measuring transient perspiratory responses on the perinasal area through thermal imaging. These responses prove to be sympathetically driven and hence, a likely indicator of stress processes in the brain. Armed with the unobtrusive measurement methodology we developed, we were able to monitor stress responses in the context of surgical training, the quintessence of human dexterity. We show that in dexterous tasking under critical conditions, novices attempt to perform a tasks step equally fast with experienced individuals. We further show that while fast behavior in experienced individuals is afforded by skill, fast behavior in novices is likely instigated by high stress levels, at the expense of accuracy. Humans avoid adjusting speed to skill and rather grow their skill to a predetermined speed level, likely defined by neurophysiological latency.

Perinasal Imaging of Physiological Stress and Its Affective Potential

In this paper, we present a novel framework for quantifying physiological stress at a distance via thermal imaging. The method captures stress-induced neurophysiological responses on the perinasal area that manifest as transient perspiration. We have developed two algorithms to extract the perspiratory signals from the thermophysiological imagery. One is based on morphology and is computationally efficient, while the other is based on spatial isotropic wavelets and is flexible; both require the support of a reliable facial tracker. We validated the two algorithms against the clinical standard in a controlled lab experiment where orienting responses were invoked on n=18 subjects via auditory stimuli. Then, we used the validated algorithms to quantify stress of surgeons (n=24) as they were performing suturing drills during inanimate laparoscopic training. This is a field application where the new methodology shines. It allows nonobtrusive monitoring of individuals who are naturally challenged with a task that is localized in space and requires directional attention. Both algorithms associate high stress levels with novice surgeons, while low stress levels are associated with experienced surgeons, raising the possibility for an affective measure (stress) to assist in efficacy determination. It is a clear indication of the methodologys promise and potential.

Automatic thermal monitoring system (ATHEMOS) for deception detection

Previous work has demonstrated the correlation of periorbital perfusion and stress levels in human beings. In this paper, we report results on a large and realistic mock-crime interrogation experiment. The interrogation is free flowing and no restrictions have been placed on the subjects. We propose a new methodology to compute the average periorbital temperature signal. The present approach addresses the deficiencies of the earlier methodology and is capable of coping with the challenges posed by the realistic setting. Specifically, it features a tandem condensation tracker to register the periorbital area in the context of a moving face. It operates on the raw temperature signal and tries to improve the information content by suppressing the noise level instead of amplifying the signal as a whole. Finally, a pattern recognition method classifies stressful (deceptive) from non-stressful (non-deceptive) subjects based on a comparative measure between the interrogation signal (baseline) and portions thereof (transient response).

Automatic Initiation of the Periorbital Signal Extraction in Thermal Imagery

User intervention in the periorbital thermal signal extraction process breaks down automation. This paper proposes a novel way to minimize user intervention. While previous work demonstrated the importance of accurate computation of the periorbital signal, the present method enables its automatic extraction at a reduced processing time. The proposed algorithm capitalizes on detection of involuntary eye blinking in the thermal imagery. The need for automation has emerged because of repetitive processing of the same subjects, aiming to validate improvements in the periorbital tissue tracking or segmentation algorithms. The proposed approach initiates the tracking and segmentation algorithms on the same spatio-temporal location in repetitive runs of the thermal clip. Thus, it does not only automate the process but also eliminates the variability introduced by manual intervention. We have tested the algorithm on thermal video clips of 39 subjects who faced stressful interrogation for a mock crime. The results show that the proposed method has reduced total processing time from a week down to a day.

Periorbital thermal signal extraction and applications

We propose a novel method that localizes the thermal footprint of the facial and ophthalmic arterial-venous complexes in the periorbital area. This footprint is used to extract the mean thermal signal over time (periorbital signal), which is a correlate of the blood supply to the ocular muscle. Previous work demonstrated that the periorbital signal is associated to autonomic responses and it changes significantly upon the onset of instantaneous stress. The present method enables accurate and consistent extraction of this signal. It aims to replace the heuristic segmentation approach that has been used in stress quantification thus far. Applications in computational psychology and particularly in deception detection are the first to benefit from this new technology. We tested the method on thermal videos of 39 subjects who faced stressful interrogation for a mock crime. The results show that the proposed approach has improved the deception classification success rate to 82%, which is 20% higher compared to the previous approach.

Evaluating smartphone-based user interface designs for a 2D psychological questionnaire

This study explored various user interface designs to transition a two dimensional (2D) questionnaire from its paper-and-pencil testing format to the mobile platform. The current administration of the test limits its usage beyond the lab environment. Creating a mobile version would facilitate ubiquitous administration of the test. Yet, the mobile design must be at least as good as its paper-based counterpart in terms of input accuracy and user interaction efforts. We developed four user interface designs, each of which featured a specific interaction approach. These approaches included displaying the 2D space of the questionnaire in its original form (M1), inputting one variable at a time on the 2D space (M2), dissolving the 2D space into two one-dimensional ordinal scales (M3), and orienting the input selections to the diagonal axes (M4). The designs were tested by a total of 34 participants, aged 18 to 52 years. The study results find the first three interaction approaches (M1-M3) effective but the fourth approach inefficient. Furthermore, the results indicate that the two-tap designs (M2 and M3) are equally as good as the one-tap design (M1).

Spatiotemporal reconstruction of the breathing function

Breathing waveform extracted via nasal thermistor is the most common method to study respiratory function in sleep studies. In essence, this is a temporal waveform of mean temperatures in the nostril region that at every time step collapses two-dimensional data into a single point. Hence, spatial heat distribution in the nostrils is lost along with valuable functional and anatomical cues. This article presents the construction and experimental validation of a spatiotemporal profile for the breathing function via thermal imaging of the nostrils. The method models nasal airflow advection by using a front-propagating level set algorithm with optimal parameter selection. It is the first time that the full two-dimensional advantage of thermal imaging is brought to the fore in breathing computation. This new multi-dimensional measure is likely to bring diagnostic value in sleep studies and beyond.

A Method to Monitor Operator Overloading

This paper describes research that aims to quantify stress levels of operators who perform multiple tasks. The proposed method is based on the thermal signature of the face. It measures physiological function from a stand-off distance and therefore, it can unobtrusively monitor a machine operator. The method was tested on 11 participants. The results show that multi-tasking elevates metabolism in the supraorbital area, which is an indirect indication of increased mental load. This local metabolic change alters heat dissipation and thus, it can be measured through thermal imaging. The methodology could serve as a benchmarking tool in scenarios where an operators divided attention may cause harmful outcomes. A classic example is the case of a vehicle driver who talks on the cell phone. This stress measurement method when combined with user performance metrics can delineate optimal operational envelopes.