Panagiotis Tsiamyrtzis

Urban surveillance systems: from the laboratory to the commercial world

Research in the surveillance domain was confined for years in the military domain. Recently, as military spending for this kind of research was reduced and the technology matured, the attention of the research and development community turned to commercial applications of surveillance. In this paper we describe a state-of-the-art monitoring system developed by a corporate R&D lab in cooperation with the corresponding security business units. It represents a sizable effort to transfer some of the best results produced by computer vision research into a viable commercial product. Our description spans both practical and technical issues. From the practical point of view we analyze the state of the commercial security market, typical cultural differences between the research team and the business team and the perspective of the potential users of the technology. These are important issues that have to be dealt with or the surveillance technology will remain in the lab for a long time. From the technical point of view we analyze our algorithmic and implementation choices. We describe the improvements we introduced to the original algorithms reported in the literature in response to some problems that arose during field testing. We also provide extensive experimental results that highlight the strong points and some weaknesses of the prototype system.

Physiology-Based Face Recognition in the Thermal Infrared Spectrum

The current dominant approaches to face recognition rely on facial characteristics that are on or over the skin. Some of these characteristics have low permanency can be altered, and their phenomenology varies significantly with environmental factors (e.g., lighting). Many methodologies have been developed to address these problems to various degrees. However, the current framework of face recognition research has a potential weakness due to its very nature. We present a novel framework for face recognition based on physiological information. The motivation behind this effort is to capitalize on the permanency of innate characteristics that are under the skin. To establish feasibility, we propose a specific methodology to capture facial physiological patterns using the bioheat information contained in thermal imagery. First, the algorithm delineates the human face from the background using the Bayesian framework. Then, it localizes the superficial blood vessel network using image morphology. The extracted vascular network produces contour shapes that are characteristic to each individual. The branching points of the skeletonized vascular network are referred to as thermal minutia points (TMPs) and constitute the feature database. To render the method robust to facial pose variations, we collect for each subject to be stored in the database five different pose images (center, midleft profile, left profile, midright profile, and right profile). During the classification stage, the algorithm first estimates the pose of the test image. Then, it matches the local and global TMP structures extracted from the test image with those of the corresponding pose images in the database. We have conducted experiments on a multipose database of thermal facial images collected in our laboratory, as well as on the time-gap database of the University of Notre Dame. The good experimental results show that the proposed methodology has merit, especially with respect to the problem of low permanence over time. More importantly, the results demonstrate the feasibility of the physiological framework in face recognition and open the way for further methodological and experimental research in the area

Imaging Facial Physiology for the Detection of Deceit

Previous work has demonstrated the correlation of increased blood perfusion in the orbital muscles and stress levels for human beings. It has also been suggested that this periorbital perfusion can be quantified through the processing of thermal video. The idea has been based on the fact that skin temperature is heavily modulated by superficial blood flow. Proof of this concept was established for two different types of stress inducing experiments: startle experiments and mock-crime polygraph interrogations. However, the polygraph interrogation scenarios were simplistic and highly constrained. In the present paper, we report results derived from a large and realistic mock-crime interrogation experiment. The interrogation is free flowing and no restrictions have been placed on the subjects. Additionally, we propose a new methodology to compute the mean periorbital temperature signal. The present approach addresses the deficiencies of the earlier methodology and is capable of coping with the challenges posed by the realistic setting. Specifically, it features a tandem CONDENSATION tracker to register the periorbital area in the context of a moving face. It operates on the raw temperature signal and tries to improve the information content by suppressing the noise level instead of amplifying the signal as a whole. Finally, a pattern recognition method classifies stressful (Deceptive) from non-stressful (Non-Deceptive) subjects based on a comparative measure between the entire interrogation signal (baseline) and a critical subsection of it (transient response). The successful classification rate is 87.2% for 39 subjects. This is on par with the success rate achieved by highly trained psycho-physiological experts and opens the way for automating lie detection in realistic settings.

Imaging Facial Signs of Neurophysiological Responses

In the present paper, we introduce an integrated framework for detecting peripheral sympathetic responses through purely imaging means. The measurements are performed on three facial areas of sympathetic importance, that is, periorbital, supraorbital, and maxillary. To the best of our knowledge, this is the first time that the sympathetic importance of the maxillary area is analyzed. Because the imaging measurements are thermal in nature and are composed of multiple components of variable frequency (i.e., blood flow, sweat gland activation, and breathing), we chose wavelets as the image analysis framework. The measurements also carry substantial noise due to imperfections in tissue tracking and segmentation. The image analysis is grounded on galvanic skin response (GSR) signals, which are still considered the golden standard in peripheral neurophysiological and psychophysiological studies. The experimental results show that monitoring of the facial channels yields similar detecting power to GSRs. However, detailed quantification of the responses, although feasible in GSR through appropriate modeling, is quite difficult in the facial channels for the moment. Further improvements in facial tissue tracking and segmentation are bound to overcome this limitation. This paper opens a new research area that leads to unobtrusive screening technologies in neurophysiology and psychophysiology.

Fast by Nature - How Stress Patterns Define Human Experience and Performance in Dexterous Tasks

In the present study we quantify stress by measuring transient perspiratory responses on the perinasal area through thermal imaging. These responses prove to be sympathetically driven and hence, a likely indicator of stress processes in the brain. Armed with the unobtrusive measurement methodology we developed, we were able to monitor stress responses in the context of surgical training, the quintessence of human dexterity. We show that in dexterous tasking under critical conditions, novices attempt to perform a tasks step equally fast with experienced individuals. We further show that while fast behavior in experienced individuals is afforded by skill, fast behavior in novices is likely instigated by high stress levels, at the expense of accuracy. Humans avoid adjusting speed to skill and rather grow their skill to a predetermined speed level, likely defined by neurophysiological latency.

Touchless monitoring of breathing function

We have developed a novel method for noncontact measurement of breathing function. The method is based on statistical modeling of dynamic thermal data captured through an infrared imaging system. The expired air has higher temperature than the typical background of indoor environments (e.g., walls). Therefore, the particles of the expired air emit at a higher power than the background, a phenomenon which is captured as a distinct thermal signature in the infrared imagery. There is significant technical difficulty in computing this signature, however, because the phenomenon is of very low intensity and transient nature. We use an advanced statistical algorithm based on the method of moments and the Jeffreys divergence measure to address the problem. So far, we were able to compute correctly the breathing waveforms for ten (10) subjects at distances ranging from 6-8 feet. The results were checked against concomitant ground-truth data collected with a traditional contact sensor. The technology is expected to find applications in the next generation of touchless polygraphy and in preventive health care.

Pose-Invariant Physiological Face Recognition in the Thermal Infrared Spectrum

We present a novel framework for face recognition based on physiological information. The motivation behind this effort is to capitalize on the permanency of innate characteristics that are under the skin. To establish feasibility, we propose a specific methodology to capture facial physiological patterns using the bioheat information contained in thermal imagery. First, the algorithm delineates the human face from the background using the Bayesian framework. Then, it localizes the superficial blood vessel network using image morphology. The extracted vascular network produces contour shapes that are characteristic to each individual. The branching points of the skeletonized vascular network are referred to as Thermal Minutia Points (TMP) and constitute the feature database. To render the method robust to facial pose variations we collect for each subject to be stored in the database five (5) different pose images (center, mid-left profile, left profile, mid-right profile, and right profile). During the classification stage, the algorithm first estimates the pose of the test image. Then, it matches the local and global TMP structures extracted from the test image with those of the corresponding pose images in the database. We have conducted experiments on a sizeable database of thermal facial images collected in our lab. The good experimental results show that the proposed methodology has merit. More important, the results demonstrate the feasibility of the physiological framework in face recognition and open the way for further methodological and experimental research in the area.

Coalitional tracking

We propose a novel tracking method that uses a network of independent particle filter trackers whose interactions are modeled using coalitional game theory. Our tracking method is general, it maintains pixel level accuracy, and can negotiate surface deformations and occlusions. We tested our method on a substantial video set featuring non-trivial motion from over 40 objects in both the infrared and visual spectra. The coalitional tracker demonstrated fault tolerant behavior that exceeds by far the performance of single particle filter trackers. Our method represents a shift from the typical tracking paradigms and may find application in demanding imaging problems across the electromagnetic spectrum.

DETER: detection of events for threat evaluation and recognition

Abstract.The current security infrastructure can be summarized as follows: (1) Security systems act locally and do not cooperate in an effective manner, (2) Very valuable assets are protected inadequately by antiquated technology systems and (3) Security systems rely on intensive human concentration to detect and assess threats. In this paper we present DETER (Detection of Events for Threat Evaluation and Recognition), a research and development (R&D) project aimed to develop a high-end automated security system. DETER can be seen as an attempt to bridge the gap between current systems reporting isolated events and an automated cooperating network capable of inferring and reporting threats, a function currently being performed by humans. The prototype DETER system is installed at the parking lot of Honeywell Laboratories (HL) in Minneapolis. The computer vision module of DETER reliably tracks pedestrians and vehicles and reports their annotated trajectories to the threat assessment module for evaluation. DETER features a systematic optical and system design that sets it apart from “toy” surveillance systems. It employs a powerful Normal mixture model at the pixel level supported by an expectation-maximization (EM) initialization, the Jeffreys divergence measure, and the method of moments. It also features a practical and accurate multicamera calibration method. The threat assessment module utilizes the computer vision information and can provide alerts for behaviors as complicated as the “hopping” of potential vehicle thieves from vehicle spot to vehicle spot. Extensive experimental results measured during actual field operations support DETER’s exceptional characteristics. DETER has recently been successfully productized. The product-grade version of DETER monitors movements across the length of a new oil pipeline.

Physiology-based face recognition

We present a novel approach for face recognition based on the physiological information extracted from thermal facial images. First, we delineate the human face from the background using a Bayesian method. Then, we extract the blood vessels present on the segmented facial tissue using image morphology. The extracted vascular network produces contour shapes that are unique for each individual. The branching points of the skeletonized vascular network are referred to as thermal minutia points (TMPs). These are reminiscent of the minutia points produced in fingerprint recognition techniques. During the classification stage, local and global structures of TMPs extracted from test images are matched with those of database images. We have conducted experiments on a large database of thermal facial images collected in our lab. The good experimental results show that our proposed approach has merit and promise.