Riad I. Hammoud

Visual learning of texture descriptors for facial expression recognition in thermal imagery

Facial expression recognition is an active research area that finds a potential application in human emotion analysis. This work presents an illumination independent approach for facial expression recognition based on long wave infrared imagery. In general, facial expression recognition systems are designed considering the visible spectrum. This makes the recognition process not robust enough to be deployed in poorly illuminated environments. Common approaches to facial expression recognition of static images are designed considering three main parts: (1) region of interest selection, (2) feature extraction, and (3) image classification. Most published articles propose methodologies that solve each of these tasks in a decoupled way. We propose a Visual Learning approach based on evolutionary computation that solves the first two tasks simultaneously using a single evolving process. The first task consists in the selection of a set of suitable regions where the feature extraction is performed. The second task consists in tuning the parameters that defines the extraction of the Gray Level Co-occurrence Matrix used to compute region descriptors, as well as the selection of the best subsets of descriptors. The output of these two tasks is used for classification by a SVM committee. A dataset of thermal images with three different expression classes is used to validate the performance. Experimental results show effective classification when compared to a human observer, as well as a PCA-SVM approach. This paper concludes that: (1) thermal Imagery provides relevant information for FER, and (2) that the developed methodology can be taken as an efficient learning mechanism for different types of pattern recognition problems.

Automatic Feature Localization in Thermal Images for Facial Expression Recognition

We propose an unsupervised Local and Global feature extraction paradigm to approach the problem of facial expression recognition in thermal images. Starting from local, low-level features computed at interest point locations, our approach combines the localization of facial features with the holistic approach. The detailed steps are as follows: First, face localization using bi-modal thresholding is accomplished in order to localize facial features by way of a novel interest point detection and clustering approach. Second, we compute representative Eigenfeatures for feature extraction. Third, facial expression classification is made with a Support Vector Machine Committiee. Finally, the experiments over the IRIS data-set show that automation was achieved with good feature localization and classification performance.

Robust Multi-Pedestrian Tracking in Thermal-Visible Surveillance Videos

In this paper we introduce a system to track pedestrians using a combined input from RGB and thermal cameras. Two major contributions are presented here. First is the novel model of the scene background where each pixel is represented as a multi-modal distribution with the changing number of modalities for both color and thermal input. We demonstrate how to eliminate the influence of shadows with this type of fusion. Second, based on our background model we introduce a pedestrian tracker designed as a particle filter. We further develop a number of informed reversible transformations to sample the model probability space in order to maximize our model posterior probability. The novelty of our tracking approach also comes from a way we formulate observation likelihoods to account for 3D locations of the bodies with respect to the camera and occlusions by other tracked human bodies as well as static objects. The results of tracking on color and thermal sequences demonstrate that our algorithm is robust to illumination noise and performs well in the outdoor environments.

Pedestrian tracking by fusion of thermal-visible surveillance videos

In this paper we introduce a system to track pedestrians using a combined input from RGB and thermal cameras. Two major contributions are presented here. First is the novel probabilistic model of the scene background where each pixel is represented as a multi-modal distribution with the changing number of modalities for both color and thermal input. We demonstrate how to eliminate the influence of shadows with this type of fusion. Second, based on our background model we introduce a pedestrian tracker designed as a particle filter. We further develop a number of informed reversible transformations to sample the model probability space in order to maximize our model posterior probability. The novelty of our tracking approach also comes from a way we formulate observation likelihoods to account for 3D locations of the bodies with respect to the camera and occlusions by other tracked human bodies as well as static objects. The results of tracking on color and thermal sequences demonstrate that our algorithm is robust to illumination noise and performs well in the outdoor environments.

Automatic Association of Chats and Video Tracks for Activity Learning and Recognition in Aerial Video Surveillance

We describe two advanced video analysis techniques, including video-indexed by voice annotations (VIVA) and multi-media indexing and explorer (MINER). VIVA utilizes analyst call-outs (ACOs) in the form of chat messages (voice-to-text) to associate labels with video target tracks, to designate spatial-temporal activity boundaries and to augment video tracking in challenging scenarios. Challenging scenarios include low-resolution sensors, moving targets and target trajectories obscured by natural and man-made clutter. MINER includes: (1) a fusion of graphical track and text data using probabilistic methods; (2) an activity pattern learning framework to support querying an index of activities of interest (AOIs) and targets of interest (TOIs) by movement type and geolocation; and (3) a user interface to support streaming multi-intelligence data processing. We also present an activity pattern learning framework that uses the multi-source associated data as training to index a large archive of full-motion videos (FMV). VIVA and MINER examples are demonstrated for wide aerial/overhead imagery over common data sets affording an improvement in tracking from video data alone, leading to 84% detection with modest misdetection/false alarm results due to the complexity of the scenario. The novel use of ACOs and chat messages in video tracking paves the way for user interaction, correction and preparation of situation awareness reports.

On Bayesian analysis of a finite generalized Dirichlet mixture via a Metropolis-within-Gibbs sampling

In this paper, we present a fully Bayesian approach for generalized Dirichlet mixtures estimation and selection. The estimation of the parameters is based on the Monte Carlo simulation technique of Gibbs sampling mixed with a Metropolis-Hastings step. Also, we obtain a posterior distribution which is conjugate to a generalized Dirichlet likelihood. For the selection of the number of clusters, we used the integrated likelihood. The performance of our Bayesian algorithm is tested and compared with the maximum likelihood approach by the classification of several synthetic and real data sets. The generalized Dirichlet mixture is also applied to the problems of IR eye modeling and introduced as a probabilistic kernel for Support Vector Machines.

Thermal-Visible Video Fusion for Moving Target Tracking and Pedestrian Classification

The paper presents a fusion-tracker and pedestrian classifier for color and thermal cameras. The tracker builds a background model as a multi-modal distribution of colors and temperatures. It is constructed as a particle filter that makes a number of informed reversible transformations to sample the model probability space in order to maximize posterior probability of the scene model. Observation likelihoods of moving objects account their 3D locations with respect to the camera and occlusions by other tracked objects as well as static obstacles. After capturing the coordinates and dimensions of moving objects we apply a pedestrian classifier based on periodic gait analysis. To separate humans from other moving objects, such as cars, we detect, in human gait, a symmetrical double helical pattern, that can then be analyzed using the Frieze Group theory. The results of tracking on color and thermal sequences demonstrate that our algorithm is robust to illumination noise and performs well in the outdoor environments.

An improved likelihood model for eye tracking

While existing eye detection and tracking algorithms can work reasonably well in a controlled environment, they tend to perform poorly under real world imaging conditions where the lighting produces shadows and the persons eyes can be occluded by e.g. glasses or makeup. As a result, pixel clusters associated with the eyes tend to be grouped together with background-features. This problem occurs both for eye detection and eye tracking. Problems that especially plague eye tracking include head movement, eye blinking and light changes, all of which can cause the eyes to suddenly disappear. The usual approach in such cases is to abandon the tracking routine and re-initialize eye detection. Of course this may be a difficult process due to missed data problem. Accordingly, what is needed is an efficient method of reliably tracking a persons eyes between successively produced video image frames, even in situations where the persons head turns, the eyes momentarily close and/or the lighting conditions are variable. The present paper is directed to an efficient and reliable method of tracking a human eye between successively produced infrared interlaced image frames where the lighting conditions are challenging. It proposes a log likelihood-ratio function of foreground and background models in a particle filter-based eye tracking framework. It fuses key information from even, odd infrared fields (dark and bright-pupil) and their corresponding subtractive image into one single observation model. Experimental validations show good performance of the proposed eye tracker in challenging conditions that include moderate head motion and significant local and global lighting changes. The paper presents also an eye detector that relies on physiological infrared eye responses and a modified version of a cascaded classifier.

On Person Authentication by Fusing Visual and Thermal Face Biometrics

Recognition algorithms that use data obtained by imaging faces in the thermal spectrum are promising in achieving invariance to extreme illumination changes that are often present in practice. In this paper we analyze the performance of a recently proposed face recognition algorithm that combines visual and thermal modalities by decision level fusion. We examine (i) the effects of the proposed data preprocessing in each domain, (ii) the contribution to improved recognition of different types of features, (iii) the importance of prescription glasses detection, in the context of both 1-to-N and 1-to-1 matching (recognition vs. verification performance). Finally, we discuss the significance of our results and, in particular, identify a number of limitations of the current state-of-the-art and propose promising directions for future research.

Multi-source Multi-modal Activity Recognition in Aerial Video Surveillance

Recognizing activities in wide aerial/overhead imagery remains a challenging problem due in part to low-resolution video and cluttered scenes with a large number of moving objects. In the context of this research, we deal with two un-synchronized data sources collected in real-world operating scenarios: full-motion videos (FMV) and analyst call-outs (ACO) in the form of chat messages (voice-to-text) made by a human watching the streamed FMV from an aerial platform. We present a multi-source multi-modal activity/event recognition system for surveillance applications, consisting of: (1) detecting and tracking multiple dynamic targets from a moving platform, (2) representing FMV target tracks and chat messages as graphs of attributes, (3) associating FMV tracks and chat messages using a probabilistic graph-based matching approach, and (4) detecting spatial-temporal activity boundaries. We also present an activity pattern learning framework which uses the multi-source associated data as training to index a large archive of FMV videos. Finally, we describe a multi-intelligence user interface for querying an index of activities of interest (AOIs) by movement type and geo-location, and for playing-back a summary of associated text (ACO) and activity video segments of targets-of-interest (TOIs) (in both pixel and geo-coordinates). Such tools help the end-user to quickly search, browse, and prepare mission reports from multi-source data.