Hubert Konik

Framework for reliable, real-time facial expression recognition for low resolution images

Automatic recognition of facial expressions is a challenging problem specially for low spatial resolution facial images. It has many potential applications in human-computer interactions, social robots, deceit detection, interactive video and behavior monitoring. In this study we present a novel framework that can recognize facial expressions very efficiently and with high accuracy even for very low resolution facial images. The proposed framework is memory and time efficient as it extracts texture features in a pyramidal fashion only from the perceptual salient regions of the face. We tested the framework on different databases, which includes Cohn-Kanade (CK+) posed facial expression database, spontaneous expressions of MMI facial expression database and FG-NET facial expressions and emotions database (FEED) and obtained very good results. Moreover, our proposed framework exceeds state-of-the-art methods for expression recognition on low resolution images.

A Spatiotemporal Saliency Model for Video Surveillance

A video sequence is more than a sequence of still images. It contains a strong spatial–temporal correlation between the regions of consecutive frames. The most important characteristic of videos is the perceived motion foreground objects across the frames. The motion of foreground objects dramatically changes the importance of the objects in a scene and leads to a different saliency map of the frame representing the scene. This makes the saliency analysis of videos much more complicated than that of still images. In this paper, we investigate saliency in video sequences and propose a novel spatiotemporal saliency model devoted for video surveillance applications. Compared to classical saliency models based on still images, such as Itti’s model, and space–time saliency models, the proposed model is more correlated to visual saliency perception of surveillance videos. Both bottom-up and top-down attention mechanisms are involved in this model. Stationary saliency and motion saliency are, respectively, analyzed. First, a new method for background subtraction and foreground extraction is developed based on content analysis of the scene in the domain of video surveillance. Then, a stationary saliency model is setup based on multiple features computed from the foreground. Every feature is analyzed with a multi-scale Gaussian pyramid, and all the features conspicuity maps are combined using different weights. The stationary model integrates faces as a supplement feature to other low level features such as color, intensity and orientation. Second, a motion saliency map is calculated using the statistics of the motion vectors field. Third, both motion saliency map and stationary saliency map are merged based on center-surround framework defined by an approximated Gaussian function. The video saliency maps computed from our model have been compared to the gaze maps obtained from subjective experiments with SMI eye tracker for surveillance video sequences. The results show strong correlation between the output of the proposed spatiotemporal saliency model and the experimental gaze maps.

Human vision inspired framework for facial expressions recognition

We present a novel human vision inspired framework that can recognize facial expressions very efficiently and accurately. We propose to computationally process small, salient region of the face to extract features as it happens in human vision. To determine which facial region(s) is perceptually salient for a particular expression, we conducted a psycho-visual experimental study with an eye-tracker. A novel feature space conducive for recognition task is proposed, which is created by extracting Pyramid Histogram of Orientation Gradients features only from the salient facial regions. By processing only salient regions, proposed framework achieved two goals: (a) reduction in computational time for feature extraction (b) reduction in feature vector dimensionality. The proposed framework achieved automatic expression recognition accuracy of 95.3% on extended Cohn-Kanade (CK+) facial expression database for six universal facial expressions.

Pain detection through shape and appearance features

In this paper we are proposing a novel computer vision system that can recognize expression of pain in videos by analyzing facial features. Usually pain is reported and recorded manually and thus carry lot of subjectivity. Manual monitoring of pain makes difficult for the medical practitioners to respond quickly in critical situations. Thus, it is desirable to design such a system that can automate this task. With our proposed model pain monitoring can be done in real-time without any human intervention. We propose to extract shape information using pyramid histogram of orientation gradients (PHOG) and appearance information using pyramid local binary pattern (PLBP) in order to get discriminative representation of face. We tested our proposed model on UNBC-McMaster Shoulder Pain Expression Archive Database and recorded results that exceeds state-of-the-art.

Predictive Saliency Maps for Surveillance Videos

When viewing video sequences, the human visual system (HVS) tends to focus on the active objects. These are perceived as the most salient regions in the scene. Additionally, human observers tend to predict the future positions of moving objects in a dynamic scene and to direct their gaze to these positions. In this paper we propose a saliency detection model that accounts for the motion in the sequence and predicts the positions of the salient objects in future frames. This is a novel technique for attention models that we call Predictive Saliency Map (PSM). PSM improves the consistency of the estimated saliency maps for video sequences. PSM uses both static information provided by static saliency maps (SSM) and motion vectors to predict future salient regions in the next frame. In this paper we focus only on surveillance videos therefore, in addition to low-level features such as intensity, color and orientation we consider high-level features such as faces as salient regions that attract naturally viewers attention. Saliency maps computed based on these static features are combined with motion saliency maps to account for saliency created by the activity in the scene. The predicted saliency map is computed using previous saliency maps and motion information. The PSMs are compared with the experimentally obtained gaze maps and saliency maps obtained using approaches from the literature. The experimental results show that our enhanced model yields higher ability to predict eye fixations in surveillance videos.

CAMSHIFT improvement on multi-hue and multi-object tracking

CAMSHIFT (Continuously Adaptive Mean-Shift) has been well accepted as one of the prominent methods in object tracking. CAMSHIFT is good for single hue object tracking and in the condition where objects color is different with backgrounds color. In this paper, we enhance CAMSHIFT so it can be used for multi-object tracking and improve the robustness of CAMSHIFT for multi-hue object tracking especially in the situation where objects colors are similar with backgrounds colors. We propose a more precise object localization by selecting each dominant color object part using a combination of Mean-Shift segmentation and region growing. Hue-distance, saturation and value color histogram are used to describe the object. We also track the dominant color object parts separately and combine them together to improve robustness of the tracking on multi-hue object. For multi-object tracking, we use a separate tracker for each object. Our experiments showed that those methods improved CAMSHIFT robustness significantly and enable CAMSHIFT for multi-object tracking.

Automatic blur detection for meta-data extraction in content-based retrieval context

During the last few years, image by content retrieval is the aim of many studies. A lot of systems were introduced in order to achieve image indexation. One of the most common method is to compute a segmentation and to extract different parameters from regions. However, this segmentation step is based on low level knowledge, without taking into account simple perceptual aspects of images, like the blur. When a photographer decides to focus only on some objects in a scene, he certainly considers very differently these objects from the rest of the scene. It does not represent the same amount of information. The blurry regions may generally be considered as the context and not as the information container by image retrieval tools. Our idea is then to focus the comparison between images by restricting our study only on the non blurry regions, using then these meta data. Our aim is to introduce different features and a machine learning approach in order to reach blur identification in scene images.

Exploring human visual system: Study to aid the development of automatic facial expression recognition framework

This paper focus on understanding human visual system when it decodes or recognizes facial expressions. Results presented can be exploited by the computer vision research community for the development of robust descriptor based on human visual system for facial expressions recognition. We have conducted psycho-visual experimental study to find which facial region is perceptually more attractive or salient for a particular expression. Eye movements of 15 observers were recorded with an eye-tracker in free viewing conditions as they watch a collection of 54 videos selected from Cohn-Kanade facial expression database, showing six universal facial expressions. The results of the study shows that for some facial expressions only one facial region is perceptually more attractive than others. Other cases shows the attractiveness of two to three facial regions. This paper also proposes a novel framework for automatic recognition of expressions which is based on psycho-visual study.

Visual attention: Effects of blur

The detection of salient regions in images is of great interest for a lot of computer vision applications as adaptive content delivery, smart resizing and auto-cropping, content based image retrieval or visually impaired people assistance. In this paper we focus on the effect of blurriness on human visual attention when observers see images with no prior knowledge. We investigate the hypothesis that sharp objects tend to capture attention irrespective of intensity, color or contrast. Eye movements of 17 subjects were recorded with an eye-tracker in free viewing conditions. Observers were asked to watch a collection of 122 color and grayscale images selected according to criteria driven by basic features of visual perception. The results of the experimental study clearly demonstrate the influence of the sharp/blur aspect of an image part on its saliency. These results indicate that blur information might be integrated in models of attention to efficiently improve the extraction of salient regions.

CAMSHIFT improvement on multi-hue object and multi-object tracking

CAMSHIFT has been well accepted as one of the prominent methods in object tracking. CAMSHIFT is good for single hue object tracking and in the condition where objects color is different with backgrounds color. In this paper, we enhance CAMSHIFT so it can be used for multi-object tracking and improve the robustness of CAMSHIFT for multi-hue object tracking especially in the situation where objects colors are similar with backgrounds colors. We propose a more precise object localization by selecting each dominant color object part using a combination of Mean-Shift segmentation and region growing. Hue-distance, saturation and value color histogram are used to describe the object. We also track the dominant color object parts separately and combine them together to improve robustness of the tracking on multi-hue object. For multi-object tracking, we use a separate tracker for each object. Our experiments showed that those methods improved CAMSHIFT robustness significantly and enable CAMSHIFT for multi-object tracking.