Caifeng Shan

Local Binary Patterns and Its Application to Facial Image Analysis: A Survey

Local binary pattern (LBP) is a nonparametric descriptor, which efficiently summarizes the local structures of images. In recent years, it has aroused increasing interest in many areas of image processing and computer vision and has shown its effectiveness in a number of applications, in particular for facial image analysis, including tasks as diverse as face detection, face recognition, facial expression analysis, and demographic classification. This paper presents a comprehensive survey of LBP methodology, including several more recent variations. As a typical application of the LBP approach, LBP-based facial image analysis is extensively reviewed, while its successful extensions, which deal with various tasks of facial image analysis, are also highlighted.

Learning local binary patterns for gender classification on real-world face images

Gender recognition is one of fundamental face analysis tasks. Most of the existing studies have focused on face images acquired under controlled conditions. However, real-world applications require gender classification on real-life faces, which is much more challenging due to significant appearance variations in unconstrained scenarios. In this paper, we investigate gender recognition on real-life faces using the recently built database, the Labeled Faces in the Wild (LFW). Local Binary Patterns (LBP) is employed to describe faces, and Adaboost is used to select the discriminative LBP features. We obtain the performance of 94.81% by applying Support Vector Machine (SVM) with the boosted LBP features. The public database used in this study makes future benchmark and evaluation possible.

Learning Discriminative LBP-Histogram Bins for Facial Expression Recognition

Local Binary Patterns (LBP) have been well exploited for facial image analysis recently. In the existing work, the LBP histograms are extracted from local facial regions, and used as a whole for the regional description. However, not all bins in the LBP histogram are necessary to be useful for facial representation. In this paper, we propose to learn discriminative LBP-Histogram (LBPH) bins for the task of facial expression recognition. Our experiments illustrate that the selected LBPH bins provide a compact and discriminative facial representation. We experimentally illustrate that it is necessary to consider multiscale LBP for representing faces, and most discriminative information is contained in uniform patterns. By adopting SVM with the selected multiscale LBPH bins, we obtain the best recognition performance of 93.1% on the Cohn-Kanade database.

Local features based facial expression recognition with face registration errors

In this paper, we extensively investigate local features based facial expression recognition with face registration errors, which has never been addressed before. Our contributions are three fold. Firstly, we propose and experimentally study the histogram of oriented gradients (HOG) descriptors for facial representation. Secondly, we present facial representations based on local binary patterns (LBP) and local ternary patterns (LTP) extracted from overlapping local regions. Thirdly, we quantitatively study the impact of face registration errors on facial expression recognition using different facial representations. Overall LBP with overlapping gives the best performance (92.9% recognition rate on the Cohn-Kanade database), while maintaining a compact feature vector and best robustness against face registration errors.

Kinect sensing of shopping related actions

Surveillance systems in shopping malls or supermarkets are usually used for detecting abnormal behavior. We used the distributed video cameras system to design digital shopping assistants which assess the behavior of customers while shopping, detect when they need assistance, and offer their support in case there is a selling opportunity. In this paper we propose a system for analyzing human behavior patterns related to products interaction, such as browse through a set of products, examine, pick products, try on, interact with the shopping cart, and look for support by waiving one hand. We used the Kinect sensor to detect the silhouettes of people and extracted discriminative features for basic action detection. Next we analyzed different classification methods, statistical and also spatio-temporal ones, which capture relations between frames, features, and basic actions. By employing feature level fusion of appearance and movement information we obtained an accuracy of 80% for the mentioned six basic actions.

Background Subtraction under Sudden Illumination Changes

Robust background subtraction under sudden illuminationchanges is a challenging problem. In this paper, wepropose an approach to address this issue, which combinesthe Eigenbackground algorithm together with a statisticalillumination model. The rst algorithm is used to give arough reconstruction of the input frame, while the secondone improves the foreground segmentation. We introduce anonline spatial likelihood model by detecting reliable backgroundand foreground pixels. Experimental results illustratethat our approach achieves consistently higher accuracycompared to several state-of-the-art algorithms

Recognizing Facial Expressions Automatically from Video

Facial expressions, resulting from movements of the facial muscles, are the face changes in response to a person’s internal emotional states, intentions, or social communications. There is a considerable history associated with the study on facial expressions. Darwin [22] was the first to describe in details the specific facial expressions associated with emotions in animals and humans, who argued that all mammals show emotions reliably in their faces. Since that, facial expression analysis has been a area of great research interest for behavioral scientists [27]. Psychological studies [48, 3] suggest that facial expressions, as the main mode for nonverbal communication, play a vital role in human face-to-face communication. For illustration, we show some examples of facial expressions in Fig. 1.

Analysis of shopping behavior based on surveillance system

Closed Circuit Television systems in shopping malls could be used to monitor the shopping behavior of people. From the tracked path, features can be extracted such as the relation with the shopping area, the orientation of the head, speed of walking and direction, pauses which are supposed to be related to the interest of the shopper. Once the interest has been detected the next step is to assess the shoppers positive or negative appreciation to the focused products by analyzing the (non-verbal) behavior of the shopper. Ultimately the system goal is to assess the opportunities for selling, by detecting if a customer needs support. In this paper we present our methodology towards developing such a system consisting of participating observation, designing shopping behavioral models, assessing the associated features and analyzing the underlying technology. In order to validate our observations we made recordings in our shop lab. Next we describe the used tracking technology and the results from experiments.

Mapping facial expression recognition algorithms on a low-power smart camera

Recent developments in the field of facial expression recognition advocate the use of feature vectors based on local binary patterns (LBP). Research on the algorithmic side addresses robustness issues when dealing with non-ideal illumination conditions. In this paper, we address the challenges related to mapping these algorithms on smart camera platforms. Algorithmic partitioning taking into account the camera architecture is investigated with a primary focus of keeping the power consumption low. Experimental results show that compute-intensive feature extraction tasks can be mapped on a massively-parallel processor with reasonable processor utilization. Although the final feature classification phase could also benefit from parallel processing, mapping on a general purpose sequential processor would suffice.

Detecting customers' buying events on a real-life database

Video Analytics covers a large set of methodologies which aim at automatically extracting information from video material. In the context of retail, the possibility to effortlessly gather statistics on customer shopping behavior is very attractive. In this work, we focus on the task of automatic classification of customer behavior, with the objecting to recognize buying events. The experiments are performed on several hours of video collected in a supermarket. Given the vast effort of the research community on the task of tracking, we assume the existence of a video tracking system capable of producing a trajectory for every individual, and currently manually annotate the input videos with trajectories. From the annotated video recordings, we extract features related to the spatio-temporal behavior of the trajectory, and to the user movement, and analyze the shopping sequences using a Hidden Markov Model (HMM). First results show that it is possible to discriminate between buying and non-buying behavior with an accuracy of 74%.