Leyuan Liu

Real-time skin color detection under rapidly changing illumination conditions

Skin color provides a useful cue for vision based human-computer interaction (HCI). However, rapidly changing illumination conditions under HCI application environment make skin color detection a challenging task, as skin colors in an image highly depend on the illumination under which the image was taken. This paper presents a method for skin color detection under rapidly changing illumination conditions. Skin colors are modeled under the Bayesian decision framework. Face detection is employed to online sample skin colors and a dynamic thresholding technique is used to update the skin color model. When there is no face detected, color correction strategy is employed to convert the colors of the current frame to those as they appear under the same illuminant of the last model updated frame. Skin color detection is then applied on the color corrected image. Face detection is time-consuming and hence should not be applied to every frame in real-time applications on general consumer hardware. To improve efficiency, a novel method is proposed to detect illumination changes, and face detection is used to update the skin color model only if the illumination has changed. Experimental results show that the proposed method can achieve satisfactory performance for skin color detection under rapidly changing illumination conditions in real-time on general consumer hardware.

A hybrid intelligence-aided approach to affect-sensitive e-learning

E-Learning has revolutionized the delivery of learning through the support of rapid advances in Internet technology. Compared with face-to-face traditional classroom education, e-learning lacks interpersonal and emotional interaction between students and teachers. In other words, although a vital factor in learning that influences a human’s ability to solve problems, affect has been largely ignored in existing e-learning systems. In this study, we propose a hybrid intelligence-aided approach to affect-sensitive e-learning. A system has been developed that incorporates affect recognition and intervention to improve the learner’s learning experience and help the learner become better engaged in the learning process. The system recognizes the learner’s affective states using multimodal information via hybrid intelligent approaches, e.g., head pose, eye gaze tracking, facial expression recognition, physiological signal processing and learning progress tracking. The multimodal information gathered is fused based on the proposed affect learning model. The system provides online interventions and adapts the online learning material to the learner’s current learning state based on pedagogical strategies. Experimental results show that interest and confusion are the most frequently occurring states when a learner interacts with a second language learning system and those states are highly related to learning levels (easy versus difficult) and outcomes. Interventions are effective when a learner is disengaged or bored and have been shown to help learners become more engaged in learning.

Robust head pose estimation using Dirichlet-tree distribution enhanced random forests

Head pose estimation (HPE) is important in human-machine interfaces. However, various illumination, occlusion, low image resolution and wide scene make the estimation task difficult. Hence, a Dirichlet-tree distribution enhanced Random Forests approach (D-RF) is proposed in this paper to estimate head pose efficiently and robustly in unconstrained environment. First, positive/negative facial patch is classified to eliminate influence of noise and occlusion. Then, the D-RF is proposed to estimate the head pose in a coarse-to-fine way using more powerful combined texture and geometric features of the classified positive patches. Furthermore, multiple probabilistic models have been learned in the leaves of the D-RF and a composite weighted voting method is introduced to improve the discrimination capability of the approach. Experiments have been done on three standard databases including two public databases and our lab database with head pose spanning from -90? to 90? in vertical and horizontal directions under various conditions, the average accuracy rate reaches 76.2% with 25 classes. The proposed approach has also been evaluated with the low resolution database collected from an overhead camera in a classroom, the average accuracy rate reaches 80.5% with 15 classes. The encouraging results suggest a strong potential for head pose and attention estimation in unconstrained environment.

Temporally aligned pooling representation for video-based person re-identification

This paper proposes an effective Temporally Aligned Pooling Representation (TAPR) for video-based person re-identification. To extract the motion information from a sequence, we propose to track the superpixels of the lowest portions of human. To perform temporal alignment of videos, we propose to select the “best” walking cycle from the noisy motion information according to the intrinsic periodicity property of walking persons, that is fitted sinusoid in our implementation. To describe the video data in the selected walking cycle, we first divide the cycle into several segments according to the sinusoid, and then describe each segment by temporally aligned pooling. Extensive experimental results on the public datasets demonstrate the effectiveness of the proposed method compared with the state-of-the-art approaches.

Saliency Based on Multi-scale Ratio of Dissimilarity

Recently, many vision applications tend to utilize saliency maps derived from input images to guide them to focus on processing salient regions in images. In this paper, we propose a simple and effective method to quantify the saliency for each pixel in images. Specially, we define the saliency for a pixel in a ratio form, where the numerator measures the number of dissimilar pixels in its center-surround and the denominator measures the total number of pixels in its center-surround. The final saliency is obtained by combining these ratios of dissimilarity over multiple scales. For images, the saliency map generated by our method not only has a high quality in resolution also looks more reasonable. Finally, we apply our saliency map to extract the salient regions in images, and compare the performance with some state-of-the-art methods over an established ground-truth which contains 1000 images.

Detecting skin colors under varying illumination

Skin color has been used as an important cue for various human related computer vision applications. However, detecting skin colors under varying illumination is a challenging task, as the appearance of skin in an image highly depends on the illumination under which the image was taken. To this end, a method for detecting skin colors under varying illumination is proposed in this paper. First, spatial illumination variation is identified and the images are segmented into different regions with different illumination. Each illumination region of color images are corrected base on the illuminant estimated by a local edge-based color constancy algorithm. Then, the corrected images are transformed into a color-space, where statistical results on a skin dataset show that the skin color cluster and non-skin color clusters are separated. Finally, the skin colors are modeled under Bayesian decision framework and classified from non-skin colors. The experimental results show that the proposed method is robust to illumination variations.

Metrics for Objective Evaluation of Background Subtraction Algorithms

Although a large number of background subtraction (BS) algorithms have been proposed, relevant objective metrics for evaluating these algorithms are still lacking. In this paper, empirical discrepancy metrics, which quantify the spatial accuracy and temporal stability of estimated masks by taking into account the potential inaccuracy of reference masks, the location of the pixel errors relative to the border of reference masks as well as the type of errors, are presented for evaluating the performance of BS algorithms. To validate the proposed metrics, they are applied to tune the optimal parameters of LBP-based background subtraction algorithm, and the experimental results confirm the efficiency of them.

A low-cost hand gesture human-computer interaction system

This paper presents a low-cost hand gesture human-computer interaction system for remote controlling of TV and Set-Top-Box (STB). The proposed system adds only a little to the hardware cost as just a webcam is used, and can run on mainstream and even low-end TV and STB without any software and hardware upgrading.

Textured image segmentation based on modulation models

We propose an approach for textured image segmentation based on amplitude-modulation frequency-modulation models. An image is modeled as a set of 2-D nonstationary sinusoids with spatially varying amplitudes and spatially varying frequency vectors. First, the demodulation procedure for the models furnishes a high-dimensional output at each pixel. Then, features including texture contrast, scale, and brightness are elaborately selected based on the high-dimensional output and the image itself. Next, a normalization and weighting scheme for feature combination is presented. Finally, simple K-means clustering is utilized for segmentation. The main characteristic of this work provides a feature vector that strengthens useful information and has fewer dimensionalities simultaneously. The proposed approach is compared with the dominant component analysis (DCA)+K-means algorithm and the DCA+ weighted curve evolution algorithm on three different datasets. The experimental results demonstrate that the proposed approach outperforms the others.

Deep peak-neutral difference feature for facial expression recognition

Facial expression recognition (FER) is important in vision-related applications. Deep neural networks demonstrate impressive performance for face recognition; however, it should be noted that this method relies heavily on a great deal of manually labeled training data, which is not available for facial expressions in real-world applications. Hence, we propose a powerful facial feature called deep peak–neutral difference (DPND) for FER. DPND is defined as the difference between two deep representations of the fully expressive (peak) and neutral facial expression frames. The difference tends to emphasize the facial parts that are changed in the transition from the neutral to the expressive face and to eliminate the face identity information retained in the fine-tuned deep neural network for facial expression, the network has been trained on large-scale face recognition dataset. Furthermore, unsupervised clustering and semi-supervised classification methods are presented to automatically acquire the neutral and peak frames from the expression sequence. The proposed facial expression feature achieved encouraging results on public databases, which suggests that it has strong potential to recognize facial expressions in real-world applications.