Syaheerah Lebai Lutfi

Towards a robust affect recognition

Proposing a fully automatic 3D facial expression recognition framework.Investigating the capability of conformal mapping in the field of expression recognition.Enhancing the probability estimation of SVM for expression recognition.The results outperformed the previous studies significantly. Facial expressions are a powerful tool that communicates a persons emotional state and subsequently his/her intentions. Compared to 2D face images, 3D face images offer more granular cues that are not available in the 2D images. However, one major setback of 3D faces is that they impose a higher dimensionality than 2D faces. In this paper, we attempt to address this problem by proposing a fully automatic 3D facial expression recognition model that tackles the high dimensionality problem in a twofold solution. First, we transform the 3D faces into the 2D plane using conformal mapping. Second, we propose a Differential Evolution (DE) based optimization algorithm to select the optimal facial feature set and the classifier parameters simultaneously. The optimal features are selected from a pool of Speed Up Robust Features (SURF) descriptors of all the prospective facial points. The proposed model yielded an average recognition accuracy of 79% using the Bosphorus database and 79.36% using the BU-3DFE database. In addition, we exploit the facial muscular movements to enhance the probability estimation (PE) of Support Vector Machine (SVM). Joint application of feature selection with the proposed enhanced PE (EPE) yielded an average recognition accuracy of 84% using the Bosphorus database and 85.81% using the BU-3DFE database, which is statistically significantly better (at p < 0.01 and p < 0.001 , respectively) if compared to the individual exploit of the optimal features only.

A satisfaction-based model for affect recognition from conversational features in spoken dialog systems

Detecting user affect automatically during real-time conversation is the main challenge towards our greater aim of infusing social intelligence into a natural-language mixed-initiative High-Fidelity (Hi-Fi) audio control spoken dialog agent. In recent years, studies on affect detection from voice have moved on to using realistic, non-acted data, which is subtler. However, it is more challenging to perceive subtler emotions and this is demonstrated in tasks such as labeling and machine prediction. This paper attempts to address part of this challenge by considering the role of user satisfaction ratings and also conversational/dialog features in discriminating contentment and frustration, two types of emotions that are known to be prevalent within spoken human-computer interaction. However, given the laboratory constraints, users might be positively biased when rating the system, indirectly making the reliability of the satisfaction data questionable. Machine learning experiments were conducted on two datasets, users and annotators, which were then compared in order to assess the reliability of these datasets. Our results indicated that standard classifiers were significantly more successful in discriminating the abovementioned emotions and their intensities (reflected by user satisfaction ratings) from annotator data than from user data. These results corroborated that: first, satisfaction data could be used directly as an alternative target variable to model affect, and that they could be predicted exclusively by dialog features. Second, these were only true when trying to predict the abovementioned emotions using annotators data, suggesting that user bias does exist in a laboratory-led evaluation.

LSESpeak: A spoken language generator for Deaf people

This paper describes the development of LSESpeak, a spoken Spanish generator for Deaf people. This system integrates two main tools: a sign language into speech translation system and an SMS (Short Message Service) into speech translation system. The first tool is made up of three modules: an advanced visual interface (where a deaf person can specify a sequence of signs), a language translator (for generating the sequence of words in Spanish), and finally, an emotional text to speech (TTS) converter to generate spoken Spanish. The visual interface allows a sign sequence to be defined using several utilities. The emotional TTS converter is based on Hidden Semi-Markov Models (HSMMs) permitting voice gender, type of emotion, and emotional strength to be controlled. The second tool is made up of an SMS message editor, a language translator and the same emotional text to speech converter. Both translation tools use a phrase-based translation strategy where translation and target language models are trained from parallel corpora. In the experiments carried out to evaluate the translation performance, the sign language-speech translation system reported a 96.45 BLEU and the SMS-speech system a 44.36 BLEU in a specific domain: the renewal of the Identity Document and Driving License. In the evaluation of the emotional TTS, it is important to highlight the improvement in the naturalness thanks to the morpho-syntactic features, and the high flexibility provided by HSMMs when generating different emotional strengths.

On the dynamic adaptation of language models based on dialogue information

We present an approach to adapt dynamically the language models (LMs) used by a speech recognizer that is part of a spoken dialogue system. We have developed a grammar generation strategy that automatically adapts the LMs using the semantic information that the user provides (represented as dialogue concepts), together with the information regarding the intentions of the speaker (inferred by the dialogue manager, and represented as dialogue goals). We carry out the adaptation as a linear interpolation between a background LM, and one or more of the LMs associated to the dialogue elements (concepts or goals) addressed by the user. The interpolation weights between those models are automatically estimated on each dialogue turn, using measures such as the posterior probabilities of concepts and goals, estimated as part of the inference procedure to determine the actions to be carried out. We propose two approaches to handle the LMs related to concepts and goals. Whereas in the first one we estimate a LM for each one of them, in the second one we apply several clustering strategies to group together those elements that share some common properties, and estimate a LM for each cluster. Our evaluation shows how the system can estimate a dynamic model adapted to each dialogue turn, which helps to significantly improve the performance of the speech recognition, which leads to an improvement in both the language understanding and the dialogue management tasks.

A survey on techniques to handle face recognition challenges: occlusion, single sample per subject and expression

Face recognition is receiving a significant attention due to the need of facing important challenges when developing real applications under unconstrained environments. The three most important challenges are facial occlusion, the problem of dealing with a single sample per subject (SSPS) and facial expression. This paper describes and analyzes various strategies that have been developed recently for overcoming these three major challenges that seriously affect the performance of real face recognition systems. This survey is organized in three parts. In the first part, approaches to tackle the challenge of facial occlusion are classified, illustrated and compared. The second part briefly describes the SSPS problem and the associated solutions. In the third part, facial expression challenge is illustrated. In addition, pros and cons of each technique are stated. Finally, several improvements for future research are suggested, providing a useful perspective for addressing new research in face recognition.

Recognizing faces prone to occlusions and common variations using optimal face subgraphs

An intuitive graph optimization face recognition approach called Harmony Search Oriented-EBGM (HSO-EBGM) inspired by the classical Elastic Bunch Graph Matching (EBGM) graphical model is proposed in this contribution. In the proposed HSO-EBGM, a recent evolutionary approach called harmony search optimization is tailored to automatically determine optimal facial landmarks. A novel notion of face subgraphs have been formulated with the aid of these automated landmarks that maximizes the similarity entailed by the subgraphs. For experimental evaluation, two sets of de facto databases (i.e., AR and Face Recognition Grand Challenge (FRGC) ver2.0) are used to validate and analyze the behavior of the proposed HSO-EBGM in terms of number of subgraphs, varying occlusion sizes, face images under controlled/ideal conditions, realistic partial occlusions, expression variations and varying illumination conditions. For a number of experiments, results justify that the HSO-EBGM shows improved recognition performance when compared to recent state-of-the-art face recognition approaches.

The Impact of Bio-Inspired Approaches Toward the Advancement of Face Recognition

An increased number of bio-inspired face recognition systems have emerged in recent decades owing to their intelligent problem-solving ability, flexibility, scalability, and adaptive nature. Hence, this survey aims to present a detailed overview of bio-inspired approaches pertaining to the advancement of face recognition. Based on a well-classified taxonomy, relevant bio-inspired techniques and their merits and demerits in countering potential problems vital to face recognition are analyzed. A synthesis of various approaches in terms of key governing principles and their associated performance analysis are systematically portrayed. Finally, some intuitive future directions are suggested on how bio-inspired approaches can contribute to the advancement of face biometrics in the years to come.

I-vector analysis for Gait-based Person Identification using smartphone inertial signals

Abstract This paper describes and evaluates an i-vector based approach for Gait-based Person Identification (GPI) using inertial signals from a smartphone. This approach includes two variability compensation strategies (Linear Discrimination Analysis (LDA) and Probabilistic LDA) for dealing with the gait variability due to different recording sessions or different activities carried out by the user. This study uses a public available dataset that includes recordings from 30 users performing three different activities: walking, walking-upstairs and walking-downstairs. The i-vector approach is compared to a Gaussian Mixture Model-Universal Background Model (GMM-UBM) system, providing significant performance improvements when incorporating the PLDA compensation strategy: the best result reports a User Recognition Error Rate (URER) of 17.7%, an Equal Error Rate (EER) of 6.1% and an F1-score of 82.7% with 30 enrolled users. For less than six enrolled users, the URER error decreases to 5%.

Recognition of occluded faces using an enhanced EBGM algorithm

A new approach to recognize occluded faces is presented in this paper to enhance the conventional Elastic Bunch Graph Matching (EBGM) technique. In the conventional EBGM approach, facial landmarks need to be chosen manually in the initial stage and a single graph per face had been modeled. Our proposed approach intuitively fuses a Harmony search based optimization algorithm over the EBGM approach to automatically choose optimal land marks for a given face. Further, instead of using a single graph, we deploy component level sub-graphs and locate optimal landmarks by maximizing the similarity between each of the sub-graphs. Experimental results show that the proposed automatic method achieves an improved recognition rate when compared to the conventional EBGM approach.

Identifying universal facial emotion markers for automatic 3D facial expression recognition

Facial expressions convey human emotions as a simple and effective non-verbal communication method. Motivated by this special characteristic, facial expression recognition rapidly gains attention in social computing fields, especially in Human Computer Interaction (HCI). Identifying the optimal set of facial emotion markers is an important technique that not only reduces the feature vector dimensionality, but also impacts the recognition accuracy. In this paper, we propose a new emotion marker identification algorithm for automatic and person-independent 3D facial expression recognition system. First, we mapped the 3D face images into the 2D plane via conformal geometry to reduce the dimensionality. Then, the identification algorithm is designed to seek the best discriminative markers and the classifier parameters simultaneously by integrating three techniques viz., Differential Evolution (DE), Support Vector Machine (SVM) and Speed Up Robust Feature (SURF). The proposed system yielded an average recognition rate of 79% and outperformed the previous studies using the Bosphorus database.