Abe Kazemzadeh

Analysis of emotion recognition using facial expressions, speech and multimodal information

The interaction between human beings and computers will be more natural if computers are able to perceive and respond to human non-verbal communication such as emotions. Although several approaches have been proposed to recognize human emotions based on facial expressions or speech, relatively limited work has been done to fuse these two, and other, modalities to improve the accuracy and robustness of the emotion recognition system. This paper analyzes the strengths and the limitations of systems based only on facial expressions or acoustic information. It also discusses two approaches used to fuse these two modalities: decision level and feature level integration. Using a database recorded from an actress, four emotions were classified: sadness, anger, happiness, and neutral state. By the use of markers on her face, detailed facial motions were captured with motion capture, in conjunction with simultaneous speech recordings. The results reveal that the system based on facial expression gave better performance than the system based on just acoustic information for the emotions considered. Results also show the complementarily of the two modalities and that when these two modalities are fused, the performance and the robustness of the emotion recognition system improve measurably.

Interpreting ambiguous emotional expressions

Emotion expression is a complex process involving dependencies based on time, speaker, context, mood, personality, and culture. Emotion classification algorithms designed for real-world application must be able to interpret the emotional content of an utterance or dialog given the modulations resulting from these and other dependencies. Algorithmic development often rests on the assumption that the input emotions are uniformly recognized by a pool of evaluators. However, this style of consistent prototypical emotion expression often does not exist outside of a laboratory environment. This paper presents methods for interpreting the emotional content of non-prototypical utterances. These methods include modeling across multiple time-scales and modeling interaction dynamics between interlocutors. This paper recommends classifying emotions based on emotional profiles, or soft-labels, of emotion expression rather than relying on just raw acoustic features or categorical hard labels. Emotion expression is both interactive and dynamic. Consequently, to accurately recognize emotional content, these aspects must be incorporated during algorithmic design to improve classification performance.

A System for Technology Based Assessment of Language and Literacy in Young Children: the Role of Multiple Information Sources

This paper describes the design and realization of an automatic system for assessing and evaluating the language and literacy skills of young children. This system was developed in the context of the TBALL (technology based assessment of language and literacy) project and aims at automatically assessing the English literacy skills of both native talkers of American English and Mexican-American children in grades K-2. The automatic assessments were carried out employing appropriate speech recognition and understanding techniques. In this paper, we describe the system focusing on the role of the multiple sources of information at our disposal. We present the content of the assessment system, discuss some issues in creating a child-friendly interface, and how to provide a suitable feedback to the teachers. In addition, we will discuss the different assessment modules and the different algorithms used for speech analysis.

Recognizing Expressions of Commonsense Psychology in English Text

Many applications of natural language processing technologies involve analyzing texts that concern the psychological states and processes of people, including their beliefs, goals, predictions, explanations, and plans. In this paper, we describe our efforts to create a robust, large-scale lexical-semantic resource for the recognition and classification of expressions of commonsense psychology in English Text. We achieve high levels of precision and recall by hand-authoring sets of local grammars for commonsense psychology concepts, and show that this approach can achieve classification performance greater than that obtained by using machine learning techniques. We demonstrate the utility of this resource for large-scale corpus analysis by identifying references to adversarial and competitive goals in political speeches throughout U.S. history.

Fuzzy Logic Models for the Meaning of Emotion Words

This paper presents two models that use interval type-2 fuzzy sets (IT2 FSs) for representing the meaning of words that refer to emotions. In the first model, the meaning of an emotion word is represented by IT2 FSs on valence, activation, and dominance scales. In the second model, the meaning of an emotion word is represented by answers to an open-ended set of questions from the game of Emotion Twenty Questions (EMO20Q). The notion of meaning in the two proposed models is made explicit using the Frege an framework of extensional and intensional components of meaning. Inter- and intra-subject uncertainty is captured by using IT2 FSs learned from interval approach surveys. Similarity and subsethood operators are used for comparing the meaning of pairs of words. For the first model, we apply similarity and subsethood operators for the task of translating one emotional vocabulary, represented as a computing with words (CWW) codebook, to another. This act of translation is shown to be an example of CWW that is extended to use the three scales of valence, activation, and dominance to represent a single variable. We experimentally evaluate the use of the first model for translations and mappings between vocabularies. Accuracy is high when using a small emotion vocabulary as an output, but performance decreases when the output vocabulary is larger. The second model was devised to deal with larger emotion vocabularies, but presents interesting technical challenges in that the set of scales underlying two different emotion words may not be the same. We evaluate the second model by comparing it with results from a single-slider survey. We discuss the theoretical insights that the two models allow and the advantages and disadvantages of each.

Emotion twenty questions: toward a crowd-sourced theory of emotions

This paper introduces a method for developing a socially-constructed theory of emotions that aims to reflect the aggregated judgments of ordinary people about emotion terms. Emotion Twenty Questions (EMO20Q) is a dialog-based game that is similar to the familiar Twenty Questions game except that the object of guessing is the name for an emotion, rather than an arbitrary object. The game is implemented as a dyadic computer chat application using the Extensible Messaging and Presence Protocol (XMPP). We describe the idea of a theory that is socially-constructed by design, or crowd-sourced, as opposed to the de facto social construction of theories by the scientific community. This paper argues that such a subtle change in paradigm is useful when studying natural usage of emotion words, which can mean different things to different people but still contain a shared, socially-defined meaning that can be arrived at through conversational dialogs. The game of EMO20Q provides a framework for demonstrating this shared meaning and, moreover, provides a standardized way for collecting the judgments of ordinary people. The paper offers preliminary results of EMO20Q pilot experiments, showing that such a game is feasible and that it generates a range of questions that can be used to describe emotions.

Acoustic correlates of user response to error in human-computer dialogues

Using tagged data from the DARPA Communicator Project, we investigate acoustic features of user responses to system errors. We measure acoustic parameters such as energy, fundamental frequency, sub-band energy, ratios of voiced, unvoiced and silent regions of speech, fundamental frequency slope, spectral slope, and spectral center of gravity. We investigate different types of user responses to the errors, including frustration and various types of corrections. It is confirmed that the most prominent acoustic parameter for responses to the errors is fundamental frequency maximum and range, while other features are found to be salient for specific reaction types. More interestingly, acoustic characteristics of user responses to the errors are found to be different depending on whether the responses are the initial or continued responses to the errors. Similarly, normal user responses can differ acoustically depending on whether or not they were preceded by responses to error. We also present results on automatic classification of error response types using these features.

Using interval type-2 fuzzy logic to translate emotion words from Spanish to English

In this paper we describe a methodology that uses interval type-2 fuzzy logic to translate words that refer to emotions from Spanish to English. We build on previous research that aimed to map between emotional labels used in different natural language corpora. In this paper we extend our previous work by showing how a similar method can be used to translate between languages. First, we show how this methodology relates to other fuzzy logic emotion research, then we describe it, and finally show it in use and discuss its results.

Automatic pronunciation verification of english letter-names for early literacy assessment of preliterate children

Children need to master reading letter-names and letter-sounds before reading phrases and sentences. Pronunciation assessment of letter-names and letter-sounds read aloud is an important component of preliterate childrens education, and automating this process can have several advantages. The goal of this work was to automatically verify letter-names spoken by kindergarteners and first graders in realistic classroom noise conditions. We applied the same techniques developed in our previous work on automatic letter-sound verification by comparing and optimizing different acoustic models, dictionaries, and decoding grammars. Our final system was unbiased with respect to the childs grade, age, and native language and achieved 93.1% agreement (0.813 kappa agreement) with human evaluators, who agreed among themselves 95.4% of the time (0.891 kappa).

Effects of emotion on different phoneme classes

This study investigates the effects of emotion on different phoneme classes using short‐term spectral features. In the research on emotion in speech, most studies have focused on prosodic features of speech. In this study, based on the hypothesis that different emotions have varying effects on the properties of the different speech sounds, we investigate the usefulness of phoneme‐class level acoustic modeling for automatic emotion classification. Hidden Markov models (HMM) based on short‐term spectral features for five broad phonetic classes are used for this purpose using data obtained from recordings of two actresses. Each speaker produces 211 sentences with four different emotions (neutral, sad, angry, happy). Using the speech material we trained and compared the performances of two sets of HMM classifiers: a generic set of ‘‘emotional speech’’ HMMs (one for each emotion) and a set of broad phonetic‐class based HMMs (vowel, glide, nasal, stop, fricative) for each emotion type considered. Comparison of ...