Rishabh Gupta

Relevance vector classifier decision fusion and EEG graph-theoretic features for automatic affective state characterization

Objective characterization of affective states during music clip watching could lead to disruptive new technologies, such as affective braincomputer interfaces, neuromarketing tools, and affective video tagging systems, to name a few. To date, the majority of existing systems have been developed based on analyzing electroencephalography (EEG) patterns in specific brain regions. With music videos, however, a complex interplay of information transfer exists between various brain regions. In this paper, we propose the use of EEG graph-theoretic analysis to characterize three emotional ratings: valence, arousal, and dominance, as well as the liking subjective rating. For characterization, graph-theoretic features were used to classify emotional states through support vector machine (SVM) and relevance vector machine (RVM) classifiers. Moreover, fusion schemes at feature and decision levels were also used to improve classification performance. In general, our study shows that the EEG graph-theoretic features are better suited for emotion classification than traditionally used EEG features such as, spectral power features (SPF) and asymmetry index (AI) features. The percentage increase in classification performance, represented by F1-scores, obtained using the proposed methodologies relative to the traditionally used SPF and AI features ranged from: Valence (79%), Arousal (38%), Dominance (56%) and Liking (47%). These findings suggest that an EEG graph-theoretical approach along with a robust classifier can better characterize human affective states evoked during music clip watching.

The effects of text-to-speech system quality on emotional states and frontal alpha band power

The tolerance limit for acceptable multimedia quality is changing as more and more high quality services approach the market. Thus, negative emotional reactions towards low quality services may cause user disappointment and are likely to increase churn rate. The current study analyzes how different levels of synthetic speech quality, obtained from different text-to-speech (TTS) systems, affect the emotional response of a user. This is achieved using two methods: subjective, by means of user reports; and neurophysiological by means of electroencephalography (EEG) analysis. More specifically, we analyzed the frontal alpha band power and correlated this with the subjective ratings based on the Self-Assessment Manikin scale. We found an increase in neuronal activity in the left frontal area with decreasing quality and argue that this is due to user disappointment with low quality TTS systems as they become harder to understand.

Affective state characterization based on electroencephalography graph-theoretic features

Affective states are typically characterized using spectral power information obtained from electroencephalography (EEG) data collected over specific brain regions. However, while experiencing a complex emotional audio-video stimuli, brain networks transfer information in a highly interactive manner. To characterize this information, we propose using graph theoretical features. Towards this end, first, we established graph theoretical features as meaningful correlates of affective states through Pearson correlation. Then we compared the classification performance of these features with that of conventional spectral power features where percentage increases in classification performance of 7% and 11% were found in arousal and valence, respectively. Moreover, feature level fusion was explored and resulted in better performance as compared to the feature sets alone thus, highlighting the complementarity of EEG graph based features and spectral powers. Overall it is hoped that this study will enhance affective state evaluation via passive brain computer interfaces, thus leading to a plethora of applications such as user experience perception modelling and affective indexing/tagging of videos, to name a few.

Multimodal Physiological Quality-of-Experience Assessment of Text-to-Speech Systems

With the growing complexity of various text-to-speech systems, it is becoming more important to understand the underlying perceptual and judgement processes that drive user Quality-of-Experience (QoE) perception. Typical QoE assessment techniques, such as listening tests with self-report ratings, are useful but provide limited insight into these underlying processes. Recent advances in neuroimaging and physiological monitoring technologies, however, have opened new doors and allowed us to better understand and measure QoE perception. In this paper, we explore the use of two neuroimaging techniques, namely electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), to better understand neuronal and cerebral haemodynamic changes resultant from synthesized speech of varying quality. Neural correlates of several QoE dimensions were derived and validated on the publicly available PhySyQX database. Fusion of EEG, fNIRS, and fNIRS-derived physiological parameters, combined with conventional features extracted from the synthesized speech signal showed to accurately represent several QoE dimensions, including those related to listener affective states. It is hoped that these findings will help researchers build better instrumental QoE models that incorporate technological, contextual, and human influence factors.

Characterization of human emotions and preferences for text-to-speech systems using multimodal neuroimaging methods

Voice user interface and speech quality are normally assessed using subjective user experience testing methods and/or objective instrumental techniques. However, the recent advances in neurophysiological tools allowed useful human behavioral constructs to be measured in real-time, such as human emotion, perception, preferences and task performance. Electroencephalography (EEG), and functional near-infrared spectroscopy (fNIRS) are well received neuroimaging tools and they are being used in variety of different domains such as health science, neuromarketing, user experience (UX) research and multimedia quality of experience (QoE) discipline. Therefore, this paper describes the impact of natural and text-to-speech (TTS) signals on a users affective state (valence and arousal) and their preferences using neuroimaging tools (EEG and fNIRS) and subjective user study. The EEG results showed that the natural and high quality TTS speech generate “positive valence”, that was inferred from a higher EEG asymmetric activation at frontal head region. fNIRS results showed the increased activation at Orbito-Frontal Cortex (OFC) region during decision making in favor of natural and high quality TTS speech signals. But natural and TTS signals have significantly different arousal levels.

Objective characterization of human behavioural characteristics for QoE assessment: A pilot study on the use of electroencephalography features

Quality of Experience (QoE) is a human-centric paradigm which produces the blue print of human behavioral states such as perception, emotion, cognition and expectation. Recent advances in neurophysiological monitoring tools have facilitated the study of frequency, time and location of neuronal activity to an unprecedented degree, as well as opened doors to a better understanding of human cognition, emotions and overall behavioral systems. These neurophysiological insights may provide more accurate and objective characterization of QoE metrics. This paper seeks to investigate neuronal activity generated by three different quality levels of a speech stimulus using electroencephalography (EEG). To this end, an electroencephalography (EEG) feature was computed based on the coupling between so-called delta and beta EEG frequency bands, which has previously been linked with negative behavioral characteristics (anxiety, frustration, dissatisfaction). The result indicates an increase in delta and beta coupling with a decrease in the speech quality levels. Additionally, neural correlates of a subjective affective scores (arousal and valence) were also computed and shown to be inversely proportional with EEG feature. These preliminary findings corroborate that emotions play a significant role in human quality and QoE perception.

Physiological quality-of-experience assessment of text-to-speech systems

With the growing complexity of various text-to-speech systems, it is becoming more important to understand the underlying perceptual and judgement processes that drive user Quality-of-Experience (QoE) perception. Typical QoE assessment techniques, such as listening tests with self-report ratings, are useful but provide limited insight into these underlying processes. Recent advances in neuroimaging and physiological monitoring technologies, however, have opened new doors and allowed us to better understand and measure QoE perception. In this paper, we explore the use of two neuroimaging techniques, namely electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), to better understand neuronal and cerebral haemodynamic changes resultant from synthesized speech of varying quality. Neural correlates of several QoE dimensions were derived and validated on the publicly available PhySyQX database. Fusion of EEG, fNIRS, and fNIRS-derived physiological parameters, combined with conventional features extracted from the synthesized speech signal showed to accurately represent several QoE dimensions, including those related to listener affective states. It is hoped that these findings will help researchers build better instrumental QoE models that incorporate technological, contextual, and human influence factors.