Patrick Thiam

Ensemble Methods for Continuous Affect Recognition: Multi-modality, Temporality, and Challenges

In this paper we present a multi-modal system based on audio, video and bio-physiological features for continuous recognition of human affect in unconstrained scenarios. We leverage the robustness of ensemble classifiers as base learners and refine the predictions using stochastic gradient descent based optimization on the desired loss function. Furthermore we provide a discussion about pre- and post-processing steps that help to improve the robustness of the regression and subsequently the prediction quality.

Methods for Person-Centered Continuous Pain Intensity Assessment From Bio-Physiological Channels

In this work, we present methods for the personalization of a system for the continuous estimation of pain intensity from bio-physiological channels. We investigate various ways to estimate the similarity of persons and to retrieve the most informative ones using meta-information, personality traits, and machine learning techniques. Given this information, specialized classifiers can be created that are both, more efficient in terms of complexity and training times and also more accurate than classifiers trained on the complete data. To capture the most information in the different bio-physiological channels, we cover a broad spectrum of different feature extraction algorithms. Furthermore, we show that the system is capable of running in real-time and discuss issues that arise when dealing with incremental data processing. In extensive experiments we verify the validity of our approach.

Multimodal Data Fusion for Person-Independent, Continuous Estimation of Pain Intensity

In this work, a method is presented for the continuous estimation of pain intensity based on fusion of bio-physiological and video features. The focus of the paper is to analyse which modalities and feature sets are suited best for the task of recognizing pain levels in a person-independent setting. A large set of features is extracted from the available bio-physiological channels (ECG, EMG and skin conductivity) and the video stream. Experimental validation demonstrates which modalities contribute the most to a robust prediction and the effects when combining them to improve the continuous estimation given unseen persons.

Detection of Emotional Events utilizing Support Vector Methods in an Active Learning HCI Scenario

In recent years the fields of affective computing and emotion recognition have experienced a steady increase in attention and especially the creation and analysis of multi-modal corpora has been the focus of intense research. Plausible annotation of this data, however is an enormous problem. In detail emotion annotation is very time consuming, cumbersome and sensitive with respect to the annotator. Furthermore emotional reactions are often very sparse in HCI scenarios resulting in a large annotation overhead to gather the interesting moments of a recording, which in turn are highly relevant for powerful features, classifiers and fusion architectures. Active learning techniques provide methods to improve the annotation processes since the annotator is asked to only label the relevant instances of a given dataset. In this work an unsupervised one-class Support Vector Machine is used to build a background model of non-emotional sequences on a novel HCI dataset. The human annotator is iteratively asked to label instances that are not well explained by the background model, which in turn renders them candidates for being interesting events such as emotional reactions that diverge from the norm. The outcome of the active learning procedure is a reduced dataset of only 14% the size of the original dataset that contains most of the significant information, in this case more than 75% of the emotional events.

On Annotation and Evaluation of Multi-modal Corpora in Affective Human-Computer Interaction

In this paper, we discuss the topic of affective human-computer interaction from a data driven viewpoint. This comprises the collection of respective databases with emotional contents, feasible annotation procedures and software tools that are able to conduct a suitable labeling process. A further issue that is discussed in this paper is the evaluation of the results that are computed using statistical classifiers. Based on this we propose to use fuzzy memberships in order to model affective user state and endorse respective fuzzy performance measures.

Adaptive confidence learning for the personalization of pain intensity estimation systems

In this work, a method is presented for the continuous estimation of pain intensity based on fusion of bio-physiological and video features. Furthermore, a method is proposed for the adaptation of the system to unknown test persons based on unlabeled data. First, an analysis is presented that shows which modalities and feature sets are suited best for the task of recognizing pain levels in a person-independent setting. For this, a large set of features is extracted from the available bio-physiological channels (ECG, EMG and skin conductivity) and the video stream. We then propose a method to learn the confidence of a regression system using a multi-stage ensemble classifier. Based on the outcome of the classifier, which is realized by a neural network, confident samples are selected by the adaptation procedure. In various experiments, we show that the algorithm is able to detect highly confident samples which can be used to improve the overall performance. We furthermore discuss the current limitations of automatic pain intensity estimation—in light of the presented approach and beyond.

Fusion Mappings for Multimodal Affect Recognition

Affect recognition is an inherently multi-modal task that makes it appealing to investigate classifier combination approaches in real world scenarios. Thus a variety of different independent classifiers can be constructed from basically independent features without having to rely on artificial feature views. In this paper we study a large variety of fusion approaches based on a multitude of features that were extracted from audio, video and physiological signals. For this purpose the RECOLA data collection is used and we show how an ensemble of classifiers can outperform the best individual classifier.

Continuous Multimodal Human Affect Estimation using Echo State Networks

A continuous multimodal human affect recognition for both arousal and valence dimensions in a non-acted spontaneous scenario is investigated in this paper. Different regression models based on Random Forests and Echo State Networks are evaluated and compared in terms of robustness and accuracy. Moreover, an extension of Echo State Networks to a bi-directional model is introduced to improve the regression accuracy. A hybrid method using Random Forests, Echo State Networks and linear regression fusion is developed and applied on the test subset of the AVEC16 challenge. Finally, the label shift and prediction delay is discussed and an annotator specific regression model, as well as fusion architecture, is proposed for future work.

Ensembles of Support Vector Data Description for Active Learning Based Annotation of Affective Corpora

The present work aims primary at developing an approach to detect irregular and spontaneous facial gestures in video sequences. The developed approach should help a system distinguish between neutral facial expressions characterized by the absence of facial gestures and facial events characterized by the presence of observable facial gestures in video sequences. For this purpose, an active learning approach is proposed in order to avoid the task of annotating an entire video sequence before proceeding with the classification. It is well known that the annotation task is hard, expensive and error prone. Each video sequence is segmented into smaller segments that are then to be investigated and annotated based on the absence or presence of facial gestures. The approach consists in first selecting a set of samples classified as uncharacteristic through the majority vote of a committee of support vector data description (SVDD) models generated randomly. The base learner then focuses on the selected outliers and not on the whole annotated corpus. Different query strategies are used to select the most informative samples among the selected outliers. Those samples are then annotated by the user and added to a pool of annotated samples. The latter is subsequently used to train the base learner again before the next iteration can take place. Experiments suggest that the proposed active learning approach performs as well as a system trained on a fully annotated corpus, while dramatically reducing the cost of annotation.

Majority-Class Aware Support Vector Domain Oversampling for Imbalanced Classification Problems

In this work, a method is presented to overcome the difficulties posed by imbalanced classification problems. The proposed algorithm fits a data description to the minority class but in contrast to many other algorithms, awareness of samples of the majority class is used to improve the estimation process. The majority samples are incorporated in the optimization procedure and the resulting domain descriptions are generally superior to those without knowledge about the majority class. Extensive experimental results support the validity of this approach.