Pascal Wiggers

Voice Stress Analysis

The nonverbal content of speech carries information about the physiological and psychological condition of the speaker. Psychological stress is a pathological element of this condition, of which the cause is accepted to be “workload”. Objective, quantifiable correlates of stress are searched for by means of measuring the acoustic modifications of the voice brought about by workload. Different voice features from the speech signal to be influenced by stress are: loudness, fundamental frequency, jitter, zero-crossing rate, speech rate and high-energy frequency ratio. To examine the effect of workload on speech production an experiment was designed. 108 native speakers of Dutch were recruited to participate in a stress test (Stroop test). The experiment and the analysis of the test results will be reported in this paper.

Kinect sensing of shopping related actions

Surveillance systems in shopping malls or supermarkets are usually used for detecting abnormal behavior. We used the distributed video cameras system to design digital shopping assistants which assess the behavior of customers while shopping, detect when they need assistance, and offer their support in case there is a selling opportunity. In this paper we propose a system for analyzing human behavior patterns related to products interaction, such as browse through a set of products, examine, pick products, try on, interact with the shopping cart, and look for support by waiving one hand. We used the Kinect sensor to detect the silhouettes of people and extracted discriminative features for basic action detection. Next we analyzed different classification methods, statistical and also spatio-temporal ones, which capture relations between frames, features, and basic actions. By employing feature level fusion of appearance and movement information we obtained an accuracy of 80% for the mentioned six basic actions.

Emotion recognition from speech by combining databases and fusion of classifiers

We explore possibilities for enhancing the generality, portability and robustness of emotion recognition systems by combining data-bases and by fusion of classifiers. In a first experiment, we investigate the performance of an emotion detection system tested on a certain database given that it is trained on speech from either the same database, a different database or a mix of both. We observe that generally there is a drop in performance when the test database does not match the training material, but there are a few exceptions. Furthermore, the performance drops when a mixed corpus of acted databases is used for training and testing is carried out on real-life recordings. In a second experiment we investigate the effect of training multiple emotion detectors, and fusing these into a single detection system. We observe a drop in the Equal Error Rate (EER) from 19.0% on average for 4 individual detectors to 4.2% when fused using FoCal [1].

Comparison between different feature extraction techniques for audio-visual speech recognition

The current audio-only speech recognition still lacks the expected robustness when the Signal to Noise Ratio (SNR) decreases. The video information is not affected by noise which makes it an ideal candidate for data fusion for speech recognition benefit. In the paper [1] the authors have shown that most of the techniques used for extraction of static visual features result in equivalent features or at least the most informative features exhibit this property. We argue that one of the main problems of existing methods is that the resulting features contain no information about the motion of the speaker’s lips. Therefore, in this paper we will analyze the importance of motion detection for speech recognition. For this we will first present the Lip Geometry Estimation(LGE) method for static feature extraction. This method combines an appearance based approach with a statistical based approach for extracting the shape of the mouth. The method was introduced in [2] and explored in detail in [3]. Further more, we introduce a second method based on a novel approach that captures the relevant motion information with respect to speech recognition by performing optical flow analysis on the contour of the speaker’s mouth. For completion, a middle way approach is also analyzed. This third method considers recovering the motion information by computing the first derivatives of the static visual features. All methods were tested and compared on a continuous speech recognizer for Dutch. The evaluation of these methods is done under different noise conditions. We show that the audio-video recognition based on the true motion features, namely obtained by performing optical flow analysis, outperforms the other settings in low SNR conditions.

Conversations with a virtual human: Synthetic emotions and human responses

To test whether synthetic emotions expressed by a virtual human elicit positive or negative emotions in a human conversation partner and affect satisfaction towards the conversation, an experiment was conducted where the emotions of a virtual human were manipulated during both the listening and speaking phase of the dialogue. Twenty-four participants were recruited and were asked to have a real conversation with the virtual human on six different topics. For each topic the virtual human’s emotions in the listening and speaking phase were different, including positive, neutral and negative emotions. The results support our hypotheses that (1) negative compared to positive synthetic emotions expressed by a virtual human can elicit a more negative emotional state in a human conversation partner, (2) synthetic emotions expressed in the speaking phase have more impact on a human conversation partner than emotions expressed in the listening phase, (3) humans with less speaking confidence also experience a conversation with a virtual human as less positive, and (4) random positive or negative emotions of a virtual human have a negative effect on the satisfaction with the conversation. These findings have practical implications for the treatment of social anxiety as they allow therapists to control the anxiety evoking stimuli, i.e., the expressed emotion of a virtual human in a virtual reality exposure environment of a simulated conversation. In addition, these findings may be useful to other virtual applications that include conversations with a virtual human.

Social acceptance of negotiation support systems: scenario-based exploration with focus groups and online survey

We investigate people’s attitudes toward the possible use of negotiation support systems (NSS) in different social contexts and the consequences for their design. To explore functional requirements and social acceptance in different use contexts, we followed a three-step approach. In the first step, we conducted a number of focus groups with negotiation experts. Second, we conducted focus groups with potential users. The focus groups were a qualitative exploration of people’s ideas about NSS that led to design guidelines for mobile NSS. Third, we conducted an online survey (a) to find out in which situations people consider a mobile NSS socially acceptable, (b) to find the factors and relationships that influence this acceptance in the different situations and social contexts, and (c) to investigate the consequences of people’s attitudes toward NSS for the system’s design. The data showed that subjective norm is an important factor influencing the intention to use the system and that the acceptance of NSS depends on the use context. Therefore, we argue that NSS should be designed not only merely as tools being used in the actual negotiation but also as social devices harnessing social networks to provide support in all negotiation phases.

Analysis of shopping behavior based on surveillance system

Closed Circuit Television systems in shopping malls could be used to monitor the shopping behavior of people. From the tracked path, features can be extracted such as the relation with the shopping area, the orientation of the head, speed of walking and direction, pauses which are supposed to be related to the interest of the shopper. Once the interest has been detected the next step is to assess the shoppers positive or negative appreciation to the focused products by analyzing the (non-verbal) behavior of the shopper. Ultimately the system goal is to assess the opportunities for selling, by detecting if a customer needs support. In this paper we present our methodology towards developing such a system consisting of participating observation, designing shopping behavioral models, assessing the associated features and analyzing the underlying technology. In order to validate our observations we made recordings in our shop lab. Next we describe the used tracking technology and the results from experiments.

The effect of priming pictures and videos on a question--answer dialog scenario in a virtual environment

Having a free-speech conversation with avatars in a virtual environment can be desirable in virtual reality applications, such as virtual therapy and serious games. However, recognizing and processing free speech seems too ambitious to realize with the current technology. As an alternative, pre-scripted conversations with keyword detection can handle a number of goal-oriented situations, as well as some scenarios in which the conversation content is of secondary importance. This is, for example, the case in virtual exposure therapy for the treatment of people with social phobia, where conversation is for exposure and anxiety arousal only. A drawback of pre-scripted dialog is the limited scope of the users answers. The system cannot handle a users response that does not match the pre-defined content, other than by providing a default reply. A new method, which uses priming material to restrict the possibility of the users response, is proposed in this paper to solve this problem. Two studies were conducted to investigate whether people can be guided to mention specific keywords with video and/or picture primings. Study 1 was a two-by-two experiment in which participants (n 20) were asked to answer a number of open questions. Prior to the session, participants watched priming videos or unrelated videos. During the session, they could see priming pictures or unrelated pictures on a whiteboard behind the person who asked the questions. The results showed that participants tended to mention more keywords both with priming videos and pictures. Study 2 shared the same experimental setting but was carried out in virtual reality instead of in the real world. Participants (n 20) were asked to answer questions of an avatar when they were exposed to priming material, before and/or during the conversation session. The same results were found: the surrounding media content had a guidance effect. Furthermore, when priming pictures appeared in the environment, people sometimes forgot to mention the content they typically would mention.

Automatic stress detection in emergency (telephone) calls

The abundance of calls to emergency lines during crises is difficult to handle by the limited number of operators. Detecting if the caller is experiencing some extreme emotions can be a solution for distinguishing the more urgent calls. Apart from these, there are several other applications that can benefit from awareness of the emotional state of the speaker. This paper describes the design of a system for selecting the calls that appear to be urgent, based on emotion detection. The system is trained using a database of spontaneous emotional speech from a call-centre. Four machine learning techniques are applied, based on either prosodic or spectral features, resulting in individual detectors. As a last stage, we investigate the effect of fusing these detectors into a single detection system. We observe an improvement in the equal error rate (EER) from 19.0% on average for four individual detectors to 4.2% when fused using linear logistic regression. All experiments are performed in a speaker independent cross-validation framework.

An Approach to Navigation for the Humanoid Robot Nao in Domestic Environments

Humanoid robot navigation in domestic environments remains a challenging task. In this paper, we present an approach for navigating such environments for the humanoid robot Nao. We assume that a map of the environment is given and focus on the localization task. The approach is based on the use only of odometry and a single camera. The camera is used to correct for the drift of odometry estimates. Additionally, scene-classification is used to obtain information about the robot’s position when it gets close to the destination. The approach is tested in an office environment to demonstrate that it can be reliably used for navigation in a domestic environment.