Benjamin Weiss

A taxonomy of quality of service and Quality of Experience of multimodal human-machine interaction

Quality of Service (QoS) and Quality of Experience (QoE) are not only important for services transmitting multimedia data, but also for services involving multimodal human-machine interaction. In order to guide the assessment and evaluation of such services, we developed a taxonomy of the most relevant QoS and QoE aspects which result from multimodal human-machine interactions. It consists of three layers: (1) The QoS-influencing factors related to the user, the system, and the context of use; (2) the QoS interaction performance aspects describing user and system behavior and performance; and (3) the QoE aspects related to the quality perception and judgment processes taking place inside the user. For each of these layers, we provide metrics which make system evaluation more systematic and comparable.

Modeling Call Quality for Time-Varying Transmission Characteristics Using Simulated Conversational Structures

This study investigates the perception of speech quality over telephone channels with time-varying transmission characteristics for simulated conversational structures. The aim is to establish a relationship between subjective quality associated with short speech samples (5―6 seconds) and quality associated with overall conversations (1― 2 minutes). Two two-part experiments were conducted. In the first part of each experiment, dialog-final ratings within the temporal structure of a telephone conversation were assessed. Varying transmission characteristics were realized with ten different degradation profiles of preprocessed speech samples obtained mainly from real mobile channels to ensure authentic types of degradation. The second part was carried out to obtain separate short-term ratings of the speech samples used in the first part. Experiments 1 and 2 tested different conversation durations (1 and 2 minutes). The results demonstrate that dialog-final ratings vary with respect to the degradation profile, revealing a recency effect and a strong impact of individual bad samples. Two related models which implement these findings are presented. With these models, dialog-final quality ratings can be estimated significantly better than by plain averaging of short sample ratings (about 10% absolute improvement). They also perform better than two algorithms taken from literature. Both models can be applied to the instrumental method described in ITU-T Rec. P.862 [1], resulting in about 13% absolute improvement. They were evaluated with the results of two different experiments, which were performed independently but on the basis of our test procedure. In these experiments similar profiles but a different type of quality degradation, different sample durations, and different speech material were used. The models proved to be valid and reliable for the time span investigated (1―2 minutes) and for the profiles used. One of them is now being recommended by the ETSI STQ mobile group.

Temporal Development of Quality of Experience

Most research on Quality of Experience treats QoE as a static event. As a result, QoE is measured for stimuli of delimited length, and the QoE which is associated with the stimulus is considered to be stable along its duration. However, this rarely happens in reality where usage episodes extend over several seconds and minutes (e.g. a phone call), hours (e.g. a video film), or regularly over periods of weeks or months (when considering QoE of a subscribed service). In this chapter, we will discuss the cognitive processes involved when QoE is integrated over usage episodes, and describe corresponding assessment methods. We will also review models for estimating episodic and multi-episodic QoE from momentary QoE judgments or their predictions.

Chapter 14 – Evaluation of Multimodal Interfaces for Ambient Intelligence

Publisher Summary This chapter reviews the assessment and evaluation principles applied to multimodal interfaces for intelligent environments. On the basis of a new taxonomy of quality aspects, quantitative metrics are identified that address different aspects of user and system performance, quality, usability, and acceptability. Example applications are presented for multimodal interfaces to domestic devices, focusing on information presentation—for example, via an embodied conversational agent. Furthermore, it is shown which methods are already available and which ones are still missing to support an efficient development of ambient intelligence systems that are well accepted by their users. This chapter provides empirical results and illustrates definitions of factors and aspects as well as information about their relationships. On this common ground, comparable evaluation can be performed, its results can be identified and categorized, and metrics for specific purposes (or missing metrics) can be identified. Because current systems cover a wide range of applications and domains, it is anticipated that an open framework will be needed to enable meaningful evaluation for specific contexts.

A Survey on Perceived Speaker Traits: Personality, Likability, Pathology and the First Challenge

The INTERSPEECH 2012 Speaker Trait Challenge aimed at a unified test-bed for perceived speaker traits – the first challenge of this kind: personality in the five OCEAN personality dimensions, likability of speakers, and intelligibility of pathologic speakers. In the present article, we give a brief overview of the state-of-the-art in these three fields of research and describe the three sub-challenges in terms of the challenge conditions, the baseline results provided by the organisers, and a new openSMILE feature set, which has been used for computing the baselines and which has been provided to the participants. Furthermore, we summarise the approaches and the results presented by the participants to show the various techniques that are currently applied to solve these classification tasks.

Evaluating multimodal systems: a comparison of established questionnaires and interaction parameters

This paper describes the analysis of established and new questionnaires concerning their applicability for the assessment of quality aspects of multimodal systems. To this purpose, an experiment with 27 participants interacting with a a smart-home system via a voice interface, a smartphone-based interface and a multimodal interface, was conducted. Interaction parameters were assessed and related to constructs measured with these questionnaires. The results indicate that some of the questionnaires are suitable for evaluating multimodal interfaces. On the basis of correlations with interaction parameters subscales of these questionnaires can be mapped to quality aspects, such as effectiveness and efficiency. Recommendations are given how to meet two important evaluation requirements, namely which questionnaire to use for comparing two or more systems or system versions and how to identify factors or components in a system that have to be improved. This is another step forward to establish evaluation methods for multimodal systems.

Evaluating talking heads for smart home systems

In this paper we report the results of a user study evaluating talking heads in the smart home domain. Three noncommercial talking head components are linked to two freely available speech synthesis systems, resulting in six different combinations. The influence of head and voice components on overall quality is analyzed as well as the correlation between them. Three different ways to assess overall quality are presented. It is shown that these three are consistent in their results. Another important result is that in this design speech and visual quality are independent of each other. Furthermore, a linear combination of both quality aspects models overall quality of talking heads to a good degree.

SmartSenior@home: Acceptance of an integrated ambient assisted living system. Results of a clinical field trial in 35 households

ABSTRACT Aims: The primary objective of the SmartSenior@home study was to examine the acceptance of the SmartSenior system by older adults. Methods: Twenty-eight partners from industry and research, including the health care sector, worked collaboratively to implement services aiming to maximize independence in old age. The prospective cohort study was conducted in Potsdam, Germany, with n = 35 older adults between 55 and 88 years of age in their apartments. All participants underwent extensive pre- and post-study visits with in-home interviews, functional assessments for cognition, fine motor skills, and mobility as well as responding to questionnaires on user acceptance and quality of life. Results: The results indicate moderate-to-high user acceptance for the SmartSenior system. In particular, the services for general assistance and health, such as audio/video communication, blood pressure monitoring, and communication with a health professional, were rated as very attractive. Less used and less accepted services were those promoting social interaction and reminder services. Conclusion: Besides reliable functioning of the SmartSenior system, the availability of a confidant seems to be the most significant acceptance factor. As one conclusion of this trial, it is possible to develop, integrate, and test an infrastructure for ambient assisted living services in real life.

A new dimension-based framework model for the quality of speech communication services

In this paper, we identify quality dimensions which are relevant for speech communication services, such as mobile telephony or Voice-over-IP. These include dimensions perceived when listening to degraded speech, talking against echoes, double-talk capabilities, interacting with delay, conversing over channels with time-varying characteristics, and service-related dimensions experienced during speech connection set-up and maintenance. For each dimension, we review subjective evaluation metrics and instrumental quality prediction models. We group these dimensions in a framework model which is able to diagnostically assess speech communication services, and may be used for monitoring and maintenance.

Quality of Experiencing Multi-Modal Interaction

In this chapter, we discuss the contributions of different modalities to the overall quality of multi-modal interaction. After reviewing some common systematics and findings concerning multi-modality, we present experimental results from several multi-modal scenarios, involving different (human-to-human and human-to-machine) interaction paradigms, different degrees of interactivity, and different (speech, audio, video, touch, gesture) modalities.The results show that the impact of each modality on overall quality in interaction depends heavily onthe scenario and degree of interactivity. Complementary modalities are not considered in this paper, but the models presented allow predicting overall system quality on the basis of individual modality ratings with an appropriate accuracy. These models still have to be validated in order to be used as tools for system developers estimating whether adding modalities will have an impact on the quality experienced by the user.