Klaus-Peter Engelbrecht

A taxonomy of quality of service and Quality of Experience of multimodal human-machine interaction

Quality of Service (QoS) and Quality of Experience (QoE) are not only important for services transmitting multimedia data, but also for services involving multimodal human-machine interaction. In order to guide the assessment and evaluation of such services, we developed a taxonomy of the most relevant QoS and QoE aspects which result from multimodal human-machine interactions. It consists of three layers: (1) The QoS-influencing factors related to the user, the system, and the context of use; (2) the QoS interaction performance aspects describing user and system behavior and performance; and (3) the QoE aspects related to the quality perception and judgment processes taking place inside the user. For each of these layers, we provide metrics which make system evaluation more systematic and comparable.

Modeling User Satisfaction with Hidden Markov Models

Models for predicting judgments about the quality of Spoken Dialog Systems have been used as overall evaluation metric or as optimization functions in adaptive systems. We describe a new approach to such models, using Hidden Markov Models (HMMs). The users opinion is regarded as a continuous process evolving over time. We present the data collection method and results achieved with the HMM model.

Predicting the quality and usability of spoken dialogue services

In this paper, we compare different approaches for predicting the quality and usability of spoken dialogue systems. The respective models provide estimations of user judgments on perceived quality, based on parameters which can be extracted from interaction logs. Different types of input parameters and different modeling algorithms have been compared using three spoken dialogue databases obtained with two different systems. The results show that both linear regression models and classification trees are able to cover around 50% of the variance in the training data, and neural networks even more. When applied to independent test data, in particular to data obtained with different systems and/or user groups, the prediction accuracy decreases significantly. The underlying reasons for the limited predictive power are discussed. It is shown that - although an accurate prediction of individual ratings is not yet possible with such models - they may still be used for taking decisions on component optimization, and are thus helpful tools for the system developer.

All Users Are (Not) Equal - The Influence of User Characteristics on Perceived Quality, Modality Choice and Performance

This study investigated if cognitive skills, mood, attitudes and personality traits influence quality perceptions, modality choice (speech vs. touch), and performance. It was shown that attitudes and mood are related to quality perceptions while performance is linked to personality traits. Modality choice is influenced by attitudes and personality. Cognitive abilities had no effect.

Modeling the behavior of users who are confronted with security mechanisms

In this paper, we describe a new approach to analyze the trade-off between usability and security frequently found in security-related user interfaces. The approach involves the simulation of potential user interaction behavior by a mixed probabilistic and rule-driven state machine. On the basis of the simulations, user behavior in security-relevant situations can be predicted and user interfaces optimizing intended behavior can be designed. The approach is evaluated in an artificial microworld setting which provides good control over the experimental factors guiding the behavior. A comparison of empirical and simulated behavior in this microworld shows that the approach is already able to accurately predict important aspects of user behavior toward security interfaces, but also identifies future work necessary to better cover all relevant aspects guiding this behavior in a real-world setting.

Analysis of a new simulation approach to dialog system evaluation

The evaluation of spoken dialog systems still relies on subjective interaction experiments for quantifying interaction behavior and user-perceived quality. In this paper, we present a simulation approach replacing subjective tests in early system design and evaluation phases. The simulation is based on a model of the system, and a probabilistic model of user behavior. Probabilities for the next user action vary in dependence of system features and user characteristics, as defined by rules. This way, simulations can be conducted before data have been acquired. In order to evaluate the simulation approach, characteristics of simulated interactions are compared to interaction corpora obtained in subjective experiments. As was previously proposed in the literature, we compare interaction parameters for both corpora and calculate recall and precision of user utterances. The results are compared to those from a comparison of real user corpora. While the real corpora are not equal, they are more similar than the simulation is to the real data. However, the simulations can predict differences between system versions and user groups quite well on a relative level. In order to derive further requirements for the model, we conclude with a detailed analysis of utterances missing in the simulated corpus and consider the believability of entire dialogs.

Making it easier for older people to talk to smart homes: the effect of early help prompts

It is well known that help prompts shape how users talk to spoken dialogue systems. This study investigated the effect of help prompt placement on older users’ interaction with a smart home interface. In the dynamic help condition, help was only given in response to system errors; in the inherent help condition, it was also given at the start of each task. Fifteen older and sixteen younger users interacted with a smart home system using two different scenarios. Each scenario consisted of several tasks. The linguistic style users employed to communicate with the system (interaction style) was measured using the ratio of commands to the overall utterance length (keyword ratio) and the percentage of content words in the user’s utterance that could be understood by the system (shared vocabulary). While the timing of help prompts did not affect the interaction style of younger users, it was early task-specific help supported older users in adapting their interaction style to the system’s capabilities. Well-placed help prompts can significantly increase the usability of spoken dialogue systems for older people.

Emotion detection in dialog systems: Applications, strategies and challenges

Emotion plays an important role in human communication and therefore also human machine dialog systems can benefit from affective processing. We present in this paper an overview of our work from the past few years and discuss general considerations, potential applications and experiments that we did with the emotional classification of human machine dialogs. Anger in voice portals as well as problematic dialog situations can be detected to some degree, but the noise in real life data and the issue of unambiguous emotion definition are still challenging. Also, a dialog system reacting emotionally might raise expectations with respect to its intellectual abilities that it can not fulfill.

Chapter 14 – Evaluation of Multimodal Interfaces for Ambient Intelligence

Publisher Summary This chapter reviews the assessment and evaluation principles applied to multimodal interfaces for intelligent environments. On the basis of a new taxonomy of quality aspects, quantitative metrics are identified that address different aspects of user and system performance, quality, usability, and acceptability. Example applications are presented for multimodal interfaces to domestic devices, focusing on information presentation—for example, via an embodied conversational agent. Furthermore, it is shown which methods are already available and which ones are still missing to support an efficient development of ambient intelligence systems that are well accepted by their users. This chapter provides empirical results and illustrates definitions of factors and aspects as well as information about their relationships. On this common ground, comparable evaluation can be performed, its results can be identified and categorized, and metrics for specific purposes (or missing metrics) can be identified. Because current systems cover a wide range of applications and domains, it is anticipated that an open framework will be needed to enable meaningful evaluation for specific contexts.

Automated Usability Evaluation of Model-Based Adaptive User Interfaces for Users with Special and Specific Needs by Simulating User Interaction

Adaptive applications have the potential to help users with special and specific needs. However, evaluating the usability of such adaptive applications tends to become very complex. This chapter presents an integrated concept for the automated usability evaluation of model-based adaptive user interfaces. The approach is supposed to be used complementary to custom usability evaluations at an early stage of development. Interaction of a user is simulated and evaluated by combining a user model with user interface models from a model-based development framework, which is capable of providing different adaptation alternatives based on user attributes and the context of use. The main benefit of the approach is that no additional descriptions of the application’s UI and tasks need to be created for the usability evaluation because they are already available from the development process. As a result, different design alternatives and adaptation variants can be compared under equal usability evaluation criteria. Further, the complexity and costs for applying automated usability evaluation to adaptive user interfaces for users with special and specific needs can be reduced.