Julia Seebode

Reliable Evaluation of Multimodal Dialogue Systems

Usability evaluation is an indispensable issue during the development of new interfaces and interaction paradigms [1]. Although a wide range of reliable usability evaluation methods exists for graphical user interfaces, mature methods are rarely available for speech-based interfaces [2]. When it comes to multimodal interfaces, no standardized approach has so far been established. In previous studies [3], it was shown that usability questionnaires initially developed for unimodal systems may lead to unreliable results when applied to multimodal systems. In the current study, we therefore used several data sources (direct and indirect measurements) to evaluate two unimodal versions and one multimodal version of an information system. We investigated, to which extent the different data showed concordance for the three system versions. The aim was to examine, if, and under which conditions, common and widely used methods originally developed for graphical user interfaces are also appropriate for speech-based and multimodal intelligent interfaces.

Usability Evaluation of Multimodal Interfaces: Is the Whole the Sum of Its Parts?

Usability evaluation of multimodal systems is a complex issue. Multimodal systems provide multiple channels to communicate with the system. Thus, the single modalities as well as their combination have to be taken into account. This paper aims to investigate how ratings of single modalities relate to the ratings of their combination. Therefore a usability evaluation study was conducted testing an information system in two unimodal versions and one multimodal version. Multiple linear regression showed that for overall and global judgments ratings of the single modalities are very good predictors for the ratings of the multimodal system. For separate usability aspects (e.g. hedonic qualities) the prediction was less accurate.

Affective quality of audio feedback in different contexts

To give feedback on mobile devices, sound is commonly used in different ways. Much research has focused on the learnability and user performance with systems that have audio feedback. But so far, there is no standardized method to evaluate the subjective quality of auditory feedback messages. We describe a study to investigate the affective impression of short audio feedback on mobile devices and their functional connotation in three different contexts. Results show that context influences the affective impression of sounds and that there is a relation between ratings according affective quality and functional applicability. We conclude that sounds can be unobtrusive, but still convey their intended meaning in a working context as well as in a leisure time situation without being perceived as disturbing.

Influence of feedback on the quality of multimodal systems

Aim of this research is to investigate the influence of system feedback in different modalities like nonverbal auditory information as well as speech, tactile and graphical feedback on perceived quality. As a first step experiments are conducted for different kinds of unimodal feedback to establish a suitable experimental paradigm. Based on these experiments certain feedback messages are selected and implemented in a multimodal mobile prototype to study the influence of those feedback messages on the interaction and the perceived quality. The developed approach will serve as a generic experimental set-up to collect user ratings and interaction data. These measures can than be used to evaluate the quality and usability of multimodal systems in mobile contexts and for quality prediction.

Assessing the Quality and Usability of Multimodal Systems

To develop a user-friendly system, it is necessary considering quality aspects as early as possible in development process.

Affective Quality and Functional Connotation of Auditory Feedback

This chapter will present a series of studies to investigate the relationship between functional applicability and affective impression of earcons and auditory icons (see Sect. 3.2).

Affective Quality and Functional Connotation of Tactile Feedback

This chapter describes a series of two studies: a Pilot Study to find meaningful vibrotactile messages of which then the affective impression and functional connotation is investigated using a smartphone, as in the previous chapter within varying staged contexts.

Affective Quality and Functional Connotation of Auditory-Tactile Feedback

So far, the investigated audio and tactile feedback messages under examination were tested unimodally. However, the combination of sound and vibration gives the possibility of audio-tactile feedback which could overcome the special disadvantages of each singe feedback modality.

Investigating Different Types of Feedback in a Mobile Interaction Paradigm

So far, the empirical work in this thesis focused on the sole perception of feedback messages without real interaction. As a next step to understand the influence of feedback in a mobile situation, an Interaction Study, which constitutes the concluding empirical part of this work, is described in this chapter.

Experimental Paradigm to Examine the Affective Quality and Functional Connotation of Feedback in Different Contexts

So far, no standardized methods have been developed to study the influence of context on feedback perception. Therefore, one aim of this thesis is to provide an easy to realize and possibly realistic setup to investigate the influence of context aspects.