Jan Felix Krebber

Evaluating spoken dialogue systems according to de-facto standards: A case study

In the present paper, we investigate the validity and reliability of de-facto evaluation standards, defined for measuring or predicting the quality of the interaction with spoken dialogue systems. Two experiments have been carried out with a dialogue system for controlling domestic devices. During these experiments, subjective judgments of quality have been collected by two questionnaire methods (ITU-T Rec. P.851 and SASSI), and parameters describing the interaction have been logged and annotated. Both metrics served the derivation of prediction models according to the PARADISE approach. Although the limited database allows only tentative conclusions to be drawn, the results suggest that both questionnaire methods provide valid measurements of a large number of different quality aspects; most of the perceptive dimensions underlying the subjective judgments can also be measured with a high reliability. The extracted parameters mainly describe quality aspects which are directly linked to the system, environmental and task characteristics. Used as an input to prediction models, the parameters provide helpful information for system design and optimization, but not general predictions of system usability and acceptability.

Experiences of Designing a Speech User Interface for a Smart Home Environment

The smart home speech dialog system INSPIRE was built because of several reasons. The two most important ones were to study new speech dialog system building techniques and to establish a new taxonomy for telephone based speech dialog systems. After the project INSPIRE was finished, the system continued as a core system for further research like user modeling approaches. This chapter will explain some fundamental aspects for designing a smart home speech dialog system, and it shows the development process of INSPIRE in the end of each section. The chapter discusses basic requirements, dialog strategies, technical realization, final check, lessons learned and closes with a glance of all points discussed.