Lars Bo Larsen

Issues in the evaluation of spoken dialogue systems using objective and subjective measures

The paper presents results and conclusions about the current evaluation methodologies for spoken dialogue systems (SDS). The PARADISE paradigm, used for evaluation in the DARPA Communicator project, is briefly introduced and discussed through the application to the OVID home banking dialogue system. It is shown to provide results consistent with those obtained by the DARPA community, but a number of problems and limitations are pointed out. The issue of user attitude measures obtained through questionnaires is discussed. This is an area that has not received much attention from the speech technology community, but is important in order to obtain valid results and conclusions about usability. A general presentation of the issues that must be addressed when developing and employing questionnaires is given with a focus on how to ensure the reliability and validity of the results. Examples of results obtained from the OVID project are used to illustrate this.

Development of an automatic pool trainer

This paper presents the development of an automatic pool trainer (APT). The rationale behind the user interface design is outlined and the system architecture and modules are described. The APT utilises a camera and a projector for visual in-and output to the system. In addition, the user can issue spoken commands to an interface agent acting as a personification of the system. The camera and projector are placed unobtrusively in the ceiling above the pool table and no modifications of the table or other equipment (cues, pool balls, etc.) are required by the system. The camera is used to detect the positions and movements of the pool balls as well as user hand gestures. The projector will display menus, text and graphics directly on the surface of the table. Optionally, the communication can be carried out by speech via the interface agent. Several user tests have shown high degree of user acceptance and that users quickly become familiar with the system and use it as intended. It is discussed how the concept can be extended to other games (or variants) than pool.

Evaluating the usefulness of mobile services based on captured usage data from longitudinal field trials

It is widely discussed whether the usefulness of mobile systems and services should be evaluated in the field or in the lab. The aim of this paper is to make a case for conducting longitudinal field trials when evaluating the usefulness of mobile services, and to partly base such evaluations on quantitative usage data automatically captured on the device. A framework for automatic capture and analysis of such data is introduced, and it is shown how it has been used to evaluate a mobile service for diabetes management. Results and experiences based on a three month field trial with a single diabetic user is presented and it is demonstrated how these results can be used to make inferences about the use of the service with regard to functionality and usage patterns.

Multi modal user interaction in an automatic pool trainer

This paper presents the human-computer interaction in an automatic pool trainer currently being developed at the Center for PersonKommunikation, Aalborg University. The aim of the system is to automate (parts of) the learning process, in this case of the game of pool. The automated pool trainer (APT) utilises multi modal, agent driven user-system communication, to facilitate the user interaction. To allow the user the necessary freedom of movement when addressing the task, system output is presented on a wall-mounted screen and is augmented by a laser drawing lines and points directly on the pool table surface. User interaction is either carried out via a spoken dialogue with an animated interface agent, or by using a touch screen panel. The paper describes the philosophy on which the system is designed, as well as the system architecture and individual modules. The user interaction is described and the paper concludes with a presentation of some test results and a discussion of the suitability of the presented and similar systems.

The Prevalence of UX Design in Agile Development Processes in Industry

The gap between how the academic world develops usability and user experience (UX) methods, and how the industry employs these methods is perceived as both broad and deep. But is that the real picture -- and has there been a change in how companies work within these fields over the past two years? By conducting interviews with eight companies, this paper tries to answer these two questions. The companies were initially interviewed in 2013 and by follow-up interviews in 2015 the paper draws a picture of how the companies work with UX and usability in an agile development environment. We identify the challenges they are facing and if, and how the work progresses. We found that the UX maturity during these two years had changed significantly. This was revealed by the fact that almost all of the companies in 2015 had implemented or were in the process of developing a UX strategy together with more formalized UX processes. They also allocated more resources to conduct UX and usability work than earlier. We found that all of the companies made use of low-fi prototyping, followed by usability testing, workshops, personas, expert evaluations, user or customer journeys, customer visits and user task analyses. Almost all the companies carried out development using the Scrum framework. All of the companies were interested in the idea of agile UX, and found the idea of using the developers as a UX resource interesting. This, together with an idea of modifying existing usability methods to be used in an agile, industrial setting could be a solution to bridge the gap between academia and the industry.

Evaluating user preferences for video transfer methods from a mobile device to a TV screen

This article reports on the evaluation of four methods for transferring video content from a mobile device to a fixed television. The methods have been investigated in a Wizard-of-Oz approach through two consecutive studies. The first experiment aimed at collecting general opinions toward such a feature as well as preferences in terms of usability. In general, participants preferred methods that resemble already known interaction paradigms, and argued against the need of moving physically. From the comments collected in the first study, six potential indicators of preference were generated and investigated in a second experiment. The results indicate that familiarity, convenience and annoyance are strong predictors of the preference level of methods for transferring video content from a mobile- to a fixed device. A discussion and potential design guidelines are finally proposed.

Voice controlled home banking-objectives and experiences of the ESPRIT OVID project

This paper reports on the ESPRIT OVID project. It describes the objectives behind the project, and the results up till now. The OVID project deals with the task of phone based home banking services. The findings of the requirements capture process are reported in detail. Most notably a number of requirement interviews uncovered that almost identical requirements exist for all the OVID banks. This was the case even though the banks have very different profiles and market strategies. This observation suggests that the conclusions may be extended to cover voice controlled home banking services in general.

Investigating a mixed-initiative dialogue management strategy

This paper discusses how to design mixed-initiative spoken dialogues with only a partial recognition of the user utterances (recognition of concepts or phrase spotting). The objective is to investigate the potential of such a technique and in particular to develop a corresponding dialogue model. The work has been carried out within the ESPRIT OVID project, which addressed a voice controlled home banking task. A number of experiments have been carried out within the project and the paper discusses the results of these. A mixed-initiative dialogue management model has been developed and implemented, and the experiments have shown that users to a very high degree are able to grab the initiative at natural points in the dialogue.

Towards Useful and Usable Services in Personal Networks

A lot of research and development is going into overcoming the technological hurdles of realizing ubiquitous computing in personal networks. For this endeavor to become successful it is essential that useful and usable services are developed to ensure acceptance of the new PN paradigm. This paper discusses the importance of involving end-users in the development process and presents a concrete approach to do this. A case study is made on the development and evaluation of a prototype service for monitoring and managing diabetes. Although it is designed as a PN based service relying on technology still under development, it has been implemented using current networking and communication technologies and tested with end-users

Evaluating Ubiquitous Systems with Users (Workshop Summary)

Evaluating ubiquitous systems with users can be a challenge, and the goal of this workshop was to take stock of current issues and novel approaches to address this challenge. In this paper, we report on the discussions we had during several plenary and small-group sessions. We first briefly review those evaluation methods that we identified as being used in ubiquitous computing, and then discuss several issues and research questions that emerged during the discussion. These issues include: data sources used for evaluation, comparing ubiquitous systems, interdisciplinary evaluation, multi-method evaluation, factoring in context and disengaged users.