Rieks op den Akker

Dialogue Act Recognition with Bayesian Networks for Dutch Dialogues

This paper presents work on using Bayesian networks for the dialogue act recognition module of a dialogue system for Dutch dialogues. The Bayesian networks can be constructed from the data in an annotated dialogue corpus. For two series of experiments - using different corpora but the same annotation scheme - recognition results are presented and evaluated.

Virtual meeting rooms: from observation to simulation

Much working time is spent in meetings and, as a consequence, meetings have become the subject of multidisciplinary research. Virtual Meeting Rooms (VMRs) are 3D virtual replicas of meeting rooms, where various modalities such as speech, gaze, distance, gestures and facial expressions can be controlled. This allows VMRs to be used to improve remote meeting participation, to visualize multimedia data and as an instrument for research into social interaction in meetings. This paper describes how these three uses can be realized in a VMR. We describe the process from observation through annotation to simulation and a model that describes the relations between the annotated features of verbal and non-verbal conversational behavior. As an example of social perception research in the VMR, we describe an experiment to assess human observers’ accuracy for head orientation.

Mutually Coordinated Anticipatory Multimodal Interaction

We introduce our research on anticipatory and coordinated interaction between a virtual human and a human partner. Rather than adhering to the turn taking paradigm, we choose to investigate interaction where there is simultaneous expressive behavior by the human interlocutor and a humanoid. Various applications in which we can study and specify such behavior, in particular behavior that requires synchronization based on predictions from performance and perception, are presented. Some observations concerning the role of predictions in conversations are presented and architectural consequences for the design of virtual humans are drawn.

Navigation assistance in virtual worlds

In this paper we report about ongoing research on navigation assistance in virtual environments. Our aim is to contribute to the development of forms of navigation assistance that enable non-professional visitors of a virtual environment to find their way without previous training. The environment used in this research is a virtual theatre that models a real world music theatre. This virtual theatre can be used for exploration as well as for transactions and goal-directed search for information. We first present some design principles for navigation assistance in virtual environments and some design criteria for assistance by personal agents. Subsequently we describe how these principles and criteria have been implemented in our experimental virtual theatre environment. Finally we give an overview of future research plans.

AFFECT IN TUTORING DIALOGUES

ABSTRACT This paper is about INES, an intelligent, multimodal tutoring environment, and how we build a tutor agent in the environment that tries to be sensitive to the mental state of the student that interacts with it. The environment was primarily designed to help students practice nursing tasks. For example, one of the implemented tasks is to give a virtual patient a subcutaneous injection. The students can interact multimodally using speech and a haptic device under the guidance of the virtual embodied tutor. INES takes into account elements of the students character and an appraisal of the students actions to estimate the mental state of the student. This information is used to plan and execute the actions and responses of the tutor agent.

Dialogue act recognition under uncertainty using bayesian networks

In this paper we discuss the task of dialogue act recognition as a part of interpreting user utterances in context. To deal with the uncertainty that is inherent in natural language processing in general and dialogue act recognition in particular we use machine learning techniques to train classifiers from corpus data. These classifiers make use of both lexical features of the (Dutch) keyboard-typed utterances in the corpus used, and context features in the form of dialogue acts of previous utterances. In particular, we consider probabilistic models in the form of Bayesian networks to be proposed as a more general framework for dealing with uncertainty in the dialogue modelling process.

Ontology-based generation of dynamic feedback on physical activity

Improving physical activity patterns is an important focus in the treatment of chronic illnesses. We describe a system to monitor activity and provide feedback to help patients reach a healthy daily pattern. The system has shown positive effects in trials on patient groups including COPD and obese patients. We describe the design and implementation of a new feedback generation module which improves interaction with the patient by providing personalised dynamic context-aware feedback. The system uses an ontology of messages to find appropriate feedback using context information to prune irrelevant paths. The system adapts using derived probabilities about user preferences for certain message types. We aim to improve patient compliance and user experience.

Automatic summarisation of discussion fora

Web-based discussion fora proliferate on the Internet. These fora consist of threads about specific matters. Existing forum search facilities provide an easy way for finding threads of interest. However, understanding the content of threads is not always trivial. This problem becomes more pressing as threads become longer. It frustrates users that are looking for specific information and also makes it more difficult to make valuable contributions to a discussion. We postulate that having a concise summary of a thread would greatly help forum users. But, how would we best create such summaries? In this paper, we present an automated method of summarising threads in discussion fora. Compared with summarisation of unstructured texts and spoken dialogues, the structural characteristics of threads give important advantages. We studied how to best exploit these characteristics. Messages in threads contain both explicit and implicit references to each other and are structured. Therefore, we term the threads hierarchical dialogues. Our proposed summarisation algorithm produces one summary of an hierarchical dialogue by ‘cherry-picking’ sentences out of the original messages that make up a thread. We try to select sentences usable for obtaining an overview of the discussion. Our method is built around a set of heuristics based on observations of real fora discussions. The data used for this research was in Dutch, but the developed method equally applies to other languages. We evaluated our approach using a prototype. Users judged our summariser as very useful, half of them indicating they would use it regularly or always when visiting fora.

Facial signs of affect during tutoring sessions

An emotionally intelligent tutoring system should be able to taking into account relevant aspects of the mental state of the student when providing feedback. The student’s facial expressions, put in context, could provide cues with respect to this state. We discuss the analysis of the facial expression displayed by students interacting with an Intelligent Tutoring System and our attempts to relate expression, situation and mental state building on Scherer’s component process model of emotion appraisal.

A dialogue agent for navigation support in virtual reality

We describe our work on designing a natural language accessible navigation agent for a virtual reality (VR) environment. The agent is part of an agent framework, which means that it can communicate with other agents. Its navigation task consists of guiding the visitors in the environment and to answer questions about this environment (a theatre building). Visitors are invited to explore this building, see what is there, ask questions and get advice from the navigation agent. A 2D map has been added to the environment so that visitors can make references to the locations and objects on this map, both in natural language and by clicking with the mouse, making it a multimodal system with cross-modality references.