Kristinn R. Thórisson

Towards a common framework for multimodal generation: the behavior markup language

This paper describes an international effort to unify a multimodal behavior generation framework for Embodied Conversational Agents (ECAs). We propose a three stage model we call SAIBA where the stages represent intent planning, behavior planning and behavior realization. A Function Markup Language (FML), describing intent without referring to physical behavior, mediates between the first two stages and a Behavior Markup Language (BML) describing desired physical realization, mediates between the last two stages. In this paper we will focus on BML. The hope is that this abstraction and modularization will help ECA researchers pool their resources to build more sophisticated virtual humans.

Natural Turn-Taking Needs No Manual: Computational Theory and Model, from Perception to Action

Decisions like these are made by dialogue participants as often as 2–3 times per second. For a 30 minute conversation that’s over 5000 decisions. And that’s just a fraction of what goes on. How do we do it? Face-to-face dialogue consists of interaction between several complex, dynamic systems — visual and auditory display of information, internal processing, knee-jerk reactions, thought-out rhetoric, learned patterns, social convention, etc. One could postulate that the power of dialogue is a direct result of this fact. However, combining a multitude of systems in one place does not guarantee a coherent outcome such as goal-directed dialogue. For this to happen the systems need to be architected in a way that guides their interaction and ensures that — complex as it may be — the interaction tends towards homeostasis in light of errors and uncertainties, towards the set of goals shared by participants.

Mind model for multimodal communicative creatures and humanoids

This paper presents a computational model of real - time task - oriented dialog skills . The model , termed Ymir, bridges multimodal perception and multimodal action and supports the creation of autonomous computer characters that afford full - duplex , real - time face - to - face interaction with a human . Ymir has been prototyped in software , and a humanoid created , called Gandalf, capable of fluid multimodal dialog . Ymir demonstrates several new ideas in the creation of communicative computer agents , including perceptual integration of multimodal events, distributed planning and decision making, an explicit handling of real time, and perceptuo-motor system layered and motor control with human characteristics. This paper describes the models architecture and explains its main elements . Examples of implementation and performance are given , and the architectures limitations and possibilities are discussed .

Gandalf: an embodied humanoid capable of real-time multimodal dialogue with people

Gandalf is a fully autonomous, humanoid agent that perceives a users multimodal actsNspeech, prosody, manual gesture, body language, gazeNand generates appropriate multimodal responses to these in real-time (speech, gaze, facial & manual gesture & head movement). Gandalf has knowledge about the solar system and can travel to and tell users about the planets and moons with speech and gesture. Gandalf demonstrates a coherent framework for psychosoical dialogue skills which enables the production of concurrent reactive and reflective behaviors and planning of communicative acts, with response cycles analogous to those found in human face-to-face dialogue, from 1/6th of a second and up. Gandalf has been tested in interaction with humans and shown to be capable of supporting and sustaining multimodal, situated, real-time dialogue. Content Areas: Interaction between people and agents, face-to-face communication, action selection and planning, real-time performance, synthetic agents.

Learning Smooth, Human-Like Turntaking in Realtime Dialogue

Giving synthetic agents human-like realtime turntaking skills is a challenging task. Attempts have been made to manually construct such skills, with systematic categorization of silences, prosody and other candidate turn-giving signals, and to use analysis of corpora to produce static decision trees for this purpose. However, for general-purpose turntaking skills which vary between individuals and cultures, a system that can learn them on-the-job would be best. We are exploring ways to use machine learning to have an agent learn proper turntaking during interaction. We have implemented a talking agent that continuously adjusts its turntaking behavior to its interlocutors based on realtime analysis of the other partys prosody. Initial results from experiments on collaborative, content-free dialogue show that, for a given subset of turn-taking conditions, our modular reinforcement learning techniques allow the system to learn to take turns in an efficient, human-like manner.

Real-time decision making in multimodal face-to-face communication

1. ABSTRACT This paper describes an architecture and mechanism for simulating real-time decision making as observed in full-duplex, multimodal face-to-face interaction between humans. The work bridges between multimodal perception and multimodal action generation and allows flexible implementation of multimodal, fullduplex, conversational characters. It is part of a broad computational model of psychosocial dialogue skills called Ymir. The architecture has been tested with a prototype humanoid, Gandalf [34][35]. Gandalf can engage in taskoriented dialogue with a person and has been shown capable of fluid turn-taking and multimodal interaction [40]. The primary focus in this paper is the real-time decision-making (action selection) mechanism of Ymir and its relation to the multimodal perception and motor control systems.

Layered modular action control for communicative humanoids

Face-to-face interaction between people is generally effortless and effective. We exchange glances, take turns speaking and make facial and manual gestures to achieve the goals of the dialogue. This paper describes an action composition and selection architecture for synthetic characters capable of full-duplex, real-time face-to-face interaction with a human. This architecture is part of a computational model of psychosocial dialogue skills, called Y_m_i_r_, that bridges between multimodal perception and multimodal action generation. To test the architecture, a prototype humanoid has been implemented, named G_a_n_d_a_ l_f_, who commands a graphical model of the solar system and can engage in task-directed dialogue with people using speech, manual and facial gesture. Gandalf has been tested in interaction with users and has been shown capable of fluid turn-taking and multimodal dialogue. The primary focus in this paper will be on the action selection mechanisms and low-level composition of motor commands. An overview is also given of the Ymir model and Gandalfs graphical representation.

Constructionist Design Methodology for Interactive Intelligences

We present a methodology for designing and implementing interactive intelligences. The constructionist design methodology (CDM) -- so called because it advocates modular building blocks and incorporation of prior work -- addresses factors that we see as key to future advances in AI, including support for interdisciplinary collaboration, coordination of teams, and large-scale systems integration. We test the methodology by building an interactive multifunctional system with a real-time perception- action loop. The system, whose construction relied entirely on the methodology, consists of an embodied virtual agent that can perceive both real and virtual objects in an augmented-reality room and interact with a user through coordinated gestures and speech. Wireless tracking technologies give the agent awareness of the environment and the users speech and communicative acts. User and agent can communicate about things in the environment, their placement, and their function, as well as about more abstract topics, such as current news, through situated multimodal dialogue. The results demonstrate the CDMs strength in simplifying the modeling of complex, multifunctional systems that require architectural experimentation and exploration of unclear subsystem boundaries, undefined variables, and tangled data flow and control hierarchies.

Cognitive Architectures and Autonomy: A Comparative Review

Abstract One of the original goals of artificial intelligence (AI) research was to create machines with very general cognitive capabilities and a relatively high level of autonomy. It has taken the field longer than many had expected to achieve even a fraction of this goal; the community has focused on building specific, targeted cognitive processes in isolation, and as of yet no system exists that integrates a broad range of capabilities or presents a general solution to autonomous acquisition of a large set of skills. Among the reasons for this are the highly limited machine learning and adaptation techniques available, and the inherent complexity of integrating numerous cognitive and learning capabilities in a coherent architecture. In this paper we review selected systems and architectures built expressly to address integrated skills. We highlight principles and features of these systems that seem promising for creating generally intelligent systems with some level of autonomy, and discuss them in the context of the development of future cognitive architectures. Autonomy is a key property for any system to be considered generally intelligent, in our view; we use this concept as an organizing principle for comparing the reviewed systems. Features that remain largely unaddressed in present research, but seem nevertheless necessary for such efforts to succeed, are also discussed.

Multi-modal natural dialogue

Eye To analyze the user’s looking behavior wc usc a head mounted, corncal reflection eye tracker. The user looks through a half-silvered mirror; an infrared LED light shines frori above and lights up the eye. An infrared-sensitive camera picks up the reflection from the cyc off the mirror and sends the resulting TV signal to an AT 286 computer for image processing. The resulting eye data is analyzed intojixations, saccudes, and blinks,