Ellen Campana

Targeted help for spoken dialogue systems: intelligent feedback improves naive users' performance

We present experimental evidence that providing naive users of a spoken dialogue system with immediate help messages related to their out-of-coverage utterances improves their success in using the system. A grammar-based recognizer and a Statistical Language Model (SLM) recognizer are run simultaneously. If the grammar-based recognizer suceeds, the less accurate SLM recognizer hypothesis is not used. When the grammar-based recognizer fails and the SLM recognizer produces a recognition hypothesis, this result is used by the Targeted Help agent to give the user feedback on what was recognized, a diagnosis of what was problematic about the utterance, and a related in-coverage example. The in-coverage example is intended to encourage alignment between user inputs and the language model of the system. We report on controlled experiments on a spoken dialogue system for command and control of a simulated robotic helicopter.

A dynamic Bayesian approach to computational Laban shape quality analysis

Laban movement analysis (LMA) is a systematic framework for describing all forms of human movement and has been widely applied across animation, biomedicine, dance, and kinesiology. LMA (especially Effort/Shape) emphasizes how internal feelings and intentions govern the patterning of movement throughout the whole body. As we argue, a complex understanding of intention via LMA is necessary for human-computer interaction to become embodied in ways that resemble interaction in the physical world. We thus introduce a novel, flexible Bayesian fusion approach for identifying LMA Shape qualities from raw motion capture data in real time. The method uses a dynamic Bayesian network (DBN) to fuse movement features across the body and across time and as we discuss can be readily adapted for low-cost video. It has delivered excellent performance in preliminary studies comprising improvisatory movements. Our approach has been incorporated in Response, a mixed-reality environment where users interact via natural, full-body human movement and enhance their bodily-kinesthetic awareness through immersive sound and light feedback, with applications to kinesiology training, Parkinsons patient rehabilitation, interactive dance, and many other areas.

Talking through procedures: an intelligent space station procedure assistant

We present a prototype system aimed at providing spoken dialogue support for complex procedures aboard the International Space Station. The system allows navigation one line at a time or in larger steps. Other user functions include issuing spoken corrections, requesting images and diagrams, recording voice notes and spoken alarms, and controlling audio volume.

Two Diverse Systems Built using Generic Components for Spoken Dialogue (Recent Progress on TRIPS)

This paper describes recent progress on the TRIPS architecture for developing spoken-language dialogue systems. The interactive poster session will include demonstrations of two systems built using TRIPS: a computer purchasing assistant, and an object placement (and manipulation) task.

Amateur Musicians, Long-Term Engagement, and HCI

Musical instruments have a property of long-term engagement: people frequently become so engaged with them that they practice and play them for years, despite receiving no compensation other than enjoyment. We examine this phenomenon by analysing how the intrinsic motives mastery, autonomy, and purpose are built into the design of musical instruments; because, according to the self-determination theory of motivation, these three motives impact whether an activity might be found enjoyable. This analysis resulted in the identification of seven abstract qualities, inherent to the activity of music making or to the design of musical instruments, which contribute to the three intrinsic motives. These seven qualities can be treated as heuristics for the design of human-computer interfaces that have long-term engagement. These heuristics can be used throughout the design process, from the preliminary stage of idea generation to the evaluation stage of finished prototypes. Interfaces with instrument-like long-term engagement would be useful in many applications, both inside and outside the realm of music: they seem particularly suited for applications based on the attainment of long-term goals, which can be found in fields such as physical fitness, rehabilitation, education, and many others. In this chapter, we discuss an interface prototype we created and its pending evaluation. This interface, a rehabilitative rhythm game, serves as a case study showing how the heuristics might be used during the design process.

Towards a Formal Representation Model of Problem Formulation in Design

Studies on design, show that problem formulation plays a major role in creative design. We plan to construct an interactive computer system that aids problem formulation. In the current stage, to improve our understanding of problem formulation, we have conducted exploratory protocol studies of novice designers and collected data from an expert designer in the form of a depositional interview. A formal representation of the design problem is needed to improve our empirical investigation. We propose a preliminary framework for such a model and we call it a problem map. It provides a basis for comparing how different designers perceive a problem. Our study is based on the design of a model aircraft for the AIAA student design competition. This preliminary analysis shows the evolution of the problem and the solution spaces in the elaboration of the problem maps through time. The problem maps also show a richer representation of attended attributes and relations for the expert and more attributes left in vacuum for the novices.© 2011 ASME

Fruit Carts: A Domain and Corpus for Research in Dialogue Systems and Psycholinguistics

We describe a novel domain, Fruit Carts, aimed at eliciting human language production for the twin purposes of (a) dialogue system research and development and (b) psycholinguistic research. Fruit Carts contains five tasks: choosing a cart, placing it on a map, painting the cart, rotating the cart, and filling the cart with fruit. Fruit Carts has been used for research in psycholinguistics and in dialogue systems. Based on these experiences, we discuss how well the Fruit Carts domain meets four desired features: unscripted, context-constrained, controllable difficulty, and separability into semi-independent subdialogues. We describe the domain in sufficient detail to allow others to replicate it; researchers interested in using the corpora themselves are encouraged to contact the authors directly.

SMALLab: a mixed-reality environment for embodied and mediated learning

In this video presentation, we introduce the Situated Multimedia Arts Learning Lab [SMALLab], a mixed-reality learning environment that supports interactive engagement through full body 3D movements and gestures within a collaborative, computationally mediated space. The video begins by describing the holistic approach to embodied and mediated learning developed by our transdisciplinary research team, grounded in understandings derived from research in the learning sciences, digital media and human computer interaction. We then outline the three core tenets of effective learning exemplified by our research -- embodiment, multimodality and collaboration. The video next demonstrates the design and functionality of the physical and digital components of SMALLab. We conclude by illustrating our partner collaborations with K12 teachers and students with four scenarios depicting Geography, Physics, Language Arts and Chemistry learning modules.

Capturing Expressive and Indicative Qualities of Conducting Gesture: An Application of Temporal Expectancy Models

Many event sequences in everyday human movement exhibit temporal structure: for instance, footsteps in walking, the striking of balls in a tennis match, the movements of a dancer set to rhythmic music, and the gestures of an orchestra conductor. These events generate prior expectancies regarding the occurrence of future events. Moreover, these expectancies play a critical role in conveying expressive qualities and communicative intent through the movement; thus they are of considerable interest in musical control contexts. To this end, we introduce a novel Bayesian framework which we call the temporal expectancy modeland use it to develop an analysis tool for capturing expressiveand indicativequalities of the conducting gesture based on temporal expectancies. The temporal expectancy model is a general dynamic Bayesian network (DBN) that can be used to encode prior knowledge regarding temporal structure to improve event segmentation. The conducting analysis tool infers beat and tempo, which are indicative and articulation which is expressive, as well as temporal expectancies regarding beat (ictusand preparationinstances) from conducting gesture. Experimental results using our analysis framework reveal a very strong correlation in how significantly the preparation expectancy builds up for staccato vs legato articulation, which bolsters the case for temporal expectancy as cognitive model for event anticipation, and as a key factor in the communication of expressive qualities of conducting gesture. Our system operates on data obtained from a marker based motion capture system, but can be easily adapted for more affordable technologies like video camera arrays.

On Cross-Modal Perception of Musical Tempo and the Speed of Human Movement

Studies in crossmodal perception often use very simplified auditory and visual contexts. While these studies have been theoretically valuable, it is sometimes difficult to see how the findings can be ecologically valid or practically valuable. This study hypothesizes that a musical parameter (tempo) may affect the perception of a human movement quality (speed) and finds that although there are clear limitations, this may be a promising first step towards widening both the contexts in which cross-modal effects are studied and the application areas in which the findings can be used.