Todd Ingalls

A gesture-driven multimodal interactive dance system

In this paper, we report a real-time gesture driven interactive system with multimodal feedback for performing arts, especially dance. The system consists of two major parts., a gesture recognition engine and a multimodal feedback engine. The gesture recognition engine provides real-time recognition of the performers gesture based on the 3D marker coordinates from a marker-based motion capture system. According to the recognition results, the multimodal feedback engine produces associated visual and audio feedback to the performer. This interactive system is simple to implement and robust to errors in 3D marker data. Satisfactory interactive dance performances have been successfully created and presented using the reported system

Movement-based interactive dance performance

Movement-based interactive dance has recently attracted great interest in the performing arts. While utilizing motion capture technology, the goal of this project was to design the necessary real-time motion analysis engine, staging, and communication systems for the completion of a movement-based interactive multimedia dance performance. The movement analysis engine measured the correlation of dance movement between three people wearing similar sets of retro-reflective markers in a motion capture volume. This analysis provided the framework for the creation of an interactive dance piece, Lucidity, which will be described in detail. Staging such a work also presented additional challenges. These challenges and our proposed solutions will be discussed. We conclude with a description of the final work and a summary of our future research objectives.

Real-time Gesture Recognition with Minimal Training Requirements and On-line Learning

In this paper, we introduce the semantic network model (SNM), a generalization of the hidden Markov model (HMM) that uses factorization of state transition probabilities to reduce training requirements, increase the efficiency of gesture recognition and on-line learning, and allow more precision in gesture modeling. We demonstrate the advantages both formally and experimentally, using examples such as full-body multimodal gesture recognition via optical motion capture and a pressure sensitive floor, as well as mouse/pen gesture recognition. Our results show that our algorithm performs much better than the traditional approach in situations where training samples are limited and/or the precision of the gesture model is high.

A dynamic Bayesian approach to computational Laban shape quality analysis

Laban movement analysis (LMA) is a systematic framework for describing all forms of human movement and has been widely applied across animation, biomedicine, dance, and kinesiology. LMA (especially Effort/Shape) emphasizes how internal feelings and intentions govern the patterning of movement throughout the whole body. As we argue, a complex understanding of intention via LMA is necessary for human-computer interaction to become embodied in ways that resemble interaction in the physical world. We thus introduce a novel, flexible Bayesian fusion approach for identifying LMA Shape qualities from raw motion capture data in real time. The method uses a dynamic Bayesian network (DBN) to fuse movement features across the body and across time and as we discuss can be readily adapted for low-cost video. It has delivered excellent performance in preliminary studies comprising improvisatory movements. Our approach has been incorporated in Response, a mixed-reality environment where users interact via natural, full-body human movement and enhance their bodily-kinesthetic awareness through immersive sound and light feedback, with applications to kinesiology training, Parkinsons patient rehabilitation, interactive dance, and many other areas.

Design of a home-based adaptive mixed reality rehabilitation system for stroke survivors

This paper presents the design of a home-based adaptive mixed reality system (HAMRR) for upper extremity stroke rehabilitation. The goal of HAMRR is to help restore motor function to chronic stroke survivors by providing an engaging long-term reaching task therapy at home. The system uses an intelligent adaptation scheme to create a continuously challenging and unique multi-year therapy experience. The therapy is overseen by a physical therapist, but day-to-day use of the system can be independently set up and completed by a stroke survivor. The HAMMR system tracks movement of the wrist and torso and provides real-time, post-trial, and post-set feedback to encourage the stroke survivor to self-assess his or her movement and engage in active learning of new movement strategies. The HAMRR system consists of a custom table, chair, and media center, and is designed to easily integrate into any home.

Amateur Musicians, Long-Term Engagement, and HCI

Musical instruments have a property of long-term engagement: people frequently become so engaged with them that they practice and play them for years, despite receiving no compensation other than enjoyment. We examine this phenomenon by analysing how the intrinsic motives mastery, autonomy, and purpose are built into the design of musical instruments; because, according to the self-determination theory of motivation, these three motives impact whether an activity might be found enjoyable. This analysis resulted in the identification of seven abstract qualities, inherent to the activity of music making or to the design of musical instruments, which contribute to the three intrinsic motives. These seven qualities can be treated as heuristics for the design of human-computer interfaces that have long-term engagement. These heuristics can be used throughout the design process, from the preliminary stage of idea generation to the evaluation stage of finished prototypes. Interfaces with instrument-like long-term engagement would be useful in many applications, both inside and outside the realm of music: they seem particularly suited for applications based on the attainment of long-term goals, which can be found in fields such as physical fitness, rehabilitation, education, and many others. In this chapter, we discuss an interface prototype we created and its pending evaluation. This interface, a rehabilitative rhythm game, serves as a case study showing how the heuristics might be used during the design process.

An interactive multimedia system for Parkinson's patient rehabilitation

This paper describes a novel real-time Multimedia Rehabilitation Environment for the rehabilitation of patients with Parkinsons Disease (PD). The system integrates two well known physical therapy techniques, multimodal sensory cueing and BIG protocol, with visual and auditory feedback to created an engaging mediated environment. The environment has been designed to fulfill the both the needs of the physical therapist and the patient.

A real-time, multimodal biofeedback system for stroke patient rehabilitation

This paper presents a novel real-time, multi-modal biofeedback system for stoke patient therapy. The problem is important as traditional mechanisms of rehabilitation are monotonous, and do not incorporate detailed quantitative assessment of recovery in addition to traditional clinical schemes. We have been working on developing an experiential media system that integrates task dependent physical therapy and cognitive stimuli within an interactive, multimodal environment. The environment provides a purposeful, engaging, visual and auditory scene in which patients can practice functional therapeutic reaching tasks, while receiving different types of simultaneous feedback indicating measures of both performance and results. There are two contributions of this paper - (a) identification of features and goals for the functional task, (b) the development of sophisticated feedback (auditory and visual) mechanisms that match the semantics of action of the task.

Interdisciplinary Concepts for Design and Implementation of Mixed Reality Interactive Neurorehabilitation Systems for Stroke

Interactive neurorehabilitation (INR) systems provide therapy that can evaluate and deliver feedback on a patients movement computationally. There are currently many approaches to INR design and implementation, without a clear indication of which methods to utilize best. This article presents key interactive computing, motor learning, and media arts concepts utilized by an interdisciplinary group to develop adaptive, mixed reality INR systems for upper extremity therapy of patients with stroke. Two INR systems are used as examples to show how the concepts can be applied within: (1) a small-scale INR clinical study that achieved integrated improvement of movement quality and functionality through continuously supervised therapy and (2) a pilot study that achieved improvement of clinical scores with minimal supervision. The notion is proposed that some of the successful approaches developed and tested within these systems can form the basis of a scalable design methodology for other INR systems. A coherent approach to INR design is needed to facilitate the use of the systems by physical therapists, increase the number of successful INR studies, and generate rich clinical data that can inform the development of best practices for use of INR in physical therapy.

SomaTech: an exploratory interface for altering movement habits

We propose SomaTech, a Kinect-based system that encourages users to expand understanding and awareness of their everyday movements. The system creates real-time auditory feedback based on the users whole action, aiming toward re-education of habitual, potentially unsound movement patterns which are often ingrained within the brain. To do this, we draw inspiration from the field of somatics, which has well-studied prophylactic benefits. Our initial evaluation shows promising results that users become more aware of movement choices and are able to improve their efficiency after using the system.