Kai-yuh Hsiao

The magic carpet: physical sensing for immersive environments

An interactive environment has been developed that uses a pair of Doppler radars to measure upper-body kinematics (velocity, direction of motion, amount of motion) and a grid of piezoelectric wires hidden under a 6 x 10 foot carpet to monitor dynamic foot position and pressure. This system has been used in an audio installation, where users launch and modify complex musical sounds and sequences as they wander about the carpet. This paper describes the floor and radar systems, quantifies their performance, and outlines the musical application.

Sensor systems for interactive surfaces

This paper describes four different systems that we have developed for capturing various manners of gesture near interactive surfaces. The first is a low-cost scanning laser rangefinder adapted to accurately track the position of bare hands in a plane just above a large projection display. The second is an acoustic system that detects the position of taps on a large, continuous surface (such as a table, wall, or window) by measuring the differential time-of-arrival of the acoustic shock impulse at several discrete locations. The third is a sensate carpet that uses a grid of piezoelectric wire to measure the dynamic location and pressure of footfalls. The fourth is a swept radio frequency (RF) tag reader that measures the height, approximate location, and other properties (orientation or a control variable like pressure) of objects containing passive, magnetically coupled resonant tags, and updates the continuous parameters of all tagged objects at 30 Hz. In addition to discussing the technologies and surveying different approaches, sample applications are given for each system.

Design and implementation of expressive footwear

As an outgrowth of our interest in dense wireless sensing and expressive applications of wearable computing, the Responsive Environments Group at the MIT Media Laboratory has developed a very versatile human-computer interface for the foot. By dense wireless sensing, we mean the remote acquisition of many different parameters with a compact, autonomous sensor cluster. We have developed such a low-power sensor card to measure over 16 continuous quantities and transmit them wirelessly to a remote base station, updating all variables at 50 Hz. We have integrated a pair of these devices onto the shoes of dancers and athletes, measuring continuous pressure at three points near the toe, dynamic pressure at the heel, bidirectional bend of the sole, height of each foot off conducting strips in the stage, angular rate of each foot about the vertical, angular position of each foot about the Earths local magnetic field, as well as foot tilt and acceleration, 3-axis shock acceleration (from kicks and jumps), and position (via an integrated sonar). This paper describes the sensor and electronics systems, then outlines several projects in which we have applied these shoes for interactive dance and the capture of high-level podiatric gesture. We conclude by outlining several applications of our sensor system that are unrelated to footwear.

Electromagnetic Tagging for Electronic Music Interfaces

This paper describes the development of a musical interface based on electromagnetic tagging technology, where an ensemble of passively tagged objects is identified and tracked in real time when placed in the vicinity of a reader. As the system is able to identify and update the state of many (30 or more) tags simultaneously, they can be used together in any combination – e.g., several can sit on a surface at fixed distances from the reader while others can be handheld or worn by a single user or multiple performers. This interface is able to detect both free gesture (position and orientation of the objects) as well as local or tactile variables (e.g., pressure). We describe a series of controllers that exploit the musical possibilities of this architecture – the somewhat constrained Musical Trinkets, where objects were tied to simple notes and musical effects, and its successor, the Musical Navigatrics, which enabled dynamic overdubbing and control of complex musical sequences and sonic textures. We close with a description of a very simple and inexpensive actively tagged tracking system capable of much wider range.

Interfacing to the foot: apparatus and applications

We describe a system that we have developed to capture detailed, multimodal gesture expressed at the foot. It is embodied in a pair of shoes, each of which measures 16 degrees of freedom (tactile, inertial, positional). No tethers or wires are attached to the shoes; data is directly telemetered wirelessly off each foot to a remote base station and host computer, yielding full state updates at 50 Hz. This system, having evolved over 3 years, has been used for real-time expressive performance by a diverse set of artists, including gymnasts, jugglers, and dancers. Ongoing work is exploring the extraction of high-level podiatric gesture.

Swept-frequency, magnetically-coupled resonant tags for realtime, continuous, multiparameter control

We have developed a passive tag reader optimized for applications in human-computer interaction. It sweeps through a 50-300 kHz read frequency, flagging any magnetically-coupled resonators in that range. It is a minimally-complicated circuit, and is able to provide the center frequency, resonance width, and amplitude for each detected tag over a serial line at 30 Hz continuous updates. The tags are easily fashioned, consisting only of an inductor and capacitor or magnetostrictor tag cut to appropriate length. We have written an engaging musical application to demonstrate this system, tagging over 11 different objects and tracking their proximity and state, launching or modifying musical sounds in accordance.