David Grunberg

Creating an autonomous dancing robot

A robot with the ability to dance autonomously has many potential applications, such as serving as a prototype dancer for choreographers or as a participant in stage performances with human dancers. A robot that dances autonomously must be able to extract several features from audio in real time, including tempo, beat, and style. It must also be able to produce a continuous sequence of humanlike gestures. We chose the Hitec RoboNova to use as a robot platform in our work on these problems. We have developed a beat identification algorithm that can extract the beat positions from audio in real time for multiple consecutive songs. Our RoboNova can now produce sequences of smooth gestures that are synchronized with the predicted beats and match the tempo of the audio. Our algorithm can also be easily moved to the HUBO, a large humanoid robot that can move in a very humanlike manner.

Robot audition and beat identification in noisy environments

In pursuit of our long-term goal of developing an interactive humanoid musician, we are developing robust methods to determine musical beat locations from live acoustic sources. A variety of beat tracking systems have been previously developed, but for the most part they are optimized for direct audio input (no acoustic channel and no noise). The presence of an acoustic channel and noise typically degrades performance substantially. A robots motors, in particular, create nonstationary noise that can be difficult for a beat detection system to accommodate, Using an algorithm previously developed by the authors, we explore techniques for reducing the effects of the acoustic channel and noise on the system, enabling a humanoid to robustly follow music under realistic conditions.

Affective gesturing with music mood recognition

The recognition of emotions and the generation of appropriate responses is a key component for facilitating more natural human-robot interaction. Music, often called the “language of emotions,” is a particularly useful medium for investigating questions involving the expression of emotion. Likewise, movements and gestures, such as dance, can also communicate specific emotions to human observers. We apply an efficient, causal technique for estimating the emotions (mood) from music audio to enable a humanoid to perform gestures reflecting the musical mood. We implement this system using Hubo, an adult-sized humanoid that has been used in several applications of musical robotics. Our preliminary experiments indicate that the system is able to produce dance-like gestures that are judged by human observers to match the perceived emotion of the music.

Enabling humanoid musical interaction and performance

Many people incorporate music into their daily lives, and the development of robots with musical awareness provides an opportunity for rich forms of human-robot interaction. Robots must, however, acquire a variety of skills before being able to participate in musical activities. In order to dance or play an instrument, for example, a robot be able to utilize substantial auditory and visual information. Our work focuses on providing such capabilities (audio and visual beat detection, pitch detection, motion control for producing musical notes) with the goal of enabling an adult-sized humanoid to be an interactive participant in a live musical ensemble. Miniature humanoids are used to prototype and refine many of these systems before deploying them on Hubo, an adult- sized humanoid developed by the Korean Advanced Institute of Science and Technology.

From RoboNova to HUBO: Platforms for Robot Dance

A robot with the ability to dance in response to music could lead to novel and interesting interactions with humans. For example, such a robot could be used to augment live performances alongside human dancers. This paper describes a system enabling humanoid robots to move in synchrony with music. A small robot, the Hitec RoboNova, was initially used to develop smooth sequences of complex gestures used in human dance. The system uses a real-time beat prediction algorithm so that the robot’s movements are synchronized with the audio. Finally, we implemented the overall system on a much larger robot, HUBO, to establish the validity of the smaller RoboNova as a useful prototyping platform.

Rapidly learning musical beats in the presence of environmental and robot ego noise

Humans can often learn high-level features of a piece of music, such as beats, from only a few seconds of audio. If robots could obtain this information just as rapidly, they would be more capable of musical interaction without needing long lead times to learn the music. The presence of robot ego noise, however, makes accurately analyzing music more difficult. In this paper, we focus on the task of learning musical beats, which are often identifiable to humans even in noisy environments such as bars. Learning beats would not only help robots to synchronize their responses to music, but could lead to learning other aspects of musical audio, such as other repeated events, timbrel aspects, and more. We introduce a novel algorithm utilizing stacked spectrograms, in which each column contains frequency bins from multiple instances in time, as well as Probabilistic Latent Component Analysis (PLCA) to learn beats in noisy audio. The stacked spectrograms are exploited to find time-varying spectral characteristics of acoustic components, and PLCA is used to learn and separate the components and find those containing beats. We demonstrate that this system can learn musical beats even when only provided with a few seconds of noisy audio.

Music-information retrieval in environments containing acoustic noise

In the field of Music-Information Retrieval (Music-IR), algorithms are used to analyze musical signals and estimate high-level features such as tempi and beat locations. These features can then be used in tasks to enhance the experience of listening to music. Most conventional Music-IR algorithms are trained and evaluated on audio that is taken directly from professional recordings with little acoustic noise. However, humans often listen to music in noisy environments, such as dance clubs, crowded bars, and outdoor concert venues. Music-IR algorithms that could function accurately even in these environments would therefore be able to reliably process more of the audio that humans hear. In this paper, I propose methods to perform Music-IR tasks on music that has been contaminated by acoustic noise. These methods incorporate algorithms such as Probabilistic Latent Component Analysis (PLCA) and Harmonic-Percussive Source Separation (HPSS) in order to identify important elements of the noisy musical signal. As an example, a noise-robust beat tracker utilizing these techniques is described.