Tony Myatt

A Video System for Recognizing Gestures by Artificial Neural Networks for Expressive Musical Control

In this paper we describe a system to recognize gestures to control musical processes. For that we applied a Time Delay Neuronal Network to match gestures processed as variation of luminance information in video streams. This resulted in recognition rates of about 90% for 3 different types of hand gestures and it is presented here as a prototype for a gestural recognition system that is tolerant to ambient conditions and environments. The neural network can be trained to recognize gestures difficult to be described by postures or sign language. This can be used to adapt to unique gestures of a performer or video sequences of arbitrary moving objects. We will discuss the outcome of extending the system to learn successfully a set of 17 hand gestures. The application was implemented in jMax to achieve real-time conditions and easy integration into a musical environment. We will describe the design and learning procedure of the using the Stuttgart Neuronal Network Simulator. The system aims to integrate into an environment that enables expressive control of musical parameters (KANSEI).

Sound in space

There has been an extraordinary amount of interest in sound spatialisation in recent years. The volume of work in this area has pushed many international organisations to feature sound spatialisation as a theme for conferences and publications, and Organised Sound is probably overdue in having a thematic issue based upon aspects of sound in space. This issue of Organised Sound contains many different views of space. We have included articles about spatialisation techniques, our understanding of sound spatialisation, composition, performance and spaces inside music as well as the spaces and environments where music can exist, including virtual spaces.

Recognition of separate hand gestures by Time-Delay Neural Networks based on multi-state spectral image patterns from cyclic hand movements

In recent years video-basedanalysis of human motion has gained increased interest, which for a large part is due to the ongoing rapid developments of computer and camera hardware, such as increased CPU power, fast and modular interfaces and high-quality image digitisation. A similar important role is played by the development of powerful approaches for the analysis of visual data from video sources. In computer music this development is reflected in a row of applications approaching the analysis of video and image data for gestural control of music and sound, such as Eyesweb, Jitter, openCV, Gem, [1], [2], [3], [4]. In this paper an approach is presented for the control of music and sound parameters through hand gestures, which are recognised through a Time-delay Neural Network (TDNN). The recognition networks were trained with appearance-based features extracted from image sequences of a video camera. Cyclic hand gestures are proposed to enable fast and seamless recording and annotation of time series as multi-state patterns. For the supervised training of the TDNNs feature maps based on spatial Fourier transformation of image sequences are proposed. The integration of the gesture recognition in an interactive music piece will be described at the end of the paper.

Strategies for interaction in construction 3

This paper describes the interactive strategies adopted during the composition and realisation of construction 3 for soprano saxophone and multiple media, by Tony Myatt and Peter Fluck. This work is an interactive multiple media composition for computer, computer-enhanced saxophone and computer graphics projected onto irregularly shaped screens. Derivation of the performer sensing system and mapping strategies to control signal processing, computer generated materials and communication gestures between the performer and computer are described. These processes include generative mapping techniques using neural networks for the recognition of gestural information and the development and application of wireless, wearable sensing technology. This work is described as a holistic approach to the derivation of performer sensing, data mapping and the application of reactive and interactive processes in the context of the creation of a new musical and multiple media composition.