Joanne L. Armitage

mConduct : A Multi-sensor Interface for the Capture and Analysis of Conducting Gesture

The art of conducting has a long and well-established history, as both a technical and expressive art form, using physical gesture to convey musical intent and expression. Conducting relies on visual communication to direct the individual instrumentalists of an ensemble as a single, coherent unit. The aim of this project is to capture and analyse the hand gestures of conducting in order to provide real-time, interactive multimodal feedback. The system encompasses a number of application contexts including gesture visualisation for conducting analysis, pedagogy and preservation. This chapter presents the design and development of the interface involving hardware sensors and software analysis modules, and discusses the application of visualisation for conducting. Visualisation software has been designed to produce a sculpture of the conductor’s gesture, to reflect the individual conductor’s style and technique. The chapter concludes with latest findings, future directions and the impact the research may have outside the realm of gesture communication application.

The Colour of Music: Real-time Music Visualisation with Synaesthetic Sound-colour Mapping

Multimodality is integrated into the majority of our multimedia experiences, most commonly with intent. However, some experience an unintentional, and subjective perceptual response to a stimulus in another sensory domain due to the neurological phenomena synaesthesia. This paper discusses the application of music-colour synaesthetic mappings in live ensemble performance. A succinct discourse on literature related to synaesthesia scientifically and creatively is presented, followed by an overview of the design and development of the system. Exemplification of the mapping strategies is offered through an illustration of one potential output. Finally, the paper concludes with a brief summary alongside ongoing developments.

Augmented Opera Performance

Operatic performance has a long and well-established history as a large-scale performance encompassing elements of music, theatre and design. Staging of such productions is an intricate process involving many parties both on the stage and behind the scenes. This paper discusses multimodal and multimedia technologies and techniques that can assist this practice. We look particularly at the conductor, as the controller of temporal progression, leading the various components as a cohesive unit. For this reason, a particular focus is upon the development of methods to communicate cues and information for lighting, visual stimuli and haptic feedback. Previous work that has informed the design of this system is also discussed. The paper concludes with the project’s current status and impact.

Configuring a haptic interface for music performance

Our daily interactions and perceptions involve multiple sensory multimodalities; most exchanges are inherently multimodal, playing an important part in our decoding and understanding of environments. Events can often engage multiple senses. Furthermore, musical experiences can be highly multisensory, with obvious auditory stimulation, the visual elements of a live performance, and physical stimulation of the body. In this paper, we will propose a means of incorporating an additional somatic channel of communication into live performances and compositional practice to further augment the physical nature of live performance. This work explores the integration of augmented vibratory, or haptic stimulation for audiences in live performance. The vibration interface is presented as an expressive and creative live performance-based tool for composers. Vibrations, or haptics, are implemented as an additional instrumental line, alongside auditory musical gestures, to expand the composers palette of expressions through augmented somatic engagement. This paper will describe the background, and design and development of a haptic interface for the purpose of audio-haptic listening-feeling. The focus of this paper is to describe a study into motor latency for informing multimedia simultaneity.