Adam R. Tindale

Training Surrogate Sensors in Musical Gesture Acquisition Systems

Capturing the gestures of music performers is a common task in interactive electroacoustic music. The captured gestures can be mapped to sounds, synthesis algorithms, visuals, etc., or used for music transcription. Two of the most common approaches for acquiring musical gestures are: 1) “hyper-instruments” which are “traditional” musical instruments enhanced with sensors for directly detecting the gestures and 2) “indirect acquisition” in which the only sensor is a microphone capturing the audio signal. Hyper-instruments require invasive modification of existing instruments which is frequently undesirable. However, they provide relatively straightforward and reliable sensor measurements. On the other hand, indirect acquisition approaches typically require sophisticated signal processing and possibly machine learning algorithms in order to extract the relevant information from the audio signal. The idea of using direct sensor(s) to train a machine learning model for indirect acquisition is proposed in this paper. The resulting trained “surrogate” sensor can then be used in place of the original direct invasive sensor(s) that were used for training. That way, the instrument can be used unmodified in performance while still providing the gesture information that a hyper-instrument would provide. In addition, using this approach, large amounts of training data can be collected with minimum effort. Experimental results supporting this idea are provided in two detection contexts: 1) strike position on a drum surface and 2) strum direction on a sitar.

Node and Message Management with the JunctionBox Interaction Toolkit

Message mapping between control interfaces and sound engines is an important task that could benefit from tools that streamline development. A new Open Sound Control (OSC) namespace called Nexus Data Exchange Format (NDEF) streamlines message mapping by offering developers the ability to manage sound engines as network nodes and to query those nodes for the messages in their OSC address spaces. By using NDEF, developers will have an easier time managing nodes and their messages, especially for scenarios in which a single application or interface controls multiple sound engines. NDEF is currently implemented in the JunctionBox interaction toolkit but could easily be implemented in other toolkits.