Federico Visi

Gesture in performance with traditional musical instruments and electronics: Use of embodied music cognition and multimodal motion capture to design gestural mapping strategies

This paper describes the implementation of gestural mapping strategies for performance with a traditional musical instrument and electronics. The approach adopted is informed by embodied music cognition and functional categories of musical gestures. Within this framework, gestures are not seen as means of control subordinated to the resulting musical sounds but rather as significant elements contributing to the formation of musical meaning similarly to auditory features. Moreover, the ecological knowledge of the gestural repertoire of the instrument is taken into account as it defines the action-sound relationships between the instrument and the performer and contributes to form expectations in the listeners. Subsequently, mapping strategies from a case study of electric guitar performance will be illustrated describing what motivated the choice of a multimodal motion capture system and how different solutions have been adopted considering both gestural meaning formation and technical constraints.

Analysis of Mimed Violin Performance Movements of Neophytes

Body movement and embodied knowledge play an important part in how we express and understand music. The gestures of a musician playing an instrument are part of a shared knowledge that contributes to musical expressivity by building expectations and influencing perception. In this study, we investigate the extent in which the movement vocabulary of violin performance is part of the embodied knowledge of individuals with no experience in playing the instrument. We asked people who cannot play the violin to mime a performance along an audio excerpt recorded by an expert. They do so by using a silent violin, specifically modified to be more accessible to neophytes. Preliminary motion data analyses suggest that, despite the individuality of each performance, there is a certain consistency among participants in terms of overall rhythmic resonance with the music and movement in response to melodic phrasing. Individualities and commonalities are then analysed using Functional Principal Component Analysis.

Modosc: A Library of Real-Time Movement Descriptors for Marker-Based Motion Capture

Marker-based motion capture systems that stream precise movement data in real-time afford interaction scenarios that can be subtle, detailed, and immediate. However, challenges to effectively utilizing this data include having to build bespoke processing systems which may not scale well, and a need for higher-level representations of movement and movement qualities. We present modosc, a set of Max abstractions for computing motion descriptors from raw motion capture data in real time. Modosc is designed to address the data handling and synchronization issues that arise when working with complex marker sets, and to structure data streams in a meaningful and easily accessible manner. This is achieved by adopting a multiparadigm programming approach using o.dot and Open Sound Control. We describe an initial set of motion descriptors, the addressing system employed, and design decisions and challenges.

Motion Controllers, Sound, and Music in Video Games: State of the Art and Research Perspectives

This chapter is dedicated to the use of motion-sensing controllers in video games, with particular attention to gameplay mechanics that link body movement, sound, an music. First, we present a review of the state of the art. This includes an overview of recent motion controllers describing the different motion-sensing technologies employed and the implications such technologies have on game design and gameplay. This is followed by a set of examples of relationships between motion control, sound, music, and gameplay mechanics in recent video games. Secondly, we present a survey on recent research in expressive movement analysis and motion-based interaction, introducing the concepts of motion descriptor and parameter mapping. We then report on two serious games designed for children affected by autism using full-body motion capture and emotion recognition, and studies of player engagement in motion-controlled video games. In light of the interdisciplinary research presented, we finally discuss perspectives for motion-based game design.

3CMS: An Interactive Decision System for Live Performance

A machine system is designed to analyze the musical aspects during the live performance, allowing an interactive and dynamic flow of new expressions and also opening new compositional forms and multimodal methods. The focus of this approach is to measure the expressiveness from distinct characters during the performance of the musical piece while decisions are made by the machine. This multimodal approach is implemented in the musical piece Tres Microcancoes de Câmara – Essencia Pierrot, Atitude Arlequim, (In)Decisao Colombina (3CMS), where audio features and body motion are used by the algorithm to choose a particular musical ending.