Luc Nijs

Assessing a clarinet player's performer gestures in relation to locally intended musical targets

Abstract Musicianship is known to display high-level skills, which involve different aspects of mental processing and corporeal control. Of particular interest is the match between the musicians mental focus on musical targets (the so-called musical intentions) and the expressive (or so-called auxiliary) body movements. To what extent are these related to each other? And what does this relationship reveal about mind–body connections? To approach these questions, a case study was set up around a clarinet solo performance played from score, covering a style of music unfamiliar to the player. The clarinetists movements were recorded with an optical movement tracking system. A statistical analysis method was developed, to account for movement data in relation to the potential musical intentions and targets. The bottom-up movement analysis method was validated with the performers annotations of targets in the musical score and the performers annotations of communicative/sound facilitating gestures in the performance video. The results reveal that the mental focus on musical targets is related to bodily expression. This finding supports the idea of an embodied model of musical syntax processing, which is strongly related to corporeal gestures.

Interacting with the Music Paint Machine: Relating the constructs of flow experience and presence

In this paper we report on the results of an experiment on the experience of flow and presence while engaging with an interactive music system, the Music Paint Machine. This music system provides a game-like environment in which a musician can create a digital painting by playing an acoustic musical instrument, by moving the body in different directions, and by selecting colours using a pressure mat. The experiment aimed at getting a better insight into the possible relationship between flow experience and presence. Based on the definition of flow as a combination of the highest level of presence (presence-as-feeling) and a positive emotional state (Riva et al., 2004a), we hypothesized that presence has a predictive value for flow. Sixty-five musicians, both amateur and professional, participated in the experiment. Flow experience was measured with the Flow State Scale (Jackson and Eklund, 2004). Presence was measured with an in-house designed presence questionnaire. Results showed a significantly strong correlation between flow and presence. Moreover, the scores for presence significantly predicted the Flow State Scale, and explained a significant proportion of variance in the Flow State Scale. Furthermore, many significant associations were found between flow and presence variables, among which the most significant were the strong correlation (Spearmans rank) between the naturalness of using the system and the Flow State Scale and between the feeling of non-mediation and the Flow State Scale.

The Music Paint Machine: Stimulating Self-monitoring Through the Generation of Creative Visual Output Using a Technology-enhanced Learning Tool

Abstract In this paper, we discuss the pedagogically grounded and research-based design of a technology-enhanced learning tool, the Music Paint Machine. This interactive music system introduces a musical experience in which the musician creates a digital painting by playing an acoustic musical instrument and by moving the body on a coloured pressure mat. As a learning tool it aims at the development of musical creativity, at the stimulation of embodied understanding of music and at the development of an intimate relationship with the musical instrument. First, the methodological approach is outlined and pedagogical and theoretical backgrounds are discussed. Then, we report on an experiment in which 51 amateur musicians participated. The experiment aimed at probing the applications potential to induce a flow experience and to learn about how participants evaluate the didactic relevance of the Music Paint Machine. Results suggest that the Music Paint Machine has the potential to evoke a flow experience. Furthermore participants acknowledged its didactic relevance with regard to learning to improvise, to developing understanding of musical parameters and to stimulating creativity.

User-Oriented Studies in Embodied Music Cognition Research

Music research aims at developing a research space that, in a proactive sense, can support the development of creative and cultural industries. In that context, we argue that a focus on users and their experiences in using tools may become more important in music research. It implies an expansion of the traditional methods of music psychology with methods that can address relevant features of musical action and musical tool use. This paper discusses epistemological and methodological issues related to this development.

Gamified music improvisation with BilliArT: a multimodal installation with balls

This paper presents the concept and the realisation of an interactive multimedia installation, called BilliArT, together with an explorative user study conducted on the data gathered during a public exhibition of the installation. The study concerns functional properties of the installation (e.g. usability, design quality) and subjective qualities of the sonic and visual feedback of the installation. The installation consists in a collaborative environment based on the carambole billiards game, which allows the users to engage in a user-driven machine-based jazz-inspired music improvisation, augmented with visual feedback. The installation is designed to promote the interaction among the users and the billiard game, stimulating the motivation to engage in the game by balancing predictable and unpredictable output, and reinforcing the feeling of reward, irrespective of their level of musical training. BilliArT introduces a new framework for expressive interaction related to the concepts of motivation and reward. The exploratory study proved the ability of the installation to activate the users’ sense of aesthetic reward, leading to a more active and satisfactory engagement in the game. Future studies may exploit these results to the advantage of the world of the arts, as well as of studies in human-computer interaction, interface design, and cultural heritage preservation.

PLXTRM: Prediction-Led eXtended-guitar Tool for Real-time Music applications and live performance

This article presents PLXTRM, a system tracking picking-hand micro-gestures for real-time music applications and live performance. PLXTRM taps into the existing gesture vocabulary of the guitar player. On the first level, PLXTRM provides a continuous controller that doesn’t require the musician to learn and integrate extrinsic gestures, avoiding additional cognitive load. Beyond the possible musical applications using this continuous control, the second aim is to harness PLXTRM’s predictive power. Using a reservoir network, string onsets are predicted within a certain time frame, based on the spatial trajectory of the guitar pick. In this time frame, manipulations to the audio signal can be introduced, prior to the string actually sounding, ’prefacing’ note onsets. Thirdly, PLXTRM facilitates the distinction of playing features such as up-strokes vs. down-strokes, string selections and the continuous velocity of gestures, and thereby explores new expressive possibilities.

On the Role of the Hand in the Expression of Music

In diverse interaction processes that characterize music experience, the human hand can be seen as a mediator and facilitator for the brain’s processing of musical expressive patterns. After a brief overview on the human expressive system for music, we consider gestures and hand articulations in music production and performance, focusing on hand dexterity and hand dystonia. Then, we discuss the role of the hand in music listening, conducting, and learning, showing that both in sound-generation and sound-accompaniment the hand mediates and facilitates action and perception in relation to musical expression. The recent use of technology in the domain of music is also considered throughout the chapter, with particular reference to sensing and motion technologies that allow users to control music parameters through hand–body movements. The hand can be considered as a co-articulated organ of the brain’s action–perception machinery. Therefore, future research on hand and music should adopt a multiperspective approach that integrates different disciplines, from motor control to music performance and expression theories.

What Is Embodied Music Cognition

Over the past decade, embodied music cognition has become an influential paradigm in music research. The paradigm holds that music cognition is strongly determined by corporeally mediated interactions with music. They determine the way in which music can be conceived in terms of goals, directions, targets, values, and reward. The chapter gives an overview of the ontological and epistemological foundations, and it introduces the core concepts that define the character of the paradigm. This is followed by an overview of some analytical and empirical studies, which illustrate contributions of the embodied music cognition approach to major topics in musical expression, timing, and prediction processing. The chapter gives a viewpoint on a music research paradigm that is in full development, both in view of the in-depth refinement of its foundations, as well as the broadening of its scope and applications.

Dalcroze meets technology: integrating music, movement and visuals with the Music Paint Machine

ABSTRACT New interactive music educational technologies are often seen as a ‘force of change’, introducing new approaches that address the shortcomings (e.g. score-based, teacher-centred and disembodied) of the so-called traditional teaching approaches. And yet, despite the growing belief in their educational potential, these new technologies have been problematised with regard to their design, reception, implementation and evaluation. A possible way to optimise the realisation of the educational potential of interactive music educational technologies is to connect their use to music educational approaches that stood the test of time and as such may inspire technologies to become a bridge between tradition and innovation. This article describes an educational technology (the Music Paint Machine) that integrates the creative use of movement and visualisation to support instrumental music teaching and learning. Next, it connects this application to such an established music educational method, the Dalcroze approach. Through the lens of a set of interconnected aspects, it is shown how the Music Paint Machine’s conceptual design aligns to the underlying principles of this approach. In this way, it is argued that integrating Dalcroze-inspired practices is a plausible way of realising the didactic potential of the system. An appendix with example exercises is provided.

Gestures in Contemporary Music Performance: A Method to Assist the Performer’s Artistic Process

In this paper, we propose a method to assist a performer in dealing with the challenges of contemporary music performance. Such a method aims at making a performer’s artistic process (which is based on cognitive and sensorimotor schemes) more explicit, in order to better understand the relationship between goals, actions, and sounds. The method does not provide a heuristic that will automatically lead to a new performance model. Rather, it is meant to assist performers in (re)framing their interpretative outlooks while rendering their performative code explicit with the help of innovative mirror-like technologies. First we provide a general framework for understanding music performance, based on the notion of performance spaces and frames, and on recent insights in embodied interactions with music. Next we elaborate on how to approach the development of such understanding through performance research. Then we apply this framework to the performance of contemporary music, describing two complementary analysis approaches, in which we focus on analysis from a third-person perspective, and from a first-person perspective, respectively. The paper concludes with a discussion about the method and its value for both research and performance.