Renee Timmers

Multimodal Analysis of Expressive Gesture in Music and Dance Performances

This paper presents ongoing research on the modelling of expressive gesture in multimodal interaction and on the development of multimodal interactive systems explicitly taking into account the role of non-verbal expressive gesture in the communication process. In this perspective, a particular focus is on dance and music as first-class conveyors of expressive and emotional content. Research outputs include (i) computational models of expressive gesture, (ii) validation by means of continuous ratings on spectators exposed to real artistic stimuli, and (iii) novel hardware and software components for the EyesWeb open platform (www.eyesweb.org), such as the recently developed Expressive Gesture Processing Library. The paper starts with a definition of expressive gesture. A unifying framework for the analysis of expressive gesture is then proposed. Finally, two experiments on expressive gesture in dance and music are discussed. This research work has been supported by the EU IST project MEGA (Multisensory Expressive Gesture Applications, www.megaproject.org) and the EU MOSART TMR Network.

The Influence of Musical Context on Tempo Rubato

Different pieces of music offer different expressive possibilities. Even a single piece of music offers the possibility to be treated in several expressive ways see (Repp, 1998). How much of this variety of possible interpretations is exhibited in actual performances of the music? Do pianists make use of the different parameters of the piece to shape their performance? Do variety in performances and variety in musical parameters relate to each other? Previous studies stress the relation between timing variations and musical structure (see Clarke, 1985), but provide no clear answer to the freedom that is allowed within this regularity, especially when multiple structural descriptions play a role simultaneously. In an experiment the melody of Variations on an Original Theme (Op. 21, No. 1) by Brahms, is set in different musical contexts derived from the Theme. Three pianists are asked to perform the melody in the different settings from a score. They repeat each performance several times. The settings are 1) the melody without bar-lines, 2) the melody with bar-lines, 3) the counter-melody, 4) the melody with the counter-melody, 5) the melody with block chords, 6) the Theme. The Theme contains all material of previous settings (the melody, counter-melody and block chords). The settings are presented in a fixed order and the pianists do not know the pieces before hand. Analysis of the recorded performances shows that pianists change the onset timing of the melody with respect to the musical context in which the melody is presented. Aspects of the context are imbedded in the timing pattern in different ways; for example, the addition of chords often causes a lengthening of the melody notes with chords, and the addition of a counter-melody constrains the lengthening of a melodic ornament. The melody proves to be the primary expressive source, while chords and a counter melody are good second ones. Both the variety in timing patterns and the extent of tempo rubato appear to increase with increasing complex conditions.

Real-Time Visual Feedback for Learning to Perform Short Rhythms with Variations in Timing and Loudness

Abstract According to learning theories and empirical observations, communication between teachers and students is a crucial factor in effective learning of musical expressions. One possibility for improving this communication could be the introduction of visual feedback (VFB) in the lesson. In the current study, a new type of real-time VFB is proposed, which represents changes in acoustical parameters (loudness and timing) as parameters of an abstract visual image (size and shape). We evaluated the effects of using VFB on imitations of timing and loudness deviations in simple rhythmic patterns. We also studied how learned skills transfer to the same task with new rhythms, as well as to new tasks. Twenty-four amateur musicians participated in the experiment that included both imitation and perception tasks. Results indicated that the VFB was helpful for improving to imitate loudness patterns, while it did not enhance improvement of learning timing patterns. Analysis of transfer of learning effects indicated that learned skills to imitate rhythms were transferred when tasks were similar: skills transferred to the same task (to imitate new rhythm) but not to new task (perception).

Vocal expression in recorded performances of Schubert songs

This exploratory study focuses on the relationship between vocal expression, musical structure, and emotion in recorded performances by famous singers of three Schubert songs. Measurement of variations in tempo, dynamics, and pitch showed highly systematic relationships with the musics structural and emotional characteristics, particularly as regards emotional activity and valence. Relationships with emotional activity were consistent across both singers and musical pieces, while relationships with emotional valence were piece-specific. Clear changes in performing style over the twentieth century were observed, including diminishing rubato, an increase followed by a decrease of the use of pitch glides, and a widening and slowing of vibrato. These systematic changes over time concern only the style of performance, not the strategies deployed to express the structural and emotional aspects of the music.

Synchronization and leadership in string quartet performance: a case study of auditory and visual cues

Temporal coordination between members of a string quartet was investigated across repeated performances of an excerpt of Haydn’s string quartet in G Major, Op. 77 No. 1. Cross-correlations between interbeat intervals of performances at different lags showed a unidirectional dependence of Viola on Violin I, and of Violin I on Cello. Bidirectional dependence was observed for the relationships between Violin II and Cello and Violin II and Viola. Own-reported dependencies after the performances reflected these measured dependencies more closely than dependencies of players reported by the other players, which instead showed more typical leader–follower patterns in which Violin I leads. On the other hand, primary leadership from Violin I was observed in an analysis of the bow speed characteristics preceding the first tone onset. The anticipatory movement of Violin I set the tempo of the excerpt. Taken together the results show a more complex and differentiated pattern of dependencies than expected from a traditional role division of leadership suggesting several avenues for further research.

Learning expressive percussion performance under different visual feedback conditions

A study was conducted to test the effect of two different forms of real-time visual feedback on expressive percussion performance. Conservatory percussion students performed imitations of recorded teacher performances while receiving either high-level feedback on the expressive style of their performances, low-level feedback on the timing and dynamics of the performed notes, or no feedback. The high-level feedback was based on a Bayesian analysis of the performances, while the low-level feedback was based on the raw participant timing and dynamics data. Results indicated that neither form of feedback led to significantly smaller timing and dynamics errors. However, high-level feedback did lead to a higher proficiency in imitating the expressive style of the target performances, as indicated by a probabilistic measure of expressive style. We conclude that, while potentially disruptive to timing processes involved in music performance due to extraneous cognitive load, high-level visual feedback can improve participant imitations of expressive performance features.

Predicting the similarity between expressive performances of music from measurements of tempo and dynamics.

Measurements of tempo and dynamics from audio files or MIDI data are frequently used to get insight into a performers contribution to music. The measured variations in tempo and dynamics are often represented in different formats by different authors. Few systematic comparisons have been made between these representations. Moreover, it is unknown what data representation comes closest to subjective perception. The reported study tests the perceptual validity of existing data representations by comparing their ability to explain the subjective similarity between pairs of performances. In two experiments, 40 participants rated the similarity between performances of a Chopin prelude and a Mozart sonata. Models based on different representations of the tempo and dynamics of the performances were fitted to these similarity ratings. The results favor other data representations of performances than generally used, and imply that comparisons between performances are made perceptually in a different way than often assumed. For example, the best fit was obtained with models based on absolute tempo and absolute tempo times loudness, while conventional models based on normalized variations, or on correlations between tempo profiles and loudness profiles, did not explain the similarity ratings well.

Perception of music performance on historical and modern commercial recordings

Performing styles as well as recording styles have changed considerably within the 20th century. To what extent do the age of a recording, the unfamiliarity with performing style, and the quality of a reproduction of a recording systematically influence how we perceive performances on record? Four exploratory experiments were run to formulate an answer to this question. Each experiment examined a different aspect of the perception of performance, including judgments of quality, perceived emotion, and dynamics. Fragments from Die junge Nonne sung by famous singers from the start, middle, and second half of the 20th century were presented in a noisy and clean version to musically trained participants. The results show independence of perception of emotional activity from recording date, strong dependence of perceived quality and emotional impact on recording date, and only limited effects of reproduction quality. Standards have clearly changed, which influence judgments of quality and age. Additionally, changes restrict the communication between early recorded performers and modern listeners to some extent as shown by systematically smaller variations in communicated dynamics and emotional valence for older recordings.

Active Drumming Experience Increases Infants' Sensitivity to Audiovisual Synchrony during Observed Drumming Actions

In the current study, we examined the role of active experience on sensitivity to multisensory synchrony in six-month-old infants in a musical context. In the first of two experiments, we trained infants to produce a novel multimodal effect (i.e., a drum beat) and assessed the effects of this training, relative to no training, on their later perception of the synchrony between audio and visual presentation of the drumming action. In a second experiment, we then contrasted this active experience with the observation of drumming in order to test whether observation of the audiovisual effect was as effective for sensitivity to multimodal synchrony as active experience. Our results indicated that active experience provided a unique benefit above and beyond observational experience, providing insights on the embodied roots of (early) music perception and cognition.

Creativity, collaboration and development in Jeremy Thurlow's Ouija for Peter Sheppard Skaerved

ABSTRACT This article documents and analyses a creative collaboration between the composer Jeremy Thurlow and the violinist Peter Sheppard Skærved in the production of Ouija, a work for solo violin and laptop computer. The article situates the account of this creative process within recent literature on distributed and collaborative creativity, and focuses on three aspects of the project: verbal interaction between the two musicians, analysed in terms of ‘creative-talk’ and ‘face-talk’, and the relationship between immediate and more contextual concerns (‘inside/outside the room’); a quantitative analysis of changes in the musical material, focusing on timing; and a qualitative analysis of the role of the violinist’s embodied and instrumental engagement with the music. The article discusses the findings in relation to forwards-orientated (process) and backwards-orientated (product) conceptions of creativity, the operation of different social components in creative collaboration and the relationship between craft, history and embodiment.