Katie Wilkie

What can the language of musicians tell us about music interaction design

It is difficult to create good interaction designs for music software or to substantially improve existing designs. One reason is that music involves diverse and complex concepts, entities, relationships, processes, terminologies, and notations. An open challenge for interaction designers is to find systematic ways of channeling the tacit, specialized knowledge of musicians into designs for intuitive user interfaces that can capably support musically skilled users, without excluding those with less technical musical knowledge. One promising new approach to this challenge involves the application of research from the theory of image schemas and conceptual metaphors This theory has already been applied with some success to analyzing musical concepts, mathematical concepts, general-purpose user-interface design, and sound-generation interaction designs involving simple musical parameters such as tempo, volume, and pitch. We propose that by identifying the conceptual metaphors and image schemas used by musical experts when analyzing an excerpt of music, and then assessing the extent to which these conceptual metaphors are supported by existing music-interaction designs, it is possible to identify areas where the designs do not match musicians’ understanding of the domain concepts. This process provides a principled basis for identifying points at which designs could be improved to better support musicians’ understanding and tasks, and for the provisional identification of possible improvements. We present the results of an analysis of dialogue between three musicians as they discuss an excerpt of music. A methodology for the systematic identification of image schemas and conceptual metaphors is detailed. We report on the use of the results of the dialogue analysis to illuminate the designs of two contrasting examples of music software. Areas in which the designs might be made more intuitive are identified, and corresponding suggestions for improvements are outlined. The present research appears to be the first to investigate the potential of conceptual metaphor theory for investigating and improving music-interaction designs that deal with complex musical concepts, such as harmonic progressions, modulation, and voice leading.

Music and Human-Computer Interaction

This agenda-setting book presents state of the art research in Music and Human-Computer Interaction (also known as Music Interaction). Music Interaction research is at an exciting and formative stage. Topics discussed include interactive music systems, digital and virtual musical instruments, theories, methodologies and technologies for Music Interaction. Musical activities covered include composition, performance, improvisation, analysis, live coding, and collaborative music making. Innovative approaches to existing musical activities are explored, as well as tools that make new kinds of musical activity possible. Music and Human-Computer Interaction is stimulating reading for professionals and enthusiasts alike: researchers, musicians, interactive music system designers, music software developers, educators, and those seeking deeper involvement in music interaction. It presents the very latest research, discusses fundamental ideas, and identifies key issues and directions for future work.

Towards a Participatory Approach for Interaction Design Based on Conceptual Metaphor Theory: A Case Study from Music Interaction

“Music Interaction” is the term for interaction design within the domain of music. In areas such as music, the ability to engage effectively in certain activities tends to be restricted to those who have acquired detailed knowledge of domain-specific theories, terminologies, concepts or processes. It can be challenging to design or enhance user interfaces for software able to support novices in these kinds of musical activities. One promising approach to this challenge involves translating musicians’ implicit domain knowledge into patterns known as conceptual metaphors, which are metaphorical extensions of recurring patterns of embodied experience applied to abstract domains, and using this information to inform interaction designs for music. This approach has been applied experimentally with some success to designing user interfaces. However, to the best of our knowledge, this present work is the first to consider in detail the use of Conceptual Metaphor Theory as a key component of a participatory design process. In this chapter we present a participatory approach to Music Interaction design based on the principles of Conceptual Metaphor Theory. We posit that such an approach will facilitate the development of innovative and intuitive interaction designs for both novices and experts alike.

Whole Body Interaction in Abstract Domains

There is little dispute that Whole Body Interaction is a good fit of interaction style for some categories of application domain, such as the motion capture of gestures for computer games and virtual physical sports. This reflects the observation that in such applications the mapping between user gesture and the desired effect is, broadly speaking, the identity function. For more abstract application areas such as mathematics, programming and musical harmony, finding appropriate mappings between gesture and effect is less straightforward. The creation of appropriate whole body interaction designs for such abstract application areas remains challenging. However, this is not to argue that whole body interaction is unsuited to abstract domains. Indeed, there is evidence, outlined below, that whole body interaction offers excellent affordances for some highly abstract applications areas.

Music Interaction: Understanding Music and Human-Computer Interaction

We introduce, review and analyse recent research in Music and Human-Computer Interaction (HCI), also known as Music Interaction. After a general overview of the discipline, we analyse the themes and issues raised by the other 15 chapters of this book, each of which presents recent research in this field. The bulk of this chapter is organised as an FAQ. Topics include: the scope of research in Music Interaction; the role of HCI in Music Interaction; and conversely, the role of Music Interaction in HCI. High-level themes include embodied cognition, spatial cognition, evolutionary interaction, gesture, formal language, affective interaction, and methodologies from social science. Musical activities covered include performance, composition, analysis, collaborative music making, and human and machine improvisation. Specific issues include: whether Music Interaction should be easy; what can be learned from the experience of being “in the groove”, and what can be learned from the commitment of musical amateurs. Broader issues include: what Music Interaction can offer traditional instruments and musical activities; what relevance it has for domains unconnected with music; and ways in which Music Interaction can enable entirely new musical activities.

Music and HCI

Music is an evolutionarily deep-rooted, abstract, real-time, complex, non-verbal, social activity. Consequently, interaction design in music can be a valuable source of challenges and new ideas for HCI. This workshop will reflect on the latest research in Music and HCI (Music Interaction for short), with the aim of strengthening the dialogue between the Music Interaction community and the wider HCI community. We will explore recent ideas from Music Interaction that may contribute new perspectives to general HCI practice, and conversely, recent HCI research in non-musical domains with implications for Music Interaction. We will also identify any concerns of Music Interaction that may require unique approaches. Contributors engaged in research in any area of Music Interaction or HCI who would like to contribute to a sustained widening of the dialogue between the distinctive concerns of the Music Interaction community and the wider HCI community will be welcome.