R. Benjamin Knapp

Multimodal interaction in music using the Electromyogram and relative position sensing

This paper describes a technique of multimodal, multichannel control of electronic musical devices using two control methodologies, the Electromyogram (EMG) and relative position sensing. Requirements for the application of multimodal interaction theory in the musical domain are discussed. We introduce the concept of bidirectional complementarity to characterize the relationship between the component sensing technologies. Each control can be used independently, but together they are mutually complementary. This reveals a fundamental difference from orthogonal systems. The creation of a concert piece based on this system is given as example.

Physiological Signals and Their Use in Augmenting Emotion Recognition for Human-Machine Interaction

In this chapter we introduce the concept of using physiological signals as an indicator of emotional state. We review the ambulatory techniques for physiological measurement of the autonomic and central nervous system as they might be used in human–machine interaction. A brief history of using human physiology in HCI leads to a discussion of the state of the art of multimodal pattern recognition of physiological signals. The overarching question of whether results obtained in a laboratory can be applied to ecological HCI remains unanswered.

An integrated home-based self-management system to support the wellbeing of older adults

With an ageing population and the constant need towards improving the quality of life for older people in our socie- ty, there comes an urgent challenge to support people where they live in an environment that adapts to their needs as they age. While much research on ubiquitous sensor systems and telehealth devices focuses on this need, many of these solutions operate at less than full capacity, and with little scope at present to assess everyday aspects of wellbeing. They focus on detecting sud- den critical physiological and behavioural changes and offer few mechanisms to support preventative actions. The challenge of predicting changes and prompting positive preventative intervention measures, aiding the avoidance of severe physical or men- tal harm, has not adequately been addressed. This paper discusses our experiences of designing, deploying and testing an inte- grated home-based ambient assisted living (AAL) system for older adults, consisting of ambient monitoring, behaviour recog- nition and feedback to support self-management of wellness, in addition to providing feedback on home security and home energy. It offers a complete system overview of an AAL solution in smart environments and discusses our lessons learned with the goal of assisting other researchers in the field in designing and deploying similar environments.

Measurement of motion and emotion during musical performance

This paper describes the use of physiological and kinematic sensors for the direct measurement of physical gesture and emotional changes in live musical performance. Initial studies on the measurement of performer and audience emotional state in controlled environments serve as the foundation for three pieces using the BioMuse system in live performance. By using both motion and emotion to control sound generation, the concept of integral music control has been achieved.

Díamair: composing for choir and integral music controller

In this paper, we describe the composition of a piece for choir and Integral Music Controller. We focus more on the aesthetic, conceptual, and practical aspects of the interface and less on the technological details. We especially stress the influence that the designed interface poses on the compositional process and how we approach the expressive organisation of musical materials during the composition of the piece, as well as the addition of nuances (personal real-time expression) by the musicians at performance time.

Form, function and performances in a musical instrument MAKErs camp

This experience report presents the planning, execution and results of an exploratory 5-day musical instrument MAKErs camp for K-12 students. Students used various hardware sensors, a graphical programming environment and different physical prototyping techniques to create musical instruments. The design of a musical instrument introduces students to the full spectrum of the design process including form factor and function. Throughout the camp, students shared and performed in front of their peers to gain feedback as they iterated through the design of a musical instrument.

Fusality: an open framework for cross-platform mirror world installations

As computing and displays become more pervasive and wireless networks are increasing the connections between people and things, humans inhabit both digital and physical realities. In this paper we describe our prototype Mirror Worlds framework, which is designed to fuse these realities: Fusality. Our goal for Fusality is to support innovative research and exhibitions in presence and collaboration, sensors and smart buildings and mixed reality in applications from engineering to art. By fusing live sensor data from the building and its occupants with online 3D environments and participants, we demonstrate a first-principles approach to online multi-entity messaging communication. This demonstration shows how the variety of Mirror Worlds clients can be supported through the open Web architecture. These technologies enable new possibilities for collaboration as well as directions for interoperability. Finally, we layout out our research agenda for the framework and discuss its transformative applications.

Towards incorporating affective feedback into context-aware intelligent environments

Determining the relevance of services from intelligent environments is a critical step in implementing a reliable context-aware ambient intelligent system. Designing the provision of explicit indications to the system is effective in communicating this relevance, however, such explicit indications come at the cost of users cognitive resources. In this work, we strive to create a novel pathway of implicit communication between the user and their ambient intelligence by employing users stress as a feedback pathway to the intelligent system. In addition, following a few very recent works, we propose using proven laboratory stressors to collect ground truth data for stressed states. We present results from a preliminary pilot study which shows promise for creating this implicit channel of communication as well as proves the feasibility of using laboratory stressors as a reliable method of ground truth collection for stressed states.

BioTools: A Biosignal Toolbox for Composers and Performers

In this paper, we present the current state of BioTools, an ongoing project to implement a modular hardware and software toolbox for composers and performers, which allows fast deployment of biosignal monitoring and measuring systems for musical applications. We discuss the motivations for this work and additionally three examples are shown of how this set of tools and the compositional strategies were used in the pieces Diamairfor choir and physiological sensors, Out of Time, a project in which BioTools was used to record and analyse biosignals for their later use to inspire and aid in composition, and Carne, an improvisational piece that uses BioTools modules as the control interface.

Biomuse: Musical performance generated by human bioelectric signals

A new type of musical instrument is described which utilizes bioelectric signals to drive a keyboard synthesizer via standard musical instrument digital interface (MIDI) code. Electrical activity from muscle (EMG), brain (EEG), heart (EKG), and eye (EOG) is detected by small disk electrodes on the skin, analyzed by a digital signal processor, and used to control sound generation from the synthesizer. Signal processing algorithms are designed to extract useful control parameters from the given physiological signals. For instance, detection of amplitude changes in the EMG and spectral changes in the EEG can be used to generate MIDI commands. Applications for the device are discussed, such as dance generated music, musician self‐mixed performances, and music production by the handicapped.