Haruhiro Katayose

Human motion capture by integrating gyroscopes and accelerometers

This paper describes a motion capturing system for human arm motion in real time; directly, simply and precisely by integrating two types of sensor; gyroscopes and accelerometers. A new sensor fusion technique is proposed which enlarges the merits of each sensor and compensates for their deficiencies. Its potential precision has been verified by experiment and simulation.

Sentiment extraction in music

Sentiments are extracted from real performance through three steps: transcription, musical primitive analysis, and music understanding. In the transcription step, the acoustic signal is transformed into notes. Musical primitives such as chord progression, melody, rhythm, and tempo are analyzed from the notes using musical knowledge. Musical primitives are meaningful for the atmosphere of music. Sentiments are extracted by firing the heuristic rules representing the relation between musical primitives and sentiments for music understanding.<<ETX>>

Distinct Inter-Joint Coordination during Fast Alternate Keystrokes in Pianists with Superior Skill

Musical performance requires motor skills to coordinate the movements of multiple joints in the hand and arm over a wide range of tempi. However, it is unclear whether the coordination of movement across joints would differ for musicians with different skill levels and how inter-joint coordination would vary in relation to music tempo. The present study addresses these issues by examining the kinematics and muscular activity of the hand and arm movements of professional and amateur pianists who strike two keys alternately with the thumb and little finger at various tempi. The professionals produced a smaller flexion velocity at the thumb and little finger and greater elbow pronation and supination velocity than did the amateurs. The experts also showed smaller extension angles at the metacarpo-phalangeal joint of the index and middle fingers, which were not being used to strike the keys. Furthermore, muscular activity in the extrinsic finger muscles was smaller for the experts than for the amateurs. These findings indicate that pianists with superior skill reduce the finger muscle load during keystrokes by taking advantage of differences in proximal joint motion and hand postural configuration. With an increase in tempo, the experts showed larger and smaller increases in elbow velocity and finger muscle co-activation, respectively, compared to the amateurs, highlighting skill level-dependent differences in movement strategies for tempo adjustment. Finally, when striking as fast as possible, individual differences in the striking tempo among players were explained by their elbow velocities but not by their digit velocities. These findings suggest that pianists who are capable of faster keystrokes benefit more from proximal joint motion than do pianists who are not capable of faster keystrokes. The distinct movement strategy for tempo adjustment in pianists with superior skill would therefore ensure a wider range of musical expression.

Expression extraction in virtuoso music performances

An approach to music interpretation by computers is discussed. A rule-based music interpretation system is being developed that generates sophisticated performance from a printed music score. The authors describe the function of learning how to play music, which is the most important process in music interpretation. The target to be learned is expression rules and grouping strategy: expression rules are used to convert dynamic marks and motives into concrete performance data, and grouping strategy is used to extract motives from sequences of notes. They are learned from a given virtuoso performance. The delicate control of attack timing and of the duration and strength of the notes is extracted by the music transcription function. The performance rules are learned by investigating how the same or similar musical primitives are played in a performance. As for the grouping strategy, the system analyzes how the player grouped music and registers dominant note sequences to extract motives.<<ETX>>

On Evaluating Systems for Generating Expressive Music Performance: the Rencon Experience

Abstract Research into music generation and into emulating human musical competence has attracted much attention in the field of computer science. In general, the results of academic research should be verified by assessing ‘objective effectiveness’, which is often represented by a ‘recognition ratio’. Although ‘objective effectiveness’ is also a requirement for research in music generation, it is meaningless unless subjective requisites are also satisfied. However, it is not easy for researchers to execute subjective evaluations within their individual endeavours. To address this difficulty within the research area of computer systems for generating expressive music performances, the Performance Rendering Contest (Rencon) was created. This is an international competition in which entrants present computer systems and the performances generated are graded, and has been held in conjunction with related international conferences. This paper presents an overview of Rencon history, highlighting the evaluative motivation of each contest. In addition, we discuss the possibilities of a new scientific research field in which future Rencons may play a role.

v.morish'09: A Morphing-Based Singing Design Interface for Vocal Melodies

This paper describes a singing design method based on morphing, the design and development of an intuitive interface to assist morphing-based singing design. The proposed interface has a function for real-time morphing, based on simple operation with a mouse, and an editor to control the singing features in detail. The user is able to enhance singing voices efficiently by using these two functions. In this paper, we discuss the requirement for an interface to assist in morphing-based singing design, and develope an interface to fulfill the requirement.

Learning performance rules in a music interpretation system

It has been reported that if music is realized as nominally described by the score, a musically unacceptable performance emerges. Music interpretation is necessary to generate a musically acceptable performance. We have been constructing a rule-based music interpretation system that generates sophisticated performance from a printed music score. The whole performance data are given by the product of expression of each dynamic mark explicitly drawn in the score and expression of analyzed motives. This paper describes the function of learning how to play music, which is the most important process in music interpretation. The target to be learned are expression rules and grouping strategy; expression rules are used in order to convert dynamic marks and motives into concrete performance data, and grouping strategy is used in order to extract motives from note sequence. They are learned from the given virtuoso performance. The delicate control of attack-timing, duration and strength of notes are extracted by the music transcription function. The performance rules are learned by investigating how the same or similar musical primitives are played in the performance. As for the grouping strategy, the system analyzes how the player grouped music, and registers dominant note-sequences to extract motives.

Control of multi-joint arm movements for the manipulation of touch in keystroke by expert pianists

BackgroundProduction of a variety of finger-key touches in the piano is essential for expressive musical performance. However, it remains unknown how expert pianists control multi-joint finger and arm movements for manipulating the touch. The present study investigated differences in kinematics and kinetics of the upper-limb movements while expert pianists were depressing a key with two different touches: pressed and struck. The former starts key-depression with the finger-tip contacting the key, whereas the latter involves preparatory arm-lift before striking the key. To determine the effect of individual muscular torque (MUS) as well as non-muscular torques on joint acceleration, we performed a series of inverse and forward dynamics computations.ResultsThe pressed touch showed smaller elbow extension velocity, and larger shoulder and finger flexion velocities during key-depression compared with the struck touch. The former touch also showed smaller elbow extension acceleration directly attributed to the shoulder MUS. In contrast, the shoulder flexion acceleration induced by elbow and wrist MUS was greater for the pressed touch than the struck touch. Towards the goal of producing the target finger-key contact dynamics, the pressed and struck touches effectively took advantage of the distal-to-proximal and proximal-to-distal inter-segmental dynamics, respectively. Furthermore, a psychoacoustic experiment confirmed that a tone elicited by the pressed touch was perceived softer than that by the struck touch.ConclusionThe present findings suggest that manipulation of tone timbre depends on control of inter-segmental dynamics in piano keystroke.

iFP: A Music Interface Using an Expressive Performance Template

This paper describes a performance interface called iFP that enables players to play music as if he or she had the hands of the virtuoso. iFP is a tapping-style musical interface and refers to a pianist’s expressiveness described in a performance template. The paper describes the scheduler that allows a player to mix her/his own intension with the expressiveness in the performance template and the user interfaces. The results of a subjective study suggest that using the expression template and the tapping-style interface contribute to the subject’s joy of playing music. This result is also supported by a brain activation study that was done using near-infrared spectroscopy.

ARMS: a trading card game using AR technology

Trading card games have become very popular among children all over the world. This paper describes an example of enhancing a trading card game involved with Augmented Reality technologies, the players of which can experience monster summoning and battles, as if they are happening on their own arms.