Rumi Hiraga

Music Puzzle: An Audio-Based Computer Game That Inspires to Train Listening Abilities

The Music Puzzle is a computer game for tablets and smartphones using sounds for the gameplay. Just like an original picture is reconstructed from pieces with jigsaw puzzle, an original sound is reconstructed from musical segments with Music Puzzle. Each segment is distorted by shifting the pitch and equalization. To finish the game, the user listens to each segment visualized as pieces on the screen, reorders them, and corrects their pitch and equalization. The game has a possibility for deaf and hard of hearing people to improve their residual hearing ability since the observation shows their concentrating the game with sounds and preference for music.

Effect of visual representation in recognizing emotion expressed in a musical performance

We have conducted a series of experiment to investigate the potential of music communication based on emotion for hearing-impaired people and people with hearing abilities. The purpose of the experiments is to design a performance assistance system to enable hearing-impaired people to enjoy taking part in a music ensemble. In this paper, we describe an experiment on recognizing emotion in two media: moving images of a professional percussionist giving an improvised performance and the sounds extracted from the performance. The experiment was designed to investigate whether the image of a performance is important in recognizing emotion in the performance. The result showed that there was no significant difference in recognition of emotion between the two media. Our results were consistent with those of previous experiments: (1) there was no significant difference in recognizing emotion between hearing-impaired people and people with hearing abilities, and (2) ldquofearrdquo was the least recognized of the four emotions.

Is visual information useful for music communication

We have conducted a series of experiments on recognizing emotions conveyed on the hearing of musical performances as a means of building a musical performance assistance system with which deaf and hard of hearing people could communicate through musical performance. Until recently, we had assumed that visual information provided simultaneously with musical performance was useful in helping the emotions emitted from music to be recognized, especially if the information conveyed the same emotions as those expressed in the musical performance. Currently we are not sure that this assumption is correct after having gone through experiments we conducted. In this paper, we describe the background of our plan to build a musical performance assistance system and review our past experiments by comparing the emotions recognized by recipients among several types of stimuli. Reviewing the results of our experiments made us unsure that multichannel information that supposedly helps deaf and hard of hearing people to recognize emotions actually works well. Based on a new question we pose about multichannel music information, we describe an experiment that compare the stimuli provided by music types only, music with video sequences, and video sequences only. The results we obtained do not clarify the question that we pose.

Appreciating Harmony-differences between the hearing-impaired, musically inexperienced, and musically experienced.

In this paper, we describe an experiment on the cognition of harmony done by three groups: people who have hearing-impairments, people who have little experience of playing music, and people who have a lot experience of playing music. By modifying the accompaniment key, we prepared five types of harmony for a single melody, provided them to the subjects, and asked for their preferred harmony. Their preferences for the accompaniments differed. People with music experience strongly preferred the original accompaniment, while hearing-impaired people cared little about whatever the harmony was. Two hearing groups preferences did not differ much for the original and other accompaniments, but people with little musical experience had the preferences they lay between those of the other two subject groups. Though many hearing-impaired people are fond of music, this experiment on the cognition of harmony did not find that they appreciate musical harmony. The results of the different preferences of people with a little and a lot of musical experience suggest their preferences may differ between in other aspects of music.

A Subjective Evaluation of Music Beat Recognition with Different Timbres by Hearing-Impaired People

For hearing-impaired people (HIs) to enjoy music more, one of the authors, who is profoundly deaf but enthusiastic about music, feels the choice of instrumental sounds is important. We conducted an experiment of beat tapping with 20 musical instruments involving one group of HIs using hearing aids and another group of HIs using cochlear implants. From the analysis of a subjective evaluation about each instrumental sound, we found the two groups differed in their perceptions of sounds. One reason for this difference seems to be the envelope of the sounds. Thus, we are going to analyze sounds to pursue suitable instrumental sounds for HIs.

A Preliminary Observation on the Effect of Visual Information in Learning Environmental Sounds for Deaf and Hard of Hearing People

Environmental sounds (ES) give us fundamental information about the world around us and are essential to our safety as well as our perception of everyday life. With improvements in noise reduction techniques, hearing aids and cochlear implants show better performance in the understanding of speech by deaf and hard of hearing (DHH) people. On the other hand, DHH children have little chance to learn ES. We are developing an ES learning system with visual information so that those with severely handicapped children could enjoy learning ES. In this paper, we describe a preliminary observation of including visual information as well as sound in learning ES.

Discrimination and identification of environmental sounds among people with hearing impairment

We report that people with hearing impairment (HI) can better discriminate environmental sounds than identifying the same sounds. In order to investigate the perception/recognition of environmental sounds by people with HI, researchers often choose identification task as a measurement tool. However, since identification ability is developed upon discrimination ability, the discrimination ability should be studied too. We used 9 environmental sounds as stimuli for both discrimination and identification tasks, and 16 participants with HI joined the experiment. In the discrimination task, the participants listened to the paired short stimuli, and reported the perceived difference. In the identification task, the participants listened to each stimulus and reported the name of the sound source. The order of discrimination and identification tasks is counterbalanced among the participants, and we also tested 10 people with normal hearing for comparison. The results showed a clear contrast between the discrimina...

Music Perception of hearing-Impaired Persons with Focus on One Test Subject

We previously investigated how hearing-impaired people perceive music in several types of musical experiments. By following-up on the results of some of these experiments with a single test subject Sd, we found that the ability of the subject to perceive music was high and that she appreciated music in a way that was different from that of hearing people. In this paper, we describe three musical experiments with hearing-impaired subjects, their results, and Sds music perception through the experiments. The three experiments involved the Music Puzzle game, the appreciation of harmony, and tempo perception. Music Puzzle is a music game we made that is played on a tablet and is intended to be used by hearing-impaired persons as a serious game with which they can improve their hearing ability by continuously playing it. The experiment on appreciation of harmony was conducted with three subject groups, and the result showed that experience with music affected the appreciation of music accompanied with the tonal code. Tempo perception was investigated with a simple game in which the subjects tapped along with the tempo of the music. By examining the subjects hearing acuity in standard medical hearing tests and crossing over the results of these musical experiments, we observed that hearing acuity is not necessarily related to the perception and understanding of music.

The effect of musical experience on rhythm perception for hearing-impaired undergraduates

This paper describes an experiment on the rhythm perception by hearing-impaired people using three musical contents: vocal-based songs, vocal-only songs, and instrumental music. The real world consists of many kinds of sounds, not only human speech but also music, songs, and environmental noises. Although typical listening ability exercises these days are primarily optimized for perception of speech rather than other sounds, we assume that musical features can help hearing-impaired people to segregate information sounds and the other sounds. Our project is intended to create a musical training that enhances hearing-impaired peoples listening ability. In this study, to begin the project, we investigated how the hearing-impaired people recognize the rhythmical features in listening to the songs. Nine hearing-impaired undergraduates participated in the tapping experiment. Experimental results showed that musical experience significantly improved their rhythm perception.

Emotion-intending drawings

We are building a musical performance assistance system that uses visual information to supplement the musical communication among deaf and hard-of-hearing people playing music together, enabling them to enjoy music communication playing music representing a certain emotion. We previously collected drawings for which the drawer had an emotion in mind in order to improve the communication in our system, which presents them to the players to enable them to focus on expressing a specific emotion. We thus need to better understand the properties of images that relate to an emotion. After computing the principal components of the image properties, we clustered the drawings into groups representing one of four emotions. We also asked viewers to categorize them on the basis of the emotion they elicited in the viewer. The results showed that the viewers based their judgment on the specific meaning of the drawing rather than the shape, if the shape of drawings was not abstract. The drawings intended to represent fear were well grouped by clustering, while the viewers categorized them as either fear or sadness. The results imply that we can add new drawings with intended emotions to our system as long as their image properties meet our analysis.