Tsuneshi Isomura

Emotional communication in finger braille

We describe analyses of the features of emotions (neutral, joy, sadness, and anger) expressed by Finger Braille interpreters and subsequently examine the effectiveness of emotional expression and emotional communication between people unskilled in Finger Braille. The goal is to develop a Finger Braille system to teach emotional expression and a system to recognize emotion. The results indicate the following features of emotional expression by interpreters. The durations of the code of joy were significantly shorter than the durations of the other emotions, the durations of the code of sadness were significantly longer, and the finger loads of anger were significantly larger. The features of emotional expression by unskilled subjects were very similar to those of the interpreters, and the coincidence ratio of emotional communication was 75.1%. Therefore, it was confirmed that people unskilled in Finger Braille can express and communicate emotions using this communication medium.

Emotion Recognition of Finger Braille

Finger Braille is one of tactual communication media of deafblind people. Deafblind people who are skilled in Finger Braille can catch up with speech conversation and express various emotions. Because there are small non-disabled people who are skilled in Finger Braille, deafblind people communicate only with interpreters. Objective of this study is development of a Finger Braille supporting device which assists not only verbal communication but also non-verbal (emotional) communication between deafblind people and non-disabled people who are not skilled in Finger Braille. In this paper, to develop emotion recognition system, we analyzed features of emotional expression (Neutral, Joy, Sadness and Anger) and derived an algorithm of emotion recognition using accelerometers worn by receiver. According to the results of discriminant analysis, it was considered that emotion recognition using the discriminant functions by standardized data and the average of posterior probabilities in a sentence was possible and independent of sentences.

Finger Braille Teaching System for People who Communicate with Deafblind People

Finger Braille is one of tactual communication media of deafblind people. In finger Braille, index finger, middle finger and ring finger of both hands are likened to keys of a Braille typewriter. A sender dots Braille code on the fingers of a receiver like whether he/she does the type of the Braille typewriter. Then the receiver recognizes the Braille code. Deafblind people who are skilled in finger Braille can catch up with speech conversation and express various emotions. Because there are small non-disabled people who are skilled in finger Braille, deafblind people communicate only with interpreters. In this paper, we developed a finger Braille teaching system and designed a teaching interface which taught clauses explicitly. The teaching system recognized non-disabled peoples speech and converted to Braille code. By parsing the Braille code, the teaching system retrieved clause information and segmented the Braille code into clauses. Then the dot pattern of the Braille code was displayed. By observing the dot pattern, non-disabled people dotted Finger Braille to deafblind people. An evaluation experiment between a blind person who was skilled in Finger Braille and two nondisabled people who were non-skilled in Finger Braille was carried out. The results showed that the fundamental functions (speech recognition, conversion to Braille code and clause segmentation) were practicable; the nondisabled senders could dot finger Braille accurately and communicate with the blind receiver directly. Therefore it was considered that the teaching system was effective.

Finger Braille Teaching System

Deafblindness is a condition that combines varying degrees of both hearing and visual impairment. All deafblind people experience problems with communication, access to information, and mobility. Deafblind people use many different communication media, depending on the age of onset of deafness and blindness and the available resources. For example, “deafblind manual alphabet” is a method of spelling out words onto a deafblind persons hand. Each letter is denoted by a particular sign or place on the hand. “Block” is a manual form of communication where words are spelled out on the palm of the deafblind persons hand. “Hands on signing” is based on sign language. With this system, the deafblind person follows the signs by placing his hands over those of the signer and feeling the signs formed. “Yubi-Tenji” (Finger Braille) is one of the tactual communication media developed by Satoshi Fukushima in Japan (see Fig. 1). In Finger Braille, the index finger, middle finger and ring finger of both hands function like the keys of a Braille typewriter. A sender dots Braille code on the fingers of a receiver as if typing on a Braille typewriter. The receiver is assumed to be able to recognize the Braille code. Deafblind people who are skilled in Finger Braille can understand speech conversation and express various emotions because of the prosody (intonation) of Finger Braille (Fukushima, 1997). Because there is such a small number of non-disabled people who are skilled in Finger Braille, deafblind people communicate only through an interpreter.

Finger Braille Recognition System

The Ministry of Health, Labour and Welfare of Japan estimates that there are nearly 22,000 deafblind people in Japan (2006). Communication is one of their largest barriers to independent living and participation. Deafblind people use many different communication media, depending on the age of onset of deafness and blindness and the available resources. “Yubi-Tenji” (Finger Braille) is one of the tactual communication media utilized by deafblind individuals (see Fig. 1). In two-handed Finger Braille, the sender’s index finger, middle finger and ring finger of both hands function like the keys of a Braille typewriter. The sender dots Braille code on the fingers of the receiver. The receiver is assumed to recognize the Braille code. In one-handed Finger Braille, the sender dots the left part of Braille code on the distal interphalangeal (DIP) joints of the three fingers of the receiver, and then the sender dots the right part of Braille code on the proximal interphalangeal (PIP) joints. Deafblind people who are skilled in Finger Braille can communicate words and express various emotions because of the prosody (intonation) of Finger Braille (Fukushima, 1997). Because there is such a small number of non-disabled people who are skilled in Finger Braille, deafblind people communicate only through an interpreter. Thus, the participation of deafblind people is greatly restricted.

Teaching of Emotional Expression Using Finger Braille

Finger Braille is one of the tactual communication media utilized by deaf blind people. Deaf blind people who are skilled in Finger Braille can understand the speech conversation and express various emotions. Because there is such a small number of non-disabled people who are skilled in Finger Braille, deaf blind people communicate only through an interpreter. The objective of this study is development of a Finger Braille support device which assists not only verbal communication but also non-verbal (emotional) communication between deaf blind people and non-disabled people who are not skilled in Finger Braille. In this paper, to develop an emotion teaching system, we developed a teaching method of emotional expression (neutral, joy, sadness and anger) using the sentences about the impression of emotional expression. The results of the evaluation experiment showed that the non-disabled subjects could change the dotting durations and finger load by the teaching of emotional expression.

Analysis of Emotional Expression of Finger Braille

Finger Braille is one of tactual communication media of deafblind people. In two-handed Finger Braille, index finger, middle finger and ring finger of both hands are likened to keys of a Braille typewriter. A sender dots Braille code on the fingers of a receiver like whether he/she does the type of the Braille typewriter. Then the receiver recognizes the Braille code. In one-handed Finger Braille, the sender dots the left column of Braille code on the DIP joints of three fingers of the receiver, and then the sender dots the right column of Braille code on the PIP joints of them. Deafblind people who are skilled in Finger Braille can catch up with speech conversation and express various emotions. Because there are small nondisabled people who are skilled in Finger Braille, deafblind people communicate only with interpreters. Objective of this study is development of a Finger Braille supporting device which assists not only verbal communication but also nonverbal (emotional) communication between deafblind people and non-disabled people who are not skilled in Finger Braille. In this paper, an experiment of emotional expression of Finger Braille was carried out. And we analyzed features of emotional expression of Finger Braille and discussed algorithm of emotion recognition. The features of emotional expression were: (1) the durations of code of Joy were particularly shorter than other emotions; (2) the durations of code of Sadness were particularly longer than other emotions; (3) the finger loads of Anger were particularly bigger than other emotions. To develop emotion recognition system, we discussed algorithm of emotion recognition using the accelerometers worn by tester. According to the results of discriminant analysis, it was considered that emotion recognition using the discriminant functions and the average of posterior probabilities in a sentence was possible.

Emotion Feature Vector of Finger Braille

Finger Braille is one of tactual communication media of deafblind people. Deafblind people who are skilled in Finger Braille can catch up with speech conversation and express various emotions. Because there are small non-disabled people who are skilled in Finger Braille, deafblind people communicate only with interpreters. Objective of this study is development of a Finger Braille supporting device which assists not only verbal communication but also non-verbal (emotional) communication between deafblind people and non-disabled people who are not skilled in Finger Braille. In this paper, to develop evaluation method of emotional expression by non-disabled people, we defined an emotional feature vector of Finger Braille and calculated distances of the emotion feature vectors as dissimilarities of them. As the results of experiment by non-disabled people, it was considered that the emotion feature vector was well characterized emotional expression and could be applied to evaluation of emotional expression.

Finger Braille recognition system for people who communicate with deafblind people

Finger Braille is one of tactual communication media of deafblind people. In one-handed finger Braille, a sender dots the left part of Braille code on the DIP joints of index, middle and ring fingers of a receiver, and then the sender dots the right part of Braille code on the PIP joints of them. Then the receiver recognizes the Braille code. Deafblind people who are skilled in Finger Braille can catch up with speech conversation and express various emotions. Because there are small non-disabled people who are skilled in Finger Braille, deafblind people communicate only with interpreters. In this paper, we developed a finger Braille recognition system. And evaluation experiments of recognition of monosyllables and sentences were carried out. As the results of the evaluation experiment of recognition of monosyllables, the accuracy of recognition of dotted fingers was 92.9% and the recognition system was independent of dotted strength, dotted position and receiver. The accuracy of recognition of dotted positions was 81.9% and the recognition system could recognize dotted positions if the receivers hand shaped a natural longitudinal arch on the desk. As the results of the evaluation experiment of recognition of sentences, the accuracy of recognition of dotted fingers by dotting was 89.7% and the accuracy of recognition of dotted positions by dotting was 92.3%. Therefore the recognition system could recognize the sentences accurately when the interpreter dotted clearly. And the receiver should re-set his/her hand on the desk when he/she noticed cut-down of accuracy of recognition.

Finger Braille Recognition System for Non-disabled People who Communicate with Deafblind People

Finger Braille is one of tactual communication medium of deafblind people. In two-handed Finger Braille, index finger, middle finger and ring finger of both hands are likened to keys of a Braille typewriter. A sender dots Braille code on the fingers of a receiver like whether he/she does the type of the Braille typewriter. Then the receiver recognizes the Braille code. Deafblind people who are skilled in Finger Braille can catch up with speech conversation and express various emotions. Because there are small non-disabled people who are skilled in Finger Braille, deafblind people communicate only with interpreters. In this study, we note one-handed Finger Braille. In one-handed Finger Braille, the sender dots the left column of Braille code on the DIP joints of three fingers of the receiver, and then the sender dots the right column of Braille code on the PIP joints of them. In this paper, we developed the Finger Braille Recognition System, which recognized deafblind people’s one-handed Finger Braille and converted to speech for non-disabled people. Non-disabled people wore small piezoelectric accelerometers on the base of index finger, middle finger and ring finger. The Recognition System detected the accelerations by dotting and recognized which fingers were dotted. Next, by parsing the recognized Braille codes, the Recognition System converted to a Japanese text. Finally, the Recognition System synthesized speech of the Japanese text. Because each accelerometer detected the acceleration by dotting of the mounted finger (self dotting) and the acceleration by dotting of the other fingers (cross talk). The Recognition System was necessary to recognize the acceleration by self dotting. We analyzed the accelerations dotted by an interpreter and derived the algorithm of recognition of dotted fingers. Estimated accuracy of recognition was 93.4%. Therefore it was considered that the algorithm was effective.