Hirohiko Sagawa

A method for recognizing a sequence of sign language words represented in a Japanese sign language sentence

To automatically interpret Japanese sign language (JSL), the recognition of signed words must be more accurate and the effects of extraneous gestures removed. We describe the parameters and the algorithms used to accomplish this. We experimented with 200 JSL sentences and demonstrated that recognition performance could be considerably improved.

Open-vocabulary keyword detection from super-large scale speech database

This paper presents our recent attempt to make a super-large scale spoken-term detection system, which can detect any keyword uttered in a 2,000-hour speech database within a few seconds. There are three problems to achieve such a system. The system must be able to detect out-of-vocabulary (OOV) terms (OOV problem). The system has to respond to the user quickly without sacrificing search accuracy (search speed and accuracy problem). The pre-stored index database should be sufficiently small (index size problem). We introduced a phoneme-based search method to detect the OOV terms, and combined it with the LVCSR-based method. To search for a keyword from large-scale speech databases accurately and quickly, we introduced a multistage rescoring strategy which uses several search methods to reduce the search space in a stepwise fashion. Furthermore, we constructed an out-of-vocabulary/in-vocabulary region classifier, which allows us to reduce the size of the index database for OOVs. We describe the prototype system and present some evaluation results.

Description and recognition methods for sign language based on gesture components

Sign language gestures are inflected in accordance with the context. To recognize such sign language properly, the structure of sign kmgoage must be made clear. It is well known that the structure of sign Iangoage is represented as a combination of basic components of gestures. Sign language can be recognized by using such components. In this paper, a format to describe sign language gestures and a method to recognize the meaning of the gesture based on the components of gestures are discussed.

A teaching system of Japanese sign language using sign language recognition and generation

In recent years, the number of sign language learners is increasing in Japan. And there are many teaching materials of sign language such as textbooks, videotapes and software for PCs. However, these teaching materials have several problems that learners cannot study sign language sufficiently because the learners can mainly study manual gestures, cannot change the direction to see signed gestures, and cannot check their signed gestures by themselves. We developed a sign language teaching system applying sign language recognition and sign language generation technologies to solve these problems and realize effective sign language study. Further, we carried out an evaluation experiment by several users and confirmed that the developed system was effective.

A Vision-Based Method for Recognizing Non-manual Information in Japanese Sign Language

This paper describes a vision-based method for recognizing the nonmanual information in Japanese Sign Language (JSL). This new modality information provides grammatical constraints useful for JSL word segmentation and interpretation. Our attention is focused on head motion, the most dominant non-manual information in JSL. We designed an interactive color-modeling scheme for robust face detection. Two video cameras are vertically arranged to take the frontal and profile image of the JSL user, and head motions are classified into eleven patterns. Moment-based feature and statistical motion feature are adopted to represent these motion patterns. Classification of the motion features is performed with linear discrimant analysis method. Initial experimental results show that the method has good recognition rate and can be realized in real-time.

A Method for Analyzing Spatial Relationships Between Words in Sign Language Recognition

There are expressions using spatial relationships in sign language that are called directional verbs. To understand a sign-language sentence that includes a directional verb, it is necessary to analyze the spatial relationship between the recognized sign-language words and to find the proper combination of a directional verb and the sign-language words related to it. In this paper, we propose an analysis method for evaluatingthe spatial relationship between a directional verb and other sign-language words according to the distribution of the parameters representing the spatial relationship.

Pattern Recognition and Synthesis for a Sign Language Translation System

Abstract Sign language is one means of communication for hearing-impaired people. Words and sentences in sign language are mainly represented by hand gestures. In this report, we describe a sign language translation system which we are developing. The system translates Japanese Sign Language into Japanese text and vice versa . In this system, hand shapes and positions are provided by hand-based input in the form of time series. Then, hand gestures are recognized and translated into Japanese sentences using continuous DP (dynamic programming) matching. Japanese text is translated into sign language by generating three-dimensional computer-graphic animation of sign language gestures.

Development of an information kiosk with a sign language recognition system

An information kiosk with a JSL (Japanese sign language) recognition system that allows hearing-impaired people to easily search for various kinds of information and services was tested in a government office. This kiosk system was favorably received by most users.

Methods to describe and recognize sign language based on gesture components represented by symbols and numerical values

Sign-language gestures inflect according to the context. To recognize such sign language properly, the structure of sign language must be made clear. It is well known that the structure of sign language is represented as a combination of gesture components. In this paper, methods for the description and recognition of sign-language gestures based on the gesture components are discussed. In these methods, which we have developed, a sign-language gesture is recognized by integrating the recognized gesture components according to the structure of the gesture. The results of an experiment on recognizing sign-language gestures are also examined and it is shown that the developed methods are effective.

Predicting Evidence of Understanding by Monitoring User's Task Manipulation in Multimodal Conversations

The aim of this paper is to develop animated agents that can control multimodal instruction dialogues by monitoring users behaviors. First, this paper reports on our Wizard-of-Oz experiments, and then, using the collected corpus, proposes a probabilistic model of fine-grained timing dependencies among multimodal communication behaviors: speech, gestures, and mouse manipulations. A preliminary evaluation revealed that our model can predict a instructors grounding judgment and a listeners successful mouse manipulation quite accurately, suggesting that the model is useful in estimating the users understanding, and can be applied to determining the agents next action.