Shinya Fujie

A conversation robot using head gesture recognition as para-linguistic information

A conversation robot that recognizes users head gestures and uses its results as para-linguistic information is developed. In the conversation, humans exchange linguistic information, which can be obtained by transcription of the utterance, and para-linguistic information, which helps the transmission of linguistic information. Para-linguistic information brings a nuance that cannot be transmitted by linguistic information, and the natural and effective conversation is realized. We recognize users head gestures as the para-linguistic information in the visual channel. We use the optical flow over the head region as the feature and model them using HMM for the recognition. In actual conversation, while the user performs a gesture, the robot may perform a gesture, too. In this situation, the image sequence captured by the camera mounted on the eyes of the robot includes sways caused by the movement of the camera. To solve this problem, we introduced two artifices. One is for the feature extraction: the optical flow of the body area is used to compensate the swayed images. The other is for the probability models: mode-dependent models are prepared by the MLLR model adaptation technique, and the models are switched according to the motion mode of the robot. Experimental results show the effectiveness of these techniques.

Designing communication activation system in group communication

Our community is facing serious problem: aging society. The population of elderly people is now increasing in Japan. Especially over 75 years old people is estimated to be up to 20 million in 2030. We have investigated in one day care centers which are facilities for elderly care. And we realized that communication is needed for its own sake in these facilities and active communication can cure even depression and dementia. Therefore we propose to cope with these problems using a robot as a communication activator in order to improve the effectiveness of group communication. We define group communication as one of the type of communication which is formed by several persons. This time, we focus on a recreation game named ldquoNandoku.rdquo Nandoku is a type of quize which can be described as group communication with a master of ceremony (MC). In this paper, we describe requirement for this system and system design. The system always selects its behavior and target (a participant in the game) to maximize ldquocommunication activeness.rdquo Communication activeness is defined as amount of several subjectspsila(ordinary three: A, B, C) participation, which are calculated with participantspsila face direction using camera information. For instance, if participant A is not fully participating by not making eye contact, the system is expected to select one of the behaviors such as ldquoCan you answer, Mr.A?rdquo to encourage A to participate in the game. We experimented with the system in a daycare center. Our results show subjectspsila participation is totally increased. That offers evidence that the robot can serve a practical role in improving the group communication as a communication activator.

Conversation Robot with the Function of Gaze Recognition

Gaze recognition for conversation robot is realized and its effectiveness is confirmed. In human conversation, in addition to speech information, visual information plays important role. Particularly, gaze direction is a very useful prompt for turn-taking. In the case that the speaker finish his utterance, for example, if he looks at the listener, then he expect the listener to speak. On the other hand, if the speaker does not look at the listener, he tries to keep his turn. Most conventional spoken dialogue systems detect the finish of users turn only by speech recognition. These systems cannot understand the user tries to keep his turn, and they wrongly begin the utterance and block the users remaining utterance. In this study, we implement the gaze recognition using the users image captured by the camera mounted on the eye of the robot and apply the recognition results to decide who should speak next. For gaze recognition, we introduce the sub-image of users eye region extracted with the active appearance model as the feature. Recognition with subspace method using this feature achieved 70% in recognition rate. Finally, the effectiveness of the gaze recognition is confirmed through the subjective experiment. The experiment is performed by the actual conversation between the conversation robot and the subject.

SCHEMA: multi-party interaction-oriented humanoid robot

Most of our daily communication occurs in groups, at school, at home, and at work, so this project proposes a robot that can participate in routine human conversations.

Multi-modal integration for personalized conversation: Towards a humanoid in daily life

Humanoid with spoken language communication ability is proposed and developed. To make humanoid live with people, spoken language communication is fundamental because we use this kind of communication every day. However, due to difficulties of speech recognition itself and implementation on the robot, a robot with such an ability has not been developed. In this study, we propose a robot with the technique implemented to overcome these problems. This proposed system includes three key features, image processing, sound source separation, and turn-taking timing control. Processing image captured with camera mounted on the robotpsilas eyes enables to find and identify whom the robot should talked to. Sound source separation enables distant speech recognition, so that people need no special device, such as head-set microphones. Turn-taking timing control is often lacked in many conventional spoken dialogue system, but this is fundamental because the conversation proceeds in real-time. The effectiveness of these elements as well as the example of conversation are shown in experiments.

An ASM fitting method based on machine learning that provides a robust parameter initialization for AAM fitting

Due to their use of information contained in texture, active appearance models (AAM) generally outperform active shape models (ASM) in terms of fitting accuracy. Although many extensions and improvements over the original AAM have been proposed, on of the main drawbacks of AAMs remains its dependence on good initial model parameters to achieve accurate fitting results. In this paper, we determine the initial model parameters for AAM fitting with ASM fitting, and use machine learning techniques to improve the scope and accuracy of ASM fitting. Combining the precision of AAM fitting with the large radius of convergence of learned ASM fitting improves the results by an order of magnitude, as our empirical evaluation on a database of publicly available benchmark images demonstrates.

Multiparty Conversation Facilitation Strategy Using Combination of Question Answering and Spontaneous Utterances

Based on our analysis of entertaining multiparty conversations, participants tend to obey the cooperative conversational principle, and provide feedback so as to understand the current topic. In addition, they also speak to express their own idea and interest. In this paper, we describe a conversational robot system designed for constantly facilitating conversations with topic tracing capability, which is able to output a combination of answers to questions and spontaneous utterances.

Upper-Body Contour Extraction Using Face and Body Shape Variance Information

We propose a fitting method using a model that integrates face and body shape variance information for upper-body contour extraction. Accurate body-contour extraction is necessary for various applications, such as pose estimation, gesture recognition, and so on. In this study, we regard it as the shape model fitting problem. A model including shape variance information can fit to the contour robustly even in the noisy case. AAMs are one of these models and can fit to a face successfully. It needs appearance information for effective fitting, but it can not be used in our case because appearance of upper-body easily changes by clothes. Instead of intensity image, proposed method uses edge image as appearance information. However, discrimination between a true contour edge of upper-body and other edges is difficult. To solve this problem, we integrate shapes of upper-body and face. It is expected that this integrated model is more robust to edges in clutter background and various locations of the body than a body shape model using only body shape information. We conduct experiments and confirm improvement in accuracy by integration of face and body variance information.

Upper-body contour extraction and tracking using face and body shape variance information

We propose a fitting method using a model that integrates face and body shape variance information for upper-body contour extraction and tracking. Accurate body-contour extraction is necessary for various applications, such as pose estimation, gesture recognition, and so on. In this study, we regard it as the shape model fitting problem. A model including shape variance information can fit to the contour robustly even in the noisy case. AAMs are one of these models and can fit to a face successfully. It needs appearance information for effective fitting, but it can not be used in our case because appearance of upper-body easily changes by clothes. Instead of intensity image, proposed method uses edge image as appearance information. However, discrimination between a true contour edge of upper-body and other edges is difficult. To solve this problem, we integrate shape models of upper body and face. It is expected that this integrated model is more robust to edges in clutter background and various locations of the body than a body shape model using only body shape information. We conduct experiments and confirm improvement in accuracy by integration of face and body variance information.

Realization of rhythmic dialogue on spoken dialogue system using paralinguistic information

Paralinguistic information recognition and its application to spoken dialogue system are discussed. Paralinguistic information brings various useful cues that cannot be obtained from linguistic information. The system may know, for example, who should speak next, how the user really feels, and so on. In this paper, we implement several paralingusitic information recognition systems, such as prosody‐based attitude recognition system, back‐channel feedback generation/recognition system, head gesture recognition system, and so on. Finally, we combine them into the spoken dialogue system on the humanoid robot, and realize a rhythmic dialogue.