Kenzaburo Miyawaki

A cooking support system for people with higher brain dysfunction

This paper reports a cooking support system for persons with higher brain dysfunction. When a human injured its brain, various brain functions are damaged, such as memory and cognition. This symptom is known as higher brain dysfunction. An effective rehabilitation methods and support systems are essential factors for better Activity of Daily Life (ADL) of the persons with brain dysfunction. However, it is not easy to help the persons, because higher brain dysfunction is invisible and hard to understand. We aim to clarify helpful user interface design for such persons. Two aspects should be considered for the design. The aspects are an analysis of both ability and disability of patients, and customization of interface for a daily life. For the purpose, we focus on cooking, one of the most important ADL. Cooking is complex task because it includes many steps. We designed a multimodal interface of cooking-navigation to show instructions of such a complex task easily. We applied the system for rehabilitation training and a patient could use it nicely.

Social Skills Training Support of Cognitive Dysfunctions by Cooperative Cooking Navigation System

We have an important issue that the people with cognitive dysfunctions should improve social skills for self supporting. This paper notices their fundamental cooking activities and proposes a cooperative cooking navigation system supporting their social skills training. We have econstructed this system to be composed of cooperative behavior expression support and cooperative behavior evaluation support. We evaluate the experimental results of applying our proposed system to patients with cognitive dysfunctions and extract the essential conditions for working this system well.

Group emotion estimation using Bayesian network based on facial expression and prosodic information

Recently, there have been many studies on the activation of group communication, and it is important to easily and stably measure the states of group communication. We focus on group emotion expressed during conversation, and propose a method for estimating group emotion reliably using a Bayesian network based on both face image features and prosodic information. In the proposed method, group emotion is derived from estimation state values of individual emotions in the Bayesian network. The effectiveness of the proposed method was verified by performing experiments involving group conversation.

Adaptive embodied entrainment control and interaction design of the first meeting introducer robot

This paper describes a robotic agent that can promote communication when people first meet. When two people meet for the first time, a communication mediator can be important because people often feel stressed and cannot talk comfortably. Our agent reduces their stress by using embodied entrainment and thus promotes communication. In the research field of embodied entrainment, suitable timing of a nod or back-channel feedback has been discussed, but situations to communicate in are limited. We have developed an embodied entrainment control system that recognizes communication states and adapts to them accordingly. For this, we focus on effective non-verbal information for communication. Using this information, our agent helps a talker and a listener to alternate their roles appropriately. We conducted communication experiments with the agent and confirmed its effectiveness. In the experiments, we used and compared different representation of the agent: an embodied robot agent, a computer graphics agent, and no agent. We report the comparison results and discuss representations for communication agents.

Cooking Support System Using Networked Robots and Sensors

This paper introduces a concept of cooking support system using networked robots and sensors. The system can recognize human activities in a kitchen and offer helpful instructions appropriately. It is based on the integration of embedded sensors and mobile robots, and can monitor cooking activities widely.

Cooking rehabilitation support for self-reliance of cognitive dysfunction patients

Cognitive dysfunction patients often interfere with cooking action and give up it. We have proposed the navigation system which supports cooking according to the grade of cognitive dysfunction, and have conducted the experiments of cognitive rehabilitation program (the cycle of cooking, eating, and reviewing) for verifying the hypothesis of having a possibility that the rehabilitation using the proposed system will promote self-reliance of their daily life. Consequently, for cognitive dysfunction patients, it was verified that their self-efficacy improves and there was also a tendency of improving their cognitive function. In this paper, we notice four basic technologies of supporting this rehabilitation program, i.e., 1) recipe transformation according to the grade of disables, 2) support technology for cooking in parallel, 3) group cooking support for improving social skills, 4) remote cooking rehabilitation, and we discuss the verification results and how there should be any rehabilitation program.

A user adaptive cooking navigation system using ubiquitous sensing environment

This paper introduces a user adaptive cooking navigation system, which can deal with concurrent cooking. The system recognizes the situation of userspsila cooking and offers the helpful guidance by a movie or a text etc. Because the frequent order rearrangement of cooking steps occurs under the concurrent cooking task, the state recognition is difficult. To solve the problem, we propose a user adaptive model of cooking, which is a combination of flow graphs and state-machine-diagrams of UML. We experimented on the system, and confirmed the effectiveness of the model.

Remote cognitive rehabilitation support system for menu and meal preparation

This paper focuses on the preparation and consumption of meals-menu planning, grocery shopping, cooking, and consumption of people with higher brain dysfunction (HBD) as a novel approach to cognitive rehabilitation. A rehab scheme can be implemented using a cooking navigation system that supports menu planning that is tailored to the disability level of patients, yet patients tend to become overly dependent on their therapists when they work together in close proximity. To break this dependence, we propose a novel remote rehab and support system that markedly improves the self-reliance of patients by removing the therapist from the scene to observe the patients from a distance by video. We conducted a pilot rehab trial using the proposed scheme on three subjects with HBD, and the results suggest a significant improvement in cognitive function.

A cooking assistant robot using intuitive onomatopoetic expressions and joint attention

A novel cooking assistance interface is proposed that guides users in the precise timing of cooking actions by means of intuitive onomatopoetic expressions associated with different cooking states and nonverbal movements on the part of a cooking assistant robot such as focusing attention through eye-gazing and gestures. The effectiveness of the scheme is demonstrated using the system to assist subjects in preparing a familiar recipe (pancakes).

Extraction of Mastication in Diet Based on Facial Deformation Pattern Descriptor

In this paper, we describe a method for extraction of mastication from image sequence. Mastication is the first step of eating and is very important. However, people do not need strong mastication muscle because the advancement of cooking and food processing technology makes soft foods. The weak mastication ability of recent human is about to become a serious problem. It will be a risk factor of many diseases. To prevent this, analysis method of mastication is essential. Several studies have been conducted in the research field, but those were not applicable to eating in daily life, because of the various restrictions. We proposed a mastication analysis method using only monocular camera. The key point is Facial Deformation Pattern Descriptor, FDPD which can represent a pattern of facial deformation. By using the FDPD, we could extract mastication from video successfully, and could develop some useful mastication analysis system for healthy eating life.