Hirotake Yamazoe

Remote gaze estimation with a single camera based on facial-feature tracking without special calibration actions

We propose a real-time gaze estimation method based on facial-feature tracking using a single video camera that does not require any special user action for calibration. Many gaze estimation methods have been already proposed; however, most conventional gaze tracking algorithms can only be applied to experimental environments due to their complex calibration procedures and lacking of usability. In this paper, we propose a gaze estimation method that can apply to daily-life situations. Gaze directions are determined as 3D vectors connecting both the eyeball and the iris centers. Since the eyeball center and radius cannot be directly observed from images, the geometrical relationship between the eyeball centers and the facial features and eyeball radius (face/eye model) are calculated in advance. Then, the 2D positions of the eyeball centers can be determined by tracking the facial features. While conventional methods require instructing users to perform such special actions as looking at several reference points in the calibration process, the proposed method does not require such special calibration action of users and can be realized by combining 3D eye-model-based gaze estimation and circle-based algorithms for eye-model calibration. Experimental results show that the gaze estimation accuracy of the proposed method is 5° horizontally and 7° vertically. With our proposed method, various application such as gaze-communication robots, gaze-based interactive signboards, etc. that require gaze information in daily-life situations are possible.

Remote and head-motion-free gaze tracking for real environments with automated head-eye model calibrations

We propose a gaze estimation method that substantially relaxes the practical constraints possessed by most conventional methods. Gaze estimation research has a long history, and many systems including some commercial schemes have been proposed. However, the application domain of gaze estimation is still limited (e.g, measurement devices for HCI issues, input devices for VDT works) due to the limitations of such systems. First, users must be close to the system (or must wear it) since most systems employ IR illumination and/or stereo cameras. Second, users are required to perform manual calibrations to get geometrically meaningful data. These limitations prevent applications of the system that capture and utilize useful human gaze information in daily situations. In our method, inspired by a bundled adjustment framework, the parameters of the 3D head-eye model are robustly estimated by minimizing pixel-wise re-projection errors between single-camera input images and eye model projections for multiple frames with adjacently estimated head poses. Since this process runs automatically, users does not need to be aware of it. Using the estimated parameters, 3D head poses and gaze directions for newly observed images can be directly determined with the same error minimization manner. This mechanism enables robust gaze estimation with single-camera-based low resolution images without user-aware preparation tasks (i.e., calibration). Experimental results show the proposed method achieves 6deg accuracy with QVGA (320 times 240) images. The proposed algorithm is free from observation distances. We confirmed that our system works with long-distance observations (10 meters).

Multiple Camera Calibration with Bundled Optimization using Silhouette Geometry Constraints

We propose a method of calibrating multiple camera systems that operates by adjusting the camera parameters and the 3D shape of objects onto silhouette observations. Our method employs frontier points, which are geometrically meaningful points on object surfaces, to determine the geometrical relations among multiple cameras. In contrast to conventional methods, both camera parameters and the 3D positions of the frontier points are jointly estimated by minimizing the 2D projection errors between the 2D projected positions of the frontier points and observed silhouette contours for all cameras. This method makes it possible to obtain accurate calibration results without using any special instruments. Experimental results using real image data demonstrate the effectiveness of our method

Wearable partner agent with anthropomorphic physical contact with awareness of user's clothing and posture

In this paper, we introduce a wearable partner agent, that makes physical contacts corresponding to the users clothing, posture, and detected contexts. Physical contacts are generated by combining haptic stimuli and anthropomorphic motions of the agent. The agent performs two types of the behaviors: a) it notifies the user of a message by patting the users arm and b) it generates emotional expression by strongly enfolding the users arm. Our experimental results demonstrated that haptic communication from the agent increases the intelligibility of the agents messages and familiar impressions of the agent.

GazeRoboard: Gaze-communicative guide system in daily life on stuffed-toy robot with interactive display board

In this paper, we propose a guide system for daily life in semipublic spaces by adopting a gaze-communicative stuffed-toy robot and a gaze-interactive display board. The system provides naturally anthropomorphic guidance through a) gaze-communicative behaviors of the stuffed-toy robot (ldquojoint attentionrdquo and ldquoeye-contact reactionsrdquo) that virtually express its internal mind, b) voice guidance, and c) projection on the board corresponding to the userpsilas gaze orientation. The userpsilas gaze is estimated by our remote gaze-tracking method. The results from both subjective/objective evaluations and demonstration experiments in a semipublic space show i) the holistic operation of the system and ii) the inherent effectiveness of the gaze-communicative guide.

Attractive, Informative, and Communicative Robot System on Guide Plate as an Attendant with Awareness of User’s Gaze

Abstract In this paper, we introduce an interactive guide plate system by adopting a gaze-communicative stuffed-toy robot and a gaze-interactive display board. An attached stuffed-toy robot on the system naturally show anthropomorphic guidance corresponding to the user’s gaze orientation. The guidance is presented through gaze-communicative behaviors of the stuffed-toy robot using joint attention and eye-contact reactions to virtually express its own mind in conjunction with b) vocal guidance and c) projection on the guide plate. We adopted our image-based remote gaze-tracking method to detect the user’s gazing orientation. The results from both empirical studies by subjective / objective evaluations and observations of our demonstration experiments in a semipublic space show i) the total operation of the system, ii) the elicitation of user’s interest by gaze behaviors of the robot, and iii) the effectiveness of the gaze-communicative guide adopting the anthropomorphic robot.

Physical contact using haptic and gestural expressions for ubiquitous partner robot

In this paper, we propose a portable robot to express physical contacts that are parallel to other modalities. It enfolds the users arm in its arms and tapping the users arm. The physical contact expressions are generated through a combination of several haptic stimuli and the robots anthropomorphic behaviors based on its internal state. The aim of our research is building a caregiver-like robot medium. The system was designed for gentle and delicate communication between the user and the robot during a users outings. The haptic stimuli express warm/cold, patting, and squeezing. Experimental results show that haptic communicative behaviors of the robot increase the intelligibility to the robots messages and familiar impressions to the robot.

Location Dependent Illumination Modeling for Multiple-Camera-based 3D Object Reconstruction

We propose a method for modeling ununiform illumination conditions using multiple-camera-based marker observations. In computer graphics applications, multiple-camera-based object reconstruction is becoming popular for modeling 3D objects. However, geometrical and photo-metrical calibrations among multiple cameras still require high computational cost. We employ multiple observations of a spherical marker in a sequence for modeling illuminations. These data are also used for calibrating camera geometries based on a bundled adjustment method to minimize re-projection errors. The illumination condition is modeled as a set of luminance patterns on the virtual planes surrounding the target observation area. An illumination model at an arbitrary position in the scene can be constructed by interpolating the luminance behavior of surrounding planes. Experimental results show the effectiveness of our proposed method.

A body-mounted camera system for head-pose estimation and user-view image synthesis

In this paper, we propose a body-mounted system to capture user experience as audio/visual information. The proposed system consists of two cameras (head-detection and wide angle) and a microphone. The head-detection camera captures user head motions, while the wide angle color camera captures user frontal view images. An image region approximately corresponding to user view is then synthesized from the wide angle image based on estimated human head motions. The synthesized image and head-motion data are stored in a storage device with audio data. This system overcomes the disadvantages of head-mounted cameras in terms of ease of putting on/taking off the device. It also has less obtrusive visual impact on third persons. Using the proposed system, we can simultaneously record audio data, images in the user field of view, and head gestures (nodding, shaking, etc.) simultaneously. These data contain significant information for recording/analyzing human activities and can be used in wider application domains such as a digital diary or interaction analysis. Experimental results demonstrate the effectiveness of the proposed system.

Talking heads synthesis from audio with deep neural networks

Talking heads synthesis with expressions from speech is proposed in this paper. Talking heads synthesis can be considered as a learning problem of sequence-to-sequence mapping, which consists of audio as input and video as output. To synthesize talking heads, we use SAVEE database which consists of videos of multiple sentences speeches recorded from front of face. Audiovisual data can be considered as two parallel sequential data of audio and visual features and it is composed of continuous value. Thus, audio and visual features of our dataset are represented by a regression model. In this research, the regression model is trained with long short-term memory (LSTM) by minimizing mean squared error (MSE). Then, audio features are used as input and visual features are used as target of LSTM. Thereby, talking heads are synthesized from speech. Our method is proposed to use lower level audio features than phonemes and it enables to synthesize talking heads with expressions while existing researches which use phonemes as audio features only can synthesize neutral expression talking heads. With SAVEE database, we achieved the minimum MSE 17.03 on our testing dataset. In experiment, we use mel-frequency cepstral coefficient (MFCC), AMFCC and A2 MFCC with energy as audio feature and active appearance model (AAM) on entire face region as visual feature.