Michael Reale

A high-resolution 3D dynamic facial expression database

Face information processing relies on the quality of data resource. From the data modality point of view, a face database can be 2D or 3D, and static or dynamic. From the task point of view, the data can be used for research of computer based automatic face recognition, face expression recognition, face detection, or cognitive and psychological investigation. With the advancement of 3D imaging technologies, 3D dynamic facial sequences (called 4D data) have been used for face information analysis. In this paper, we focus on the modality of 3D dynamic data for the task of facial expression recognition. We present a newly created high-resolution 3D dynamic facial expression database, which is made available to the scientific research community. The database contains 606 3D facial expression sequences captured from 101 subjects of various ethnic backgrounds. The database has been validated through our facial expression recognition experiment using an HMM based 3D spatio-temporal facial descriptor. It is expected that such a database shall be used to facilitate the facial expression analysis from a static 3D space to a dynamic 3D space, with a goal of scrutinizing facial behavior at a higher level of detail in a real 3D spatio-temporal domain.

A Multi-Gesture Interaction System Using a 3-D Iris Disk Model for Gaze Estimation and an Active Appearance Model for 3-D Hand Pointing

In this paper, we present a vision-based human-computer interaction system, which integrates control components using multiple gestures, including eye gaze, head pose, hand pointing, and mouth motions. To track head, eye, and mouth movements, we present a two-camera system that detects the face from a fixed, wide-angle camera, estimates a rough location for the eye region using an eye detector based on topographic features, and directs another active pan-tilt-zoom camera to focus in on this eye region. We also propose a novel eye gaze estimation approach for point-of-regard (POR) tracking on a viewing screen. To allow for greater head pose freedom, we developed a new calibration approach to find the 3-D eyeball location, eyeball radius, and fovea position. Moreover, in order to get the optical axis, we create a 3-D iris disk by mapping both the iris center and iris contour points to the eyeball sphere. We then rotate the fovea accordingly and compute the final, visual axis gaze direction. This part of the system permits natural, non-intrusive, pose-invariant POR estimation from a distance without resorting to infrared or complex hardware setups. We also propose and integrate a two-camera hand pointing estimation algorithm for hand gesture tracking in 3-D from a distance. The algorithms of gaze pointing and hand finger pointing are evaluated individually, and the feasibility of the entire system is validated through two interactive information visualization applications.

A high-resolution spontaneous 3D dynamic facial expression database

Facial expression is central to human experience. Its efficient and valid measurement is a challenge that automated facial image analysis seeks to address. Most publically available databases are limited to 2D static images or video of posed facial behavior. Because posed and un-posed (aka “spontaneous”) facial expressions differ along several dimensions including complexity and timing, well-annotated video of un-posed facial behavior is needed. Moreover, because the face is a three-dimensional deformable object, 2D video may be insufficient, and therefore 3D video archives are needed. We present a newly developed 3D video database of spontaneous facial expressions in a diverse group of young adults. Well-validated emotion inductions were used to elicit expressions of emotion and paralinguistic communication. Frame-level ground-truth for facial actions was obtained using the Facial Action Coding System. Facial features were tracked in both 2D and 3D domains using both person-specific and generic approaches. The work promotes the exploration of 3D spatiotemporal features in subtle facial expression, better understanding of the relation between pose and motion dynamics in facial action units, and deeper understanding of naturally occurring facial action.

Recognizing partial facial action units based on 3D dynamic range data for facial expression recognition

Research on automatic facial expression recognition has benefited from work in psychology, specifically the Facial Action Coding System (FACS). To date, most existing approaches are primarily based on 2D images or videos. With the emergence of real-time 3D dynamic imaging technologies, however, 3D dynamic facial data is now available, thus opening up an alternative to detect facial action units in dynamic 3D space. In this paper, we investigate how to use this new modality to improve action unit (AU) detection. We select a subset of AUs from both the upper and lower parts of a facial area, apply the active appearance model (AAM) method and take the correspondence between textures and range models to track the pre-defined facial features across the 3D model sequences. A Hidden Markov Model (HMM) based classifier is employed to recognize the partial AUs. The experiments show that our 3D dynamic tracking based approach outperforms the compared 2D feature tracking based approach. The results are also comparable with the manually-picked 3D facial features based method. Finally, we extend our approach to validate the experiment for recognizing six prototypic facial expressions.

Nebula feature: A space-time feature for posed and spontaneous 4D facial behavior analysis

In this paper, we propose a new, compact, 4D spatio-temporal “Nebula” feature to improve expression and facial movement analysis performance. Given a spatio-temporal volume, the data is voxelized and fit to a cubic polynomial. A label is assigned based on the principal curvature values, and the polar angles of the direction of least curvature are computed. The labels and angles for each feature are used to build a histogram for each region of the face. The concatenated histograms from each region give us our final feature vector. This feature description is tested on the posed expression database BU-4DFE and on a new 4D spontaneous expression database. Various region configurations, histogram sizes, and feature parameters are tested, including a non-dynamic version of the approach. The LBP-TOP approach on the texture image as well as on the depth image is also tested for comparison. The onsets of the six canonical expressions are classified for 100 subjects in BU-4DFE, while the onset, offset, and non-existence of 12 Action Units (AUs) are classified for 16 subjects from our new spontaneous database. For posed expression recognition, the Nebula feature approach shows improvement over LBPTOP on the depth images and significant improvement over the non-dynamic 3D-only approach. Moreover, the Nebula feature performs better for AU classification than the compared approaches for 11 of the AUs tested in terms of accuracy as well as Area Under Receiver Operating Characteristic Curve (AUC).

Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis

Emotion is expressed in multiple modalities, yet most research has considered at most one or two. This stems in part from the lack of large, diverse, well-annotated, multimodal databases with which to develop and test algorithms. We present a well-annotated, multimodal, multidimensional spontaneous emotion corpus of 140 participants. Emotion inductions were highly varied. Data were acquired from a variety of sensors of the face that included high-resolution 3D dynamic imaging, high-resolution 2D video, and thermal (infrared) sensing, and contact physiological sensors that included electrical conductivity of the skin, respiration, blood pressure, and heart rate. Facial expression was annotated for both the occurrence and intensity of facial action units from 2D video by experts in the Facial Action Coding System (FACS). The corpus further includes derived features from 3D, 2D, and IR (infrared) sensors and baseline results for facial expression and action unit detection. The entire corpus will be made available to the research community.

Viewing direction estimation based on 3D eyeball construction for HRI

Natural human-robot interaction requires leveraging viewing direction information in order to recognize, respond to, and even emulate human behavior. Knowledge of the eye gaze and point of regard gives us insight into what the subject is interested in and/or who the subject is addressing. In this paper, we present a novel eye gaze estimation approach for point-of-regard (PoG) tracking. To allow for greater head pose freedom, we introduce a new calibration approach to find the 3D eyeball location, eyeball radius, and fovea position. To estimate gaze direction, we map both the iris center and iris contour points to the eyeball sphere (creating a 3D iris disk), giving us the optical axis. We then rotate the fovea accordingly and compute our final, visual axis gaze direction. Our intention is to integrate this eye gaze approach with a dual-camera system we have developed that detects the face and eyes from a fixed, wide-angle camera and directs another active pan-tilt-zoom camera to focus in on this eye region. The final system will permit natural, non-intrusive, pose-invariant PoG estimation in distance and allow user translational freedom without resorting to infrared equipment or complex hardware setups.

Pointing with the eyes: Gaze estimation using a static/active camera system and 3D iris disk model

The ability to capture the direction the eyes point in while the subject is a distance away from the camera offers the potential for intuitive human-computer interfaces, allowing for a greater interactivity, more intelligent HCI behavior, and increased flexibility. In this paper, we present a two-camera system that detects the face from a fixed, wide-angle camera, estimates a rough location for the eye region using an eye detector based on topographic features, and directs another active pan-tilt-zoom camera to focus in on this eye region. We also propose a novel eye gaze estimation approach for point-of-regard (PoG) tracking on a large viewing screen. To allow for greater head pose freedom, we developed a new calibration approach to find the 3D eyeball location, eyeball radius, and fovea position. Moreover, we map both the iris center and iris contour points to the eyeball sphere (creating a 3D iris disk) to get the optical axis; we then rotate the fovea accordingly and compute our final, visual axis gaze direction. We intend to integrate this gaze estimation approach with our two-camera system, permitting natural, non-intrusive, pose-invariant PoG estimation in distance and allowing user translational freedom without resorting to infrared or complex hardware setups such as stereo-cameras or “smart rooms.”

3D Head Pose Estimation Based on Scene Flow and Generic Head Model

Head pose is an important indicator of a persons attention, gestures, and communicative behavior with applications in human computer interaction, multimedia and vision systems. In this paper, we present a novel head pose estimation system by performing head region detection using the Kinect [2], followed by face detection, feature tracking, and finally head pose estimation using an active camera. Ten feature points on the face are defined and tracked by an Active Appearance Model (AAM). We propose to use the scene flow approach to estimate the head pose from 2D video sequences. This estimation is based upon a generic 3D head model through the prior knowledge of the head shape and the geometric relationship between the 2D images and a 3D generic model. We have tested our head pose estimation algorithm with various cameras at various distances in real time. The experiments demonstrate the feasibility and advantages of our system.

Using eye gaze, head pose, and facial expression for personalized non-player character interaction

True immersion of a user within a game is only possible when the world simulated looks and behaves as close to reality as possible. This implies that the game must ascertain, among other things, the users focus and his/her attitude towards the object or person focused on. As part of the effort to achieve this goal, we propose an eye gaze, head pose, and facial expression system for use in real-time games. Both the eye gaze and head pose components utilize underlying 3D models, while the expression recognition module uses the effective but efficient LBP-TOP approach. We then demonstrate the utility of this system in a test application wherein the user looks at one of three non-player characters (NPC) and performs one of the 7 prototypic expressions; the NPC responds based on its personality. To increase the speed and efficiency of the system, the eye gaze and expression recognition modules leverage CUDA and GLSL pixel shaders.