Shaun J. Canavan

A Multi-Gesture Interaction System Using a 3-D Iris Disk Model for Gaze Estimation and an Active Appearance Model for 3-D Hand Pointing

In this paper, we present a vision-based human-computer interaction system, which integrates control components using multiple gestures, including eye gaze, head pose, hand pointing, and mouth motions. To track head, eye, and mouth movements, we present a two-camera system that detects the face from a fixed, wide-angle camera, estimates a rough location for the eye region using an eye detector based on topographic features, and directs another active pan-tilt-zoom camera to focus in on this eye region. We also propose a novel eye gaze estimation approach for point-of-regard (POR) tracking on a viewing screen. To allow for greater head pose freedom, we developed a new calibration approach to find the 3-D eyeball location, eyeball radius, and fovea position. Moreover, in order to get the optical axis, we create a 3-D iris disk by mapping both the iris center and iris contour points to the eyeball sphere. We then rotate the fovea accordingly and compute the final, visual axis gaze direction. This part of the system permits natural, non-intrusive, pose-invariant POR estimation from a distance without resorting to infrared or complex hardware setups. We also propose and integrate a two-camera hand pointing estimation algorithm for hand gesture tracking in 3-D from a distance. The algorithms of gaze pointing and hand finger pointing are evaluated individually, and the feasibility of the entire system is validated through two interactive information visualization applications.

A high-resolution spontaneous 3D dynamic facial expression database

Facial expression is central to human experience. Its efficient and valid measurement is a challenge that automated facial image analysis seeks to address. Most publically available databases are limited to 2D static images or video of posed facial behavior. Because posed and un-posed (aka “spontaneous”) facial expressions differ along several dimensions including complexity and timing, well-annotated video of un-posed facial behavior is needed. Moreover, because the face is a three-dimensional deformable object, 2D video may be insufficient, and therefore 3D video archives are needed. We present a newly developed 3D video database of spontaneous facial expressions in a diverse group of young adults. Well-validated emotion inductions were used to elicit expressions of emotion and paralinguistic communication. Frame-level ground-truth for facial actions was obtained using the Facial Action Coding System. Facial features were tracked in both 2D and 3D domains using both person-specific and generic approaches. The work promotes the exploration of 3D spatiotemporal features in subtle facial expression, better understanding of the relation between pose and motion dynamics in facial action units, and deeper understanding of naturally occurring facial action.

Hand Pointing Estimation for Human Computer Interaction Based on Two Orthogonal-Views

Hand pointing has been an intuitive gesture for human interaction with computers. Big challenges are still posted for accurate estimation of finger pointing direction in a 3D space. In this paper, we present a novel hand pointing estimation system based on two regular cameras, which includes hand region detection, hand finger estimation, two views’ feature detection, and 3D pointing direction estimation. Based on the idea of binary pattern face detector, we extend the work to hand detection, in which a polar coordinate system is proposed to represent the hand region, and achieved a good result in terms of the robustness to hand orientation variation. To estimate the pointing direction, we applied an AAM based approach to detect and track 14 feature points along the hand contour from a top view and a side view. Combining two views of the hand features, the 3D pointing direction is estimated. The experiments have demonstrated the feasibility of the system.

Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis

Emotion is expressed in multiple modalities, yet most research has considered at most one or two. This stems in part from the lack of large, diverse, well-annotated, multimodal databases with which to develop and test algorithms. We present a well-annotated, multimodal, multidimensional spontaneous emotion corpus of 140 participants. Emotion inductions were highly varied. Data were acquired from a variety of sensors of the face that included high-resolution 3D dynamic imaging, high-resolution 2D video, and thermal (infrared) sensing, and contact physiological sensors that included electrical conductivity of the skin, respiration, blood pressure, and heart rate. Facial expression was annotated for both the occurrence and intensity of facial action units from 2D video by experts in the Facial Action Coding System (FACS). The corpus further includes derived features from 3D, 2D, and IR (infrared) sensors and baseline results for facial expression and action unit detection. The entire corpus will be made available to the research community.

A dynamic curvature based approach for facial activity analysis in 3D space

This paper presents a novel dynamic curvature based approach (dynamic shape-index based approach) for 3D face analysis. This method is inspired by the idea of 2D dynamic texture and 3D surface descriptors. The dynamic texture (DT) based approaches [30][31][32] encode and model the local texture features in the temporal axis, and have achieved great success in applications of 2D facial expression recognition. In this paper, we propose a so-called Dynamic Curvature (DC) approach for 3D facial activity analysis. To do so, the 3D dynamic surface is described by its surface curvature-based shape-index information. The surface features are characterized in local regions along the temporal axis. A dynamic curvature descriptor is constructed from local regions as well as temporal domains. To locate the local regions, we also applied a 3D tracking model based method for detecting and tracking 3D features across 3D dynamic sequences. Our method is validated through our experiment on 3D facial activity analysis for distinguishing neutral vs. non-neutral expressions, prototypic expressions, and their intensities.

Face Recognition by Multi-Frame Fusion of Rotating Heads in Videos

This paper presents a face recognition study that implicitly utilizes the 3D information in 2D video sequences through multi-sample fusion. The approach is based on the hypothesis that continuous and coherent intensity variations in video frames caused by a rotating head can provide information similar to that of explicit shapes or range images. The fusion was done on the image level to prevent information loss. Experiments were carried out using a data set of over 100 subjects and promising results have been obtained: (1) under regular indoor lighting conditions, rank one recognition rate increased from 91% using a single frame to 100% using 7-frame fusion; (2) under strong shadow conditions, rank one recognition rate increased from 63% using a single frame to 85% using 7-frame fusion.

A biometric database with rotating head videos and hand-drawn face sketches

The past decade has witnessed a significant progress in biometric technologies, to a large degree, due to the availability of a wide variety of public databases that enable benchmark performance evaluations. In this paper, we describe a new database that includes: (i) Rotating head videos of 259 subjects; (ii) 250 hand-drawn face sketches of 50 subjects. Rotating head videos were acquired under both normal indoor lighting and shadow conditions. Each video captured four expressions: neutral, smile, surprise, and anger. For each subject, video frames of ten pose angles were manually labeled using reference images and empirical rules, to facilitate the investigation of multi-frame fusion. The database can also be used to study 3D face recognition by reconstructing a 3D face model from videos. In addition, this is the only currently available database that has a large number of face sketches drawn by multiple artists. The face sketches are valuable resource for many researches, such as forensic analysis of eyewitness recollection, impact assessment of face degradation on recognition rate, as well as comparative evaluation of sketch recognitions by humans and algorithms.

Landmark localization on 3D/4D range data using a shape index-based statistical shape model with global and local constraints

We propose a method for landmark localization on 3D and 4D range data.A new Shape Index-Based Statistical Shape Model is proposed.Five Benchmark 3D/4D face databases are tested on.The accuracy of the landmarks is compared to ground truth data, and state-of-the-art methods.The efficacy of the landmarks is validated through expression analysis and pose estimation. In this paper we propose a novel method for detecting and tracking facial landmark features on 3D static and 3D dynamic (a.k.a. 4D) range data. Our proposed method involves fitting a shape index-based statistical shape model (SI-SSM) with both global and local constraints to the input range data. Our proposed model makes use of the global shape of the facial data as well as local patches, consisting of shape index values, around landmark features. The shape index is used due to its invariance to both lighting and pose changes. The fitting is performed by finding the correlation between the shape model and the input range data. The performance of our proposed method is evaluated in terms of various geometric data qualities, including data with noise, incompletion, occlusion, rotation, and various facial motions. The accuracy of detected features is compared to the ground truth data as well as to start of the art results. We test our method on five publicly available 3D/4D databases: BU-3DFE, BU-4DFE, BP4D-Spontaneous, FRGC 2.0, and Eurecom Kinect Face Dataset. The efficacy of the detected landmarks is validated through applications for geometric based facial expression classification for both posed and spontaneous expressions, and head pose estimation. The merit of our method is manifested as compared to the state of the art feature tracking methods.

Recognizing face sketches by a large number of human subjects: A perception-based study for facial distinctiveness

Understanding how humans recognize face sketches drawn by artists is of significant value to both criminal investigators and researchers in computer vision, face biometrics and cognitive psychology. However, large scale experimental studies of hand-drawn face sketches are still very limited in terms of the number of artists, the number of sketches, and the number of human evaluators involved. In this paper, we reported the results of a series of psychological experiments in which 406 volunteers were asked to recognize 250 sketches drawn by 5 different artists. The primary findings are: (i) Sketch quality (artist factor) has a significant effect on human performance. Inter-artist variation as measured by the mean recognition rate can be as high as 31%; (ii) Participants showed a higher tendency to match multiple sketches to one photo than to second-guess their answers. The multi-match ratio seems correlated to the recognition rate, while second-guessing had no significant effect on human performance; (iii) For certain highly recognized faces, their rankings were very consistent using three measuring parameters: recognition rate, multi-match ratio, and second-guess ratio, suggesting that the three parameters could provide valuable information to quantify facial distinctiveness.

Fitting and tracking 3D/4D facial data using a temporal deformable shape model

In this paper, we propose a novel method for detecting and tracking landmark facial features on purely geometric 3D and 4D range models. Our proposed method involves fitting a new multi-frame constrained 3D temporal deformable shape model (TDSM) to range data sequences. We consider this a temporal based deformable model as we concatenate consecutive deformable shape models into a single model driven by the appearance of facial expressions. This allows us to simultaneously fit multiple models over a sequence of time with one TDSM. To our knowledge, it is the first work to address multiple shape models as a whole to track 3D dynamic range sequences without assistance of any texture information. The accuracy of the tracking results is evaluated by comparing the detected landmarks to the ground truth. The efficacy of the 3D feature detection and tracking over range model sequences has also been validated through an application in 3D geometric based face and expression analysis and expression sequence segmentation. We tested our method on the publicly available databases, BU-3DFE [15], BU-4DFE [16], and FRGC 2.0 [12]. We also validated our approach on our newly developed 3D dynamic spontaneous expression database [17].