Chengchao Qu

Fast, robust and automatic 3D face model reconstruction from videos

This paper presents a fully automatic system that recovers 3D face models from sequences of facial images. Unlike most 3D Morphable Model (3DMM) fitting algorithms that simultaneously reconstruct the shape and texture from a single input image, our approach builds on a more efficient least squares method to directly estimate the 3D shape from sparse 2D landmarks, which are localized by face alignment algorithms. The inconsistency between self-occluded 2D and 3D feature positions caused by head pose is ad-dressed. A novel framework to enhance robustness across multiple frames selected based on their 2D landmarks combined with individual self-occlusion handling is proposed. Evaluation on groundtruth 3D scans shows superior shape and pose estimation over previous work. The whole system is also evaluated on an “in the wild” video dataset [12] and delivers personalized and realistic 3D face shape and texture models under less constrained conditions, which only takes seconds to process each video clip.

Towards robust cascaded regression for face alignment in the wild

Most state-of-the-art solutions for localizing facial feature landmarks build on the recent success of the cascaded regression framework [7, 15, 34], which progressively predicts the shape update based on the previous shape estimate and its feature calculation. We revisit several core aspects of this framework and show that proper selection of regression method, local image feature and fine-tuning of further fitting strategies can achieve top performance for face alignment using the generic cascaded regression algorithm. In particular, our strongest model features Iteratively Reweighted Least Squares (IRLS) [18] for training robust regressors in the presence of outliers in the training data, RootSIFT [2] as the image patch descriptor that replaces the original Euclidean distance in SIFT [24] with the Hellinger distance, as well as coarse-to-fine fitting and in-plane pose normalization during shape update. We show the benefit of each proposed improvement by extensive individual experiments compared to the baseline approach [34] on the LFPW dataset [4]. On the currently most challenging 300-W dataset [28] and COFW dataset [4], we report state-of-the-art results that are superior to or on par with recently published algorithms.

Adaptive Contour Fitting for Pose-Invariant 3D Face Shape Reconstruction.

Motivation Direct reconstruction of 3D face shape—solely based on a sparse set of 2D feature points localized by a facial landmark detector— offers an automatic, efficient and illumination-invariant alternative to the widely known analysis-by-synthesis framework, which is extremely timeconsuming considering the enormous parameter space for both shape and photometric properties. Given 2D landmarks y and their correspondence on the 3D Morphable Model (3DMM), the 3D shape can be recovered by minimizing the distance between 2D and the projected 3D landmarks

Capturing ground truth super-resolution data

Super-resolution (SR) offers an effective approach to boost quality and details of low-resolution (LR) images to obtain high-resolution (HR) images. Despite the theoretical and technical advances in the past decades, it still lacks plausible methodology to evaluate and compare different SR algorithms. The main cause to this problem lies in the missing ground truth data for SR. Unlike in many other computer vision tasks, where existing image datasets can be utilized directly, or with a little extra annotation work, evaluating SR requires that the dataset contain both LR and the corresponding HR ground truth images of the same scene captured at the same time. This work presents a novel prototype camera system to address the aforementioned difficulties of acquiring ground truth SR data. Two identical camera sensors equipped with a wide-angle lens and a telephoto lens respectively, share the same optical axis by placing a beam splitter in the optical path. The back-end program can then trigger their shutters simultaneously and precisely register the region of interests (ROIs) of the LR and HR image pairs in an automated manner free of sub-pixel interpolation. Evaluation results demonstrate the special characteristics of the captured ground truth HR-LR face images compared to the simulated ones. The dataset is made freely available for noncommercial research purposes.

Low-resolution video face recognition with face normalization and feature adaptation

Face analysis is a challenging topic, especially when addressing low-resolution data. While face detection is working satisfactorily on such data, further facial analysis often struggles. We specifically address the issues of face registration, face normalization and facial feature extraction to perform low-resolution face recognition. For face registration, an approach for landmark detection, pose estimation and pose normalization is presented. In addition, a strategy to mirror the visible face half in the case of a rotated face is suggested. Next, the normalized face is used to extract the features for recognition. Using situation adapted local binary patterns (LBP) which are collected according to the proposed framework, including several scales and spatial overlaps, boosts the recognition performance well above the baseline. Results are presented on the YouTube Faces Database which is the current state-of-the-art dataset for video face recognition. Proper adjustments are made to convert this high-resolution dataset to a low-resolution one. We show that the presented adaptations increase face recognition performance significantly for low-resolution scenarios, closing a large part of the gap to high resolution face recognition.

Impact of resolution and image quality on video face analysis

Low-resolution face analysis suffers more significantly from quality degradations than high-resolution analysis. In this work, we will investigate how several face analysis steps are influenced by low image quality and how this relates to the low resolution. In the first step, a simulation of different effects on image quality, namely low resolution, compression artifacts, motion blur and noise is performed and the impact on face detection, registration and recognition is analyzed. Depending on the situation, it becomes obvious that the low resolution is sometimes a minor degrading effect, outmatched by a single one or a combination of the further effects. When addressing real-world face recognition from surveillance data, the combination of the challenging effects is the biggest problem because typical counter measures are individual to one single effect.

Realistic texture extraction for 3D face models robust to self-occlusion

In the context of face modeling, probably the most well-known approach to represent 3D faces is the 3D Morphable Model (3DMM). When 3DMM is fitted to a 2D image, the shape as well as the texture and illumination parameters are simultaneously estimated. However, if real facial texture is needed, texture extraction from the 2D image is necessary. This paper addresses the possible problems in texture extraction of a single image caused by self-occlusion. Unlike common approaches that leverage the symmetric property of the face by mirroring the visible facial part, which is sensitive to inhomogeneous illumination, this work first generates a virtual texture map for the skin area iteratively by averaging the color of neighbored vertices. Although this step creates unrealistic, overly smoothed texture, illumination stays constant between the real and virtual texture. In the second pass, the mirrored texture is gradually blended with the real or generated texture according to the visibility. This scheme ensures a gentle handling of illumination and yet yields realistic texture. Because the blending area only relates to non-informative area, main facial features still have unique appearance in different face halves. Evaluation results reveal realistic rendering in novel poses robust to challenging illumination conditions and small registration errors.

3D vs. 2D: On the Importance of Registration for Hallucinating Faces Under Unconstrained Poses

Face Hallucination (FH) differs from generic single-image super-resolution (SR) algorithms in its specific domain of application. By exploiting the common structures of human faces, magnification of lower resolution images can be achieved. Despite the growing interest in recent years, considerably less attention is paid to a crucial step in FH -- registration of facial images. In this work, registration techniques employed in the literature are first summarized and the importance of using well-aligned training and test images is demonstrated. A novel method to inversely map the high-resolution (HR) 3D training texture to the low-resolution (LR) 2D test image in arbitrary poses is then presented, which prevents information loss in LR images and is thus beneficial to SR. The effectiveness of our 3D approach is evaluated on the Multi-PIE and the PUT face databases. Superior qualitative and quantitative FH results to the state-of-the-art methods in all tested poses prove the necessity of accurate registration in FH. The merit of 3D FH in generating super-resolved frontal faces is also verified, revealing 30% improvement in face recognition over the 2D approach under 30° of yaw rotation on the Multi-PIE dataset.

Resolution-aware Constrained Local Model with mixture of local experts

Deformable model fitting to high-resolution facial images has been extensively studied for over two decades. However, due to the ill-posed problem caused by low-resolution images, most existing work cannot be applied directly and degrades quickly as the resolution decreases. To address this issue, this paper extends the Constrained Local Model (CLM) to a multi-resolution model consisting of a 4-level patch pyramid, and deploys various feature descriptors for the local patch experts as well. We evaluate the proposed work on the BioID, the MUCT and the Multi-PIE datasets. Superior results are achieved on almost all resolution levels, demonstrating the effectiveness and necessity of our resolution-aware approach for the low-resolution fitting. Improved performance of patch models employing several feature combinations over the single intensity feature under different conditions is also presented.

Towards mobile and touchless fingerprint verification

Touchless finger detection for the biometric fingerprint verification/identification process with mobile devices is considered in this paper. Fingerprint capturing is based on a camera system with bright-field illumination. For finger detection, a machine learning based algorithm with Aggregated Channel Features (ACFs) and a skin-color based finger segmentation with a geometric shape based approach for fingertip detection are considered, respectively. Results demonstrate the performance of both algorithms.