Yuru Pei

Unsupervised Image Matching Based on Manifold Alignment

This paper challenges the issue of automatic matching between two image sets with similar intrinsic structures and different appearances, especially when there is no prior correspondence. An unsupervised manifold alignment framework is proposed to establish correspondence between data sets by a mapping function in the mutual embedding space. We introduce a local similarity metric based on parameterized distance curves to represent the connection of one point with the rest of the manifold. A small set of valid feature pairs can be found without manual interactions by matching the distance curve of one manifold with the curve cluster of the other manifold. To avoid potential confusions in image matching, we propose an extended affine transformation to solve the nonrigid alignment in the embedding space. The comparatively tight alignments and the structure preservation can be obtained simultaneously. The point pairs with the minimum distance after alignment are viewed as the matchings. We apply manifold alignment to image set matching problems. The correspondence between image sets of different poses, illuminations, and identities can be established effectively by our approach.

The Craniofacial Reconstruction from the Local Structural Diversity of Skulls

The craniofacial reconstruction is employed as an initialization of the identification from skulls in forensics. In this paper, we present a two‐level craniofacial reconstruction framework based on the local structural diversity of the skulls. On the low level, the holistic reconstruction is formulated as the superimposition of the selected tissue map on the novel skull. The crux is the accurate map registration, which is implemented as a warping guided by the 2D feature curve patterns. The curve pattern extraction under an energy minimization framework is proposed for the automatic feature labeling on the skull depth map. The feature configuration on the warped tissue map is expected to resemble that on the novel skull. In order to make the reconstructed faces personalized, on the high level, the local facial features are estimated from the skull measurements via a RBF model. The RBF model is learnt from a dataset of the skull and the face feature pairs extracted from the head volume data. The experiments demonstrate the facial outlooks can be reconstructed feasibly and efficiently.

Transferring of Speech Movements from Video to 3D Face Space

We present a novel method for transferring speech animation recorded in low quality videos to high resolution 3D face models. The basic idea is to synthesize the animated faces by an interpolation based on a small set of 3D key face shapes which span a 3D face space. The 3D key shapes are extracted by an unsupervised learning process in 2D video space to form a set of 2D visemes which are then mapped to the 3D face space. The learning process consists of two main phases: 1) isomap-based nonlinear dimensionality reduction to embed the video speech movements into a low-dimensional manifold and 2) k-means clustering in the low-dimensional space to extract 2D key viseme frames. Our main contribution is that we use the isomap-based learning method to extract intrinsic geometry of the speech video space and thus to make it possible to define the 3D key viseme shapes. To do so, we need only to capture a limited number of 3D key face models by using a general 3D scanner. Moreover, we also develop a skull movement recovery method based on simple anatomical structures to enhance 3D realism in local mouth movements. Experimental results show that our method can achieve realistic 3D animation effects with a small number of 3D key face models

Tissue map based craniofacial reconstruction and facial deformation using RBF network

In this paper we present a novel craniofacial reconstruction method employing statistical tissue thickness information. The tissue thickness data gotten from CT images are represented as 2D tissue maps. The input (target) skull model is parameterized onto a 2D planar map and the landmarks are utilized to train a RBFN (radial basis function network), which realizes warping of planar maps between the target tissue and the generic tissue. The generic tissue is aligned onto the target skull by applying the trained network onto it, and thus the target facial map can be obtained by a simple addition of the warped generic maps. Finally, we interactively deform the model based on a RBFN to make the facial meshes more personalized, and map the texture from orthogonal photos onto the reconstructed model to improve rendering effects. Experiment results show that the proposed approach is helpful in improving the recognition ability in forensic applications.

Personalized Tooth Shape Estimation From Radiograph and Cast

Three-dimensional geometric information of teeth is usually needed in pre- and postoperative diagnoses of orthodontic dentistry. The computerized tomography can provide comprehensive 3-D teeth geometries. However, there is still a discussion on computed tomography (CT) as a routine in orthodontic dentistry due to radiation dose. Moreover, the CT is useless when a dentist needs to extract 3-D structures from old archive files with only radiographs and casts, where patient’s teeth changed ever since. In this paper, we propose a reconstruction framework for patient-specific teeth based on an integration of 2-D radiographs and digitized casts. The reconstruction is under a template-fitting framework. The shape and orientation of teeth templates are tuned in accordance with patient’s radiographs. Specially, the tooth root morphology is controlled by 2-D contours in radiographs. With ray tracing and a contour plane assumption, 2-D root contours in radiographs are projected back to 3-D space, and guide tooth root deformations. Moreover, the template’s crown is deformed nonrigidly to fit digitized casts that bear patient’s crown details. The system allows 3-D tooth reconstruction with patient-specific geometric details from just casts and 2-D radiographs.

Enhanced Random Forest with Image/Patch-Level Learning for Image Understanding

Image understanding is an important research domain in the computer vision due to its wide real-world applications. For an image understanding framework that uses the Bag-of-Words model representation, the visual codebook is an essential part. Random forest (RF) as a tree-structure discriminative codebook has been a popular choice. However, the performance of the RF can be degraded if the local patch labels are poorly assigned. In this paper, we tackle this problem by a novel way to update the RF codebook learning for a more discriminative codebook with the introduction of the soft class labels, estimated from the pLSA model based on a feedback scheme. The feedback scheme is performed on both the image and patch levels respectively, which is in contrast to the state-of-the-art RF codebook learning that focused on either image or patch level only. Experiments on 15-Scene and C-Pascal datasets had shown the effectiveness of the proposed method in image understanding task.

Vision based speech animation transferring with underlying anatomical structure

We present a novel method to transfer speech animation recorded in low resolution videos onto realistic 3D facial models. Unsupervised learning is utilized on a speech video corpus to find underlying manifold of facial configurations. K-means clustering is applied on the low dimensional space to find key speaking-related facial shapes. With a small set of laser scanner captured 3D models related to the clustering centroid, the facial animation in 2D videos is transferred onto 3D shapes. Especially by virtue of a weak perspective projection model, the underlying mandible rotation is recovered from videos and is utilized to drive 3D skull movements. The adaption of a generic skull onto facial models is guided by a 2D image, Tissue Map. With parsimonious data requirements, our system realizes the animation transferring and gains a realistic rendering effect with the underlying anatomical structure.

Superimposition of Cone-Beam Computed Tomography Images by Joint Embedding

Objective: The superimposition of cone-beam computed tomography (CBCT) images is an essential step to evaluate shape variations of pre and postorthodontic operations due to pose variations and the bony growth. The aim of this paper is to present and discuss the latest accomplishments in voxel-based craniofacial CBCT superimpositions along with structure discriminations. Methods: We propose a CBCT superimposition method based on joint embedding of subsets extracted from CBCT images. The subset is defined at local extremes of the first-order difference of Gaussian-smoothed volume images to reduce the data involved in the computation. A rotation-invariant integral operator is proposed as the context-aware textural descriptor of subsets. We cope with subset correspondences by joint embedding with matching identifications in manifolds, which take into account the structure of subsets as a whole to avoid mapping ambiguities. Once given subset correspondences, the rigid transformations, as well as the superimposition of volume images, are obtained. Our system allows users to specify the structure-of-interest based on a semisupervised label propagation technique. Results: The performance of the proposed method is evaluated on ten pairs of pre and postoperative CBCT images of adult patients and ten pairs of growing patients, respectively. The experiments demonstrate that the craniofacial CBCT superimposition can be performed effectively, and outperform state of the arts. Conclusion: The integration of sparse subsets with context-aware spherical intensity integral descriptors and correspondence establishment by joint embedding enables the reliable and efficient CBCT superimposition. Significance: The potential of CBCT superimposition techniques discussed in this paper is highlighted and related challenges are addressed.

Non-rigid Craniofacial 2D-3D Registration Using CNN-Based Regression

The 2D-3D registration is a cornerstone to align the inter-treatment X-ray images with the available volumetric images. In this paper, we propose a CNN regression based non-rigid 2D-3D registration method. An iterative refinement scheme is introduced to update the reference volumetric image and the digitally-reconstructed-radiograph (DRR) for convergence to the target X-ray image. The CNN-based regressor represents the mapping between an image pair and the in-between deformation parameters. In particular, the short residual connections in the convolution blocks and long jump connections for the multi-scale feature map fusion facilitate the information propagation in training the regressor. The proposed method has been applied to 2D-3D registration of synthetic X-ray and clinically-captured CBCT images. Experimental results demonstrate the proposed method realizes an accurate and efficient 2D-3D registration of craniofacial images.

Interactive modeling of 3D facial expressions with hierarchical Gaussian process latent variable models

The natural expressions play an important role in the daily communication. The efficient and intuitive facial expression editing based on the limited constraints is desirable in the facial animation. In this paper, we present an interactive 3D facial expression editing system with the hierarchical Gaussian process latent variable model (HGPLVM). The hierarchical model incorporates the joint work of the local facial features to produce the natural expressions. To deal with the holistic expression modeling from the local constraints, the inverse mapping between the low-level feature nodes and the high-level facial region nodes is established by the RBF regression model in the latent space. A propagation algorithm is introduced to predict the holistic facial configurations. The experiments demonstrate the 3D facial expressions satisfying the user constraints can be produced efficiently.