Yuanqi Su

A parameterized representation for the cartoon sample space

In this paper, the cartoon sample space and its organization are presented. The samples consisting the faces and the corresponding facial cartoons drawn by artists, are parameterized according to the proposed cartoon face model, forming the painting parameter definition (PPD) and the reference shape definition (RSD). And tow conversions constitute the maps between RSD and PPD. Using the cartoon sample space, the parametric representation and organization of the samples across different styles can be resolved in a unified framework, which forms the basis for automatic cartoon face modeling and facial cartoon rendering based on the sample learning.

Expression transfer for facial sketch animation

This paper presents a hierarchical animation method for transferring facial expressions extracted from a performance video to different facial sketches. Without any expression example obtained from target faces, our approach can transfer expressions by motion retargetting to facial sketches. However, in practical applications, the image noise in each frame will reduce the feature extraction accuracy from source faces. And the shape difference between source and target faces will influence the animation quality for representing expressions. To solve these difficulties, we propose a robust neighbor-expression transfer (NET) model, which aims at modeling the spatial relations among sparse facial features. By learning expression behaviors from neighbor face examples, the NET model can reconstruct facial expressions from noisy signals. Based on the NET model, we present a hierarchical method to animate facial sketches. The motion vectors on the source face are adjusted from coarse to fine on the target face. Accordingly, the animation results are generated to replicate source expressions. Experimental results demonstrate that the proposed method can effectively and robustly transfer expressions by noisy animation signals.

Nonlinear deformation learning for face alignment across expression and pose

As a fundamental work for the automatic emotional health analysis, we present a non-linear deformation learning approach to align face images and extract feature points undergoing a variety of expression and pose variations. To face the application in practice, only a single template of the query face is offered for the proposed method. Accurate alignment depends on estimating the optimal deformable shape parameters. We find that facial deformation characters can constrain the shape parameters indirectly. By formulating the non-linear deformation as several piece-wise convex combinations of local neighbor samples, the deformation constraint is imposed to the objective function. An adaptive optimization method is presented, in which the local neighbors are updated correspondingly. Our fitting model satisfies both the global shape prior and the deformation correlations among all feature points. Moreover, to avoid the optimization stacking into local minimal, a discriminative method is further designed to guide the facial deformation in each iterative search. Thus the proposed method is suitable for fully automatic applications. Extensive experiments demonstrate the accuracy and effectiveness of our approach.

Three-Dimensional Traffic Scenes Simulation From Road Image Sequences

In this paper, we present a novel framework to allow users to tour simulated traffic scenes from the first-person view. Constructing 3-D scenes from road image sequences is in general difficult, due to the intrinsic complexity of dynamic road scenes, which are composed of a drastically moving background, not to mention numerous other surrounding vehicles. With the definitions of the traffic scene models, we first introduce the construction process of the simple traffic scenes. After the detection of road boundaries by a semantic fast two-cycle (FTC) level set method, we generate the control points on road sides to construct the “floor-wall” background scene that is subsequently propagated to each frame. Furthermore, we approach the cluttered traffic scenes through a three-component processing pipeline as follows: 1) traffic elements segmentation; 2) background images inpainting; and 3) traffic scenes construction. The traffic elements in the cluttered images are segmented by the semantic FTC level set method first. A Gaussian mixture model is then employed to inpaint the occluded background utilizing the optical flows. The cluttered traffic scenes can be constructed after the segmentation and inpainting components. The foreground polygons such as vehicles and traffic signs are then modeled. Users can change their viewpoints according to their own interpretations. We present the evaluations of each technical component, followed by our findings from comprehensive user studies, which well demonstrate the effectiveness of the proposed framework in delivering good touring experience to users.

Fast two-cycle curve evolution with narrow perception of background for object tracking and contour refinement

The problem of object contour tracking in image sequences remains challenging, especially those with cluttered backgrounds. In this paper, the fast two-cycle level set method with narrow perception of background (FTCNB) is proposed to extract the foreground objects, e.g. vehicles from road image sequences. The curve evolution of the level set method is implemented by computing the signs of region competition terms on two linked lists of contour pixels rather than by solving partial differential equations (PDEs). The curve evolution process mainly consists of two cycles: one cycle for contour pixel evolution and a second cycle for contour pixel smoothness. Based on the curve evolution process, we introduce two tracking stages for the FTCNB method. For coarse tracking stage, the speed function is defined by region competition term combining color and texture features. For contour refinement stage which requires higher tracking accuracy, the likelihood models of the Maximum a posterior (MAP) expressions are incorporated for the speed function. Both the tracking and refinement stages utilize the fast two-cycle curve evolution process with the narrow perception of background regions. With these definitions, we conduct extensive experiments and comparisons for the proposed method. The comparisons with other baseline methods well demonstrate the effectiveness of our work. Graphical abstract� HighlightsA novel level set method without solving partial differential equations.Two tracking levels proposed: object tracking and contour refinement.The speed functions concerning the feature difference between foreground and close background.Fast two-cycle curve evolution process.Extensive evaluations and comparisons with the baseline methods.

Iteratively parsing contour fragments for object detection

In this paper, we explicitly consider the effect of contour fragments on the object detection performance and propose a new approach for linking edges into contour fragments. Our main observation is that the covering condition describing how contour fragments cover objects of interest is the critical factor affecting the detection accuracy. We utilize the general min-cover framework to explain an edge map. During the optimization procedure, we sequentially select best contour fragments in each connected component of the edge map for obtaining locally optimal contour fragments. Furthermore, the sequential selection of best contour fragments is reduced to an iterative parsing procedure. We conduct experiments on the ETHZ and INRIA horse datasets and compare the proposed method with other typical methods of generating contour fragments. Experimental results illustrate that our method achieves a proper covering condition and produces contour fragments that lead to better object detection performance. Besides, the proposed method is easy to compute, leading to a variety of potential real-time applications.

A face cartoon producer for digital content service

Reproducing face cartoon has the potential to be an important digital content service which could be widely used in mobile communication. Here, a face cartoon producer named “NatureFace”, integrated with some novel techniques, is introduced. To generate a face cartoon for a given face involves proper modeling for face, and efficient representation and rendering for cartoon. For both face and cartoon, new definition are introduced for modeling. For face modeling, it is the reference shape; while for cartoons, the painting pattern for corresponding cartoon. Sufficient samples are collected for two parts from the materials supplied by invited artists according to the pre-assigned artistic styles. Given an input face image, the process for generating cartoon has the following steps. First shape features are extracted. Then, painting entities for facial components are selected and deformed to fit current face. Finally, the cartoon rendering engine synthesizes painting entities with rendering rules originated from designated artistic style, resulting in the cartoon. To validate the proposed algorithm, cartoons generated for a group of face images with rare interaction are evaluated and ranked in an artistic style way.

Fast Two-Cycle level set tracking with narrow perception of background

The problem of tracking foreground objects in a video sequence with moving background remains challenging. In this paper, we propose the Fast Two-Cycle level set method with Narrow band Background (FTCNB) to automatically extract the foreground objects in such video sequences. The level set curve evolution process consists of two successive cycles: one cycle for data dependent term and a second cycle for smoothness regularization. The curve evolution is implemented by computing the signs of region competition terms on two linked lists of contour pixels rather than solving any Partial Differential Equations (PDEs). Maximum A Posterior (MAP) optimization is applied in the FTCNB method for curve refinement with the assistance of optical flows. The comparison with other level set methods demonstrate the tracking accuracy of our method. The tracking speed of the proposed method also outperforms the traditional level set methods.

A Method for Deforming-Driven Exaggerated Facial Animation Generation

This paper proposes a method for automatically generating facial animation with exaggerated features from a facial image. According to facial diversity and identity, an exaggerated face can be determined by the neutral face and the exaggeration effect difference that is represented with the deforming parameters. The proposed method utilizes the central point and the feature vector to represent a facial component and then transforms these feature vectors under the control of deforming parameters to generate the exaggerated facial features. The exaggerated facial animation can be driven to generate by the sequence of the deforming parameters. Experimental results prove that the proposed method has the advantages of simplicity, flexibility and directness, and the generated facial animations are expressive.

Multi-modal Emotion Recognition with Temporal-Band Attention Based on LSTM-RNN

Emotion recognition is a key problem in Human-Computer Interaction (HCI). The multi-modal emotion recognition was discussed based on untrimmed visual signals and EEG signals in this paper. We propose a model with two attention mechanisms based on multi-layer Long short-term memory recurrent neural network (LSTM-RNN) for emotion recognition, which combines temporal attention and band attention. At each time step, the LSTM-RNN takes the video and EEG slice as inputs and generate representations of two signals, which are fed into a multi-modal fusion unit. Based on the fusion, our network predicts the emotion label and the next time slice for analyzing. Within the process, the model applies different levels of attention to different frequency bands of EEG signals through the band attention. With the temporal attention, it determines where to analyze next signal in order to suppress the redundant information for recognition. Experiments on Mahnob-HCI database demonstrate the encouraging results; the proposed method achieves higher accuracy and boosts the computational efficiency.