Shu-Fan Wang

Fast JND-Based Video Carving With GPU Acceleration for Real-Time Video Retargeting

A recently developed image resizing technique, seam carving, has been proved to be a useful tool for content-adaptive spatially nonuniform image resizing with the purpose of optimal display on a screen of reduced resolution or different aspect ratio. In this paper, we present a fast algorithm for real-time content-aware video retargeting based on the improved seam carving method proposed in this paper. The proposed algorithm is designed to be highly parallelizable and suitable for running on a multicore architecture. First, two novel operators, i.e., seam update and seam split, are introduced to analyze an image for detecting the local and global seams with minimum costs very efficiently. With these operators, parallel processing can be achieved to determine multiple seams simultaneously. In addition, the saliency measure is extended with a just-noticeable-distortion model which makes the resized video more consistent with human perception. We demonstrate the efficiency of the above new components with a graphics processing unit (GPU) implementation. In addition, the proposed fast seam carving algorithm is extended for video retargeting. To the best of our knowledge, this is the first paper based on the seam carving method to achieve real-time video retargeting on a GPU. Experimental results on video sequences of various characteristics are demonstrated to show the superior performance of the proposed algorithm in comparison with the existing content-adaptive image/video resizing methods.

Fast structure-preserving image retargeting

Several different methods have been proposed for image/video retargeting while retaining the content. However, they sometimes produce some artifacts, such as ridge or structure twist. In this paper, we present a structure-preserving image resizing technique for the image retargeting applications. Based on the warping-based retargeting technique proposed by Wolf et al.[13], we propose an efficient and adaptive image resizing algorithm that preserves the content and image structure as best as possible. We first downsample the size of the original image by using bilinear interpolation. In order to preserve the content, we introduce the structure constraints derived from the line detection into the large linear system. Then, the mapping matrices are enlarged to the original size by joint-bilateral upsampling and the resized image can be produced to preserve the content and structure as best as possible. Most of the computation is on the low-resolution layer and therefore it can be very efficient. From our experiments, the proposed method can provide resized images with higher image quality and faster speed than that in [13].

Reconstructing 3D Face Model with Associated Expression Deformation from a Single Face Image via Constructing a Low-Dimensional Expression Deformation Manifold

Facial expression modeling is central to facial expression recognition and expression synthesis for facial animation. In this work, we propose a manifold-based 3D face reconstruction approach to estimating the 3D face model and the associated expression deformation from a single face image. With the proposed robust weighted feature map (RWF), we can obtain the dense correspondences between 3D face models and build a nonlinear 3D expression manifold from a large set of 3D facial expression models. Then a Gaussian mixture model in this manifold is learned to represent the distribution of expression deformation. By combining the merits of morphable neutral face model and the low-dimensional expression manifold, a novel algorithm is developed to reconstruct the 3D face geometry as well as the facial deformation from a single face image in an energy minimization framework. Experimental results on simulated and real images are shown to validate the effectiveness and accuracy of the proposed algorithm.

Compressibility-Aware Media Retargeting With Structure Preserving

A number of algorithms have been proposed for intelligent image/video retargeting with image content retained as much as possible. However, they usually suffer from some artifacts in the results, such as ridge or structure twist. In this paper, we present a structure-preserving media retargeting technique that preserves the content and image structure as best as possible. Different from the previous pixel or grid based methods, we estimate the image content saliency from the structure of the content. A block structure energy is introduced with a top-down strategy to constrain the image structure inside to deform uniformly in either x or y direction. However, the flexibilities for retargeting are quite different for different images. To cope with this problem, we propose a compressibility assessment scheme for media retargeting by combining the entropies of image gradient magnitude and orientation distributions. Thus, the resized media is produced to preserve the image content and structure as best as possible. Our experiments demonstrate that the proposed method provides resized images/videos with better preservation of content and structure than those by the previous methods.

Estimating 3D Face Model and Facial Deformation from a Single Image Based on Expression Manifold Optimization

Facial expression modeling is central to facial expression recognition and expression synthesis for facial animation. Previous works reported that modeling the facial expression with low-dimensional manifold is more appropriate than using a linear subspace. In this paper, we propose a manifold-based 3D face reconstruction approach to estimating the 3D face model and the associated expression deformation from a single face image. In the training phase, we build a nonlinear 3D expression manifold from a large set of 3D facial expression models to represent the facial shape deformations due to facial expressions. Then a Gaussian mixture model in this manifold is learned to represent the distribution of expression deformation. By combining the merits of morphable neutral face model and the low-dimensional expression manifold, we propose a new algorithm to reconstruct the 3D face geometry as well as the 3D shape deformation from a single face image with expression in an energy minimization framework. Experimental results on CMU-PIE image database and FG-Net video database are shown to validate the effectiveness and accuracy of the proposed algorithm.

Efficient 3d face reconstruction from a single 2d image by combining statistical and geometrical information

In this paper, we present an efficient algorithm for reconstructing 3D head model from a single 2D image based on using a 3D eigenhead model. This system is composed of two components, offline training of the eigenhead model and online reconstruction of a 3D head model. For the first part, we propose a new 3D head alignment algorithm based on an iterative coarse-to-fine scheme to establish dense point correspondences between 3D head model in the cylindrical coordinate to align the 3D head models in the training data set. In addition, we apply the radial basis function technique to establish dense correspondences between each 3D face model and a reference face model, followed by the principal component analysis technique to compute the statistical eigenhead model. For the 3D face reconstruction from a single image, the proposed algorithm finds the best linear combination of the eigenhead bases that minimizes an energy function composed of distances between the corresponding facial feature points and a one-way partial Haussdorf distance between the facial contours in the image domain. This energy minimization is accomplished by the iterative Levenberg-Marquardt algorithm with the initial guess determined by solving a linear system derived from the image projection constraints for the corresponding facial feature points. Experimental results show that the proposed 3D face reconstruction algorithm provides satisfactory results and takes less than 10 seconds on a regular PC.

Manifold-Based 3D Face Caricature Generation with Individualized Facial Feature Extraction

Caricature is an interesting art to express exaggerated views of different persons and things through drawing. The face caricature is popular and widely used for different applications. To do this, we have to properly extract unique/specialized features of a persons face. A persons facial feature not only depends on his/her natural appearance, but also the associated expression style. Therefore, we would like to extract the neutural facial features and personal expression style for different applicaions. In this paper, we represent the 3D neutral face models in BU–3DFE database by sparse signal decomposition in the training phase. With this decomposition, the sparse training data can be used for robust linear subspace modeling of public faces. For an input 3D face model, we fit the model and decompose the 3D model geometry into a neutral face and the expression deformation separately. The neutral geomertry can be further decomposed into public face and individualized facial feature. We exaggerate the facial features and the expressions by estimating the probability on the corresponding manifold. The public face, the exaggerated facial features and the exaggerated expression are combined to synthesize a 3D caricature for a 3D face model. The proposed algorithm is automatic and can effectively extract the individualized facial features from an input 3D face model to create 3D face caricature.

Technical Section: Image-based three-dimensional model reconstruction for Chinese treasure-Jadeite Cabbage with Insects

This paper presents a novel 3D reconstruction system for the famous Chinese treasure, Jadeite Cabbage with Insects, from uncalibrated image sequences. There are two major challenges for this 3D model reconstruction problem. The first is the difficult image registration problem due to the semi-diaphaneity and the highly specular property of jadeite materials. Secondly, the unknown camera information, including the intrinsic (calibration) and extrinsic (position and orientation) parameters, to be recovered from the uncalibrated image sequences makes the 3D reconstruction problem very challenging. The proposed 3D modeling process first recovers the camera information as well as sparse 3D structure by using a robust structure from motion algorithm. Then an approximate 3D object model is recovered from the silhouettes at the corresponding multiple views by using the visual hull technique. The final process refines the 3D model by further integrating the 3D information extracted from dense correspondences between image patches of different views. In the proposed 3D reconstruction system, we successfully combine the structure from motion and visual hull techniques to accomplish this challenging task of reconstructing an accurate 3D model for jadeite object from uncalibrated multi-view images. Finally, we assess the 3D reconstruction results for the Chinese jadeite treasure and simulated data by using the proposed 3D reconstruction system.

Surface simplification by image retargeting

Surface simplification aims to reduce the complexity of a 3D model while maintaining a good approximation to the original model. In this work, we propose a novel combination of geometry images and content-aware image resizing to achieve efficient surface simplification. There are two main advantages to simplify surface based on geometry images. First, it is relatively simple to simplify the surface in the parameterized 2D space because the features of a 3D surface can be easily represented by the gradient energy. Second, the regularity and features on 3D surface can also be preserved without additional effort. The proposed retargeting algorithm performs well both on real images and 3D surface simplification.

Parallelized Face Based RMS System on a Multi-core Embedded Computing Platform

A new framework for the Recognition, Mining and Synthesis (RMS)system, has been proposed to make meaningful use of the enormous amount of information. Based on the same concept, we propose a face RMS system, which consists of face detection, facial expression recognition, and facial expression exaggeration components, for generating exaggerated views of different expressions for an input face video. In this paper, the parallel algorithms of the face RMS system were developed to reduce the execution time on a multi-core embedded system. The experimental results show the robustness and efficiency of face RMS system under complex environments. The quantitative comparisons indicate the proposed parallelized strategies has a significant increase in computational speedup compared to the single-processor implementation on a multi-core embedded platform.