Marcelo Bernardes Vieira

Automatic 3d facial expression analysis in videos

We introduce a novel framework for automatic 3D facial expression analysis in videos. Preliminary results demonstrate editing facial expression with facial expression recognition. We first build a 3D expression database to learn the expression space of a human face. The real-time 3D video data were captured by a camera/projector scanning system. From this database, we extract the geometry deformation independent of pose and illumination changes. All possible facial deformations of an individual make a nonlinear manifold embedded in a high dimensional space. To combine the manifolds of different subjects that vary significantly and are usually hard to align, we transfer the facial deformations in all training videos to one standard model. Lipschitz embedding embeds the normalized deformation of the standard model in a low dimensional generalized manifold. We learn a probabilistic expression model on the generalized manifold. To edit a facial expression of a new subject in 3D videos, the system searches over this generalized manifold for optimal replacement with the ‘target’ expression, which will be blended with the deformation in the previous frames to synthesize images of the new expression with the current head pose. Experimental results show that our method works effectively.

A Camera-Projector System for Real-Time 3D Video

In this paper, we describe a real-time 3D video system that is based on active stereo. Combining a NTSC standard camera/ projector equipment and a suitable color code, the geometric and photometric information of a scene is robustly obtained in 30fps. Our main motivation to develop this system is to create a platform for investigating the issues that will be posed by the next generation of digital video and how it can shape up new media.

Segmentation into fuzzy regions using topographic distance

This paper exposes an algorithm which leads to a fuzzy segmentation. This algorithm performs, as in the watershed method, a progressive flood of the gradient image from pixels of lowest gradients. It uses a new distance, called topographic distance. Any local minimum of the gradient norm image constitutes a seed for the region growing, avoiding the use of a marker image. These seeds constitute the cores of the initial fuzzy regions. Then the sites are gradually agglomerated to the region, while their membership degrees to the region decrease, according to the distance to the core and to the gradient norms, by the way of the topographic distance. The numerous fuzzy regions are then merged and the membership degrees of pixels to final regions are computed. Applications concern crisp segmentation of colour or gray scale images and pattern recognition from fuzzy regions.

A tensor motion descriptor based on histograms of gradients and optical flow

This paper presents a new tensor motion descriptor only using optical flow and HOG3D information: no interest points are extracted and it is not based on a visual dictionary. We propose a new aggregation technique based on tensors. This is a double aggregation of tensor descriptors. The first one represents motion by using polynomial coefficients which approximates the optical flow. The other represents the accumulated data of all histograms of gradients of the video. The descriptor is evaluated by a classification of KTH, UCF11 and Hollywood2 datasets, using a SVM classifier. Our method reaches 93.2% of recognition rate with KTH, comparable to the best local approaches. For the UCF11 and Hollywood2 datasets, our recognition achieves fairly competitive results compared to local and learning based approaches.

A Tensor Based on Optical Flow for Global Description of Motion in Videos

Motion is one of the main characteristics that describe the semantic information of videos. In this work, a global video descriptor based on orientation tensors is proposed. This descriptor is obtained by combining polynomial coefficients calculated for each image in a video. The coefficients are found through the projection of the optical flow on Legendre polynomials, reducing the dimension of per frame motion estimations. The sequence of coefficients are then combined using orientation tensors. The global tensor descriptor created is evaluated by a classification of the KTH video database with a SVM classifier.

Range-enhanced active foreground extraction

We describe a new technique that uses active scene illumination to perform foreground-background segmentation and recover partial HDR information. We explore the fact that relative tones can be recovered by varying illumination intensity, without knowing the camera response function. In our approach, the scene is illuminated with an uncalibrated projector and two images of the scene are captured under different illumination conditions. By taking advantage of the fact that the projector can be set up to illuminate only the foreground, we are able to distinguish the foreground from the background. The output of our system is a segmentation mask, together with a image with additional tonal information for the foreground pixels. As an application, we show how to produce spatially variant tone mapped images, where background and foreground receive different treatments. The segmentation and the visualization algorithms are implemented in real-time, and can be used to produce range-enhanced video sequences.

Combining Orientation Tensors for Human Action Recognition

This paper presents a new tensor motion descriptor based on histogram of oriented gradients. We model the temporal evolution of gradient distribution with orientation tensors in equally sized blocks throughout the video sequence. Subsequently, these blocks are concatenated to create the final descriptor. Using a SVM classifier, even without any bag-of-feature based approach, our method achieves recognition rates greater than those found by other HOG techniques on KTH dataset and a competitive recognition rate for UCF11 and Hollywood2 datasets.

Using graph cuts in GPUs for color based human skin segmentation

In this paper we propose a new method to deal with the problem of automatic human skin segmentation in RGB color space model. The problem is modeled as a minimum cost graph cut problem on a graph whose vertices represent the image color characteristics. Skin and non-skin elements are assigned by evaluating label costs of vertices associated to the weight edges of the graph. A novel approach based on an energy function defined in terms of a database of skin and non-skin tones is used to define the costs of the edges of the graph. Finally, the graph cut problem is solved in Graphics Processing Units (GPU) using the Compute Unified Device Architecture (CUDA) technology yielding very promising skin segmentation results for standard resolution video sequences. Our method was evaluated under several conditions, indicating when correct or incorrect results are generated. The overall experiments have shown that this automatic method is simple, efficient, and yields very reliable results.

Tracking and Matching Connected Components from 3D Video

This work presents a method for the detection, tracking and spatial matching of connected components in a 3D video stream. The video image information is combined with 3D sites in order to align pieces of surfaces that are seen in subsequent frames. This is a key step in 3D video analysis for enabling several applications such as compression, geometric integration and scene reconstruction, to name a few. Our approach is to detect salient features in both image and world spaces for further alignment of texture and geometry. We use a projector-camera system to obtain high quality texture and geometry at 30 fps. Successful experimental results corroborating our method are shown.

A Tensor Motion Descriptor Based on Multiple Gradient Estimators

This work presents a novel approach for motion description in videos using multiple band-pass filters which act as first order derivative estimators. The filters response on each frame are coded into individual histograms of gradients to reduce their dimensionality. They are combined using orientation tensors. No local features are extracted and no learning is performed, i.e., the descriptor depends uniquely on the input video. Motion description can be enhanced even using multiple filters with similar or overlapping frequency response. For the problem of human action recognition using the KTH database, our descriptor achieved the recognition rate of 93.3% using three Daubechies filters, one extra filter designed to correlate them, two-fold protocol and a SVM classifier. It is superior to most global descriptor approaches and fairly comparable to the state-of-the-art methods.