Yap-Peng Tan

Color filter array demosaicking: new method and performance measures

Single-sensor digital cameras capture imagery by covering the sensor surface with a color filter array (CFA) such that each sensor pixel only samples one of three primary color values. To render a full-color image, an interpolation process, commonly referred to as CFA demosaicking, is required to estimate the other two missing color values at each pixel. In this paper, we present two contributions to the CFA demosaicking: a new and improved CFA demosaicking method for producing high quality color images and new image measures for quantifying the performance of demosaicking methods. The proposed demosaicking method consists of two successive steps: an interpolation step that estimates missing color values by exploiting spatial and spectral correlations among neighboring pixels, and a post-processing step that suppresses noticeable demosaicking artifacts by adaptive median filtering. Moreover, in recognition of the limitations of current image measures, we propose two types of image measures to quantify the performance of different demosaicking methods; the first type evaluates the fidelity of demosaicked images by computing the peak signal-to-noise ratio and CIELAB DeltaE(*)(ab) for edge and smooth regions separately, and the second type accounts for one major demosaicking artifact-zipper effect. We gauge the proposed demosaicking method and image measures using several existing methods as benchmarks, and demonstrate their efficacy using a variety of test images.

Discriminative Deep Metric Learning for Face Verification in the Wild

This paper presents a new discriminative deep metric learning (DDML) method for face verification in the wild. Different from existing metric learning-based face verification methods which aim to learn a Mahalanobis distance metric to maximize the inter-class variations and minimize the intra-class variations, simultaneously, the proposed DDML trains a deep neural network which learns a set of hierarchical nonlinear transformations to project face pairs into the same feature subspace, under which the distance of each positive face pair is less than a smaller threshold and that of each negative pair is higher than a larger threshold, respectively, so that discriminative information can be exploited in the deep network. Our method achieves very competitive face verification performance on the widely used LFW and YouTube Faces (YTF) datasets.

Rapid estimation of camera motion from compressed video with application to video annotation

As digital video becomes more pervasive, efficient ways of searching and annotating video according to content will be increasingly important. Such tasks arise, for example, in the management of digital video libraries for content-based retrieval and browsing. We develop tools based on camera motion for analyzing and annotating a class of structured video using the low-level information available directly from MPEG-compressed video. In particular, we show that in certain structured settings, it is possible to obtain reliable estimates of camera motion by directly processing data easily obtained from the MPEG format. Working directly with the compressed video greatly reduces the processing time and enhances storage efficiency. As an illustration of this idea, we have developed a simple basketball annotation system which combines the low-level information extracted from an MPEG stream with the prior knowledge of basketball structure to provide high-level content analysis, annotation, and browsing for events such as wide-angle and close-up views, fast breaks, probable shots at the basket, etc. The methods used in this example should also be useful in the analysis of high-level content of structured video in other domains.

Discriminative Multimanifold Analysis for Face Recognition from a Single Training Sample per Person

Conventional appearance-based face recognition methods usually assume that there are multiple samples per person (MSPP) available for discriminative feature extraction during the training phase. In many practical face recognition applications such as law enhancement, e-passport, and ID card identification, this assumption, however, may not hold as there is only a single sample per person (SSPP) enrolled or recorded in these systems. Many popular face recognition methods fail to work well in this scenario because there are not enough samples for discriminant learning. To address this problem, we propose in this paper a novel discriminative multimanifold analysis (DMMA) method by learning discriminative features from image patches. First, we partition each enrolled face image into several nonoverlapping patches to form an image set for each sample per person. Then, we formulate the SSPP face recognition as a manifold-manifold matching problem and learn multiple DMMA feature spaces to maximize the manifold margins of different persons. Finally, we present a reconstruction-based manifold-manifold distance to identify the unlabeled subjects. Experimental results on three widely used face databases are presented to demonstrate the efficacy of the proposed approach.

Neighborhood repulsed metric learning for kinship verification

Kinship verification from facial images is a challenging problem in computer vision, and there is a very few attempts on tackling this problem in the literature. In this paper, we propose a new neighborhood repulsed metric learning (NRML) method for kinship verification. Motivated by the fact that interclass samples (without kinship relations) with higher similarity usually lie in a neighborhood and are more easily misclassified than those with lower similarity, we aim to learn a distance metric under which the intraclass samples (with kinship relations) are pushed as close as possible and interclass samples lying in a neighborhood are repulsed and pulled as far as possible, simultaneously, such that more discriminative information can be exploited for verification. Moreover, we propose a multiview NRM-L (MNRML) method to seek a common distance metric to make better use of multiple feature descriptors to further improve the verification performance. Experimental results are presented to demonstrate the efficacy of the proposed methods.

Adaptive Filtering for Color Filter Array Demosaicking

Most digital still cameras acquire imagery with a color filter array (CFA), sampling only one color value for each pixel and interpolating the other two color values afterwards. The interpolation process is commonly known as demosaicking. In general, a good demosaicking method should preserve the high-frequency information of imagery as much as possible, since such information is essential for image visual quality. We discuss in this paper two key observations for preserving high-frequency information in CFA demosaicking: (1) the high frequencies are similar across three color components, and 2) the high frequencies along the horizontal and vertical axes are essential for image quality. Our frequency analysis of CFA samples indicates that filtering a CFA image can better preserve high frequencies than filtering each color component separately. This motivates us to design an efficient filter for estimating the luminance at green pixels of the CFA image and devise an adaptive filtering approach to estimating the luminance at red and blue pixels. Experimental results on simulated CFA images, as well as raw CFA data, verify that the proposed method outperforms the existing state-of-the-art methods both visually and in terms of peak signal-to-noise ratio, at a notably lower computational cost.

Effective use of spatial and spectral correlations for color filter array demosaicking

To minimize cost and size, most commercial digital cameras acquire imagery using a single electronic sensor (CCD or CMOS) overlaid with a color filter array (CFA) such that each sensor pixel only samples one of the three primary color values. To restore a full-color image from CFA samples, the two missing color values at each pixel need to be estimated from the neighboring samples, a process that is commonly known as CFA demosaicking or interpolation. In this paper we present two contributions to CFA demosaicking. First, we stress the importance of well exploiting both image spatial and spectral correlations, and characterize the demosaicking artifacts due to inadequate use of either correlation. Second, based on the insights gained from our empirical study, we propose effective schemes to enhance two existing state-of-the-art demosaicking methods. Experimental results show that our enhanced methods achieve notable improvements over the existing methods, in terms of both subjective and objective evaluations, on a large variety of test images. In addition, the computational complexities of the enhanced methods are comparable to the originals.

Regularized Locality Preserving Projections and Its Extensions for Face Recognition

We propose in this paper a parametric regularized locality preserving projections (LPP) method for face recognition. Our objective is to regulate the LPP space in a parametric manner and extract useful discriminant information from the whole feature space rather than a reduced projection subspace of principal component analysis. This results in better locality preserving power and higher recognition accuracy than the original LPP method. Moreover, the proposed regularization method can easily be extended to other manifold learning algorithms and to effectively address the small sample size problem. Experimental results on two widely used face databases demonstrate the efficacy of the proposed method.

Automated analysis and annotation of basketball video

Automated analysis and annotation of video sequences are important for digital video libraries, content-based video browsing and data mining projects. A successful video annotation system should provide users with useful video content summary in a reasonable processing time. Given the wide variety of video genres available today, automatically extracting meaningful video content for annotation still remains hard by using current available techniques. However, a wide range video has inherent structure such that some prior knowledge about the video content can be exploited to improve our understanding of the high-level video semantic content. In this paper, we develop tools and techniques for analyzing structured video by using the low-level information available directly from MPEG compressed video. Being able to work directly in the video compressed domain can greatly reduce the processing time and enhance storage efficiency. As a testbed, we have developed a basketball annotation system which combines the low-level information extracted from MPEG stream with the prior knowledge of basketball video structure to provide high level content analysis, annotation and browsing for events such as wide- angle and close-up views, fast breaks, steals, potential shots, number of possessions and possession times. We expect our approach can also be extended to structured video in other domains.

A framework for measuring video similarity and its application to video query by example

The usefulness of a video database relies on whether the video of interest can be easily located. To allow exploring, browsing, and retrieving videos according to their visual content, efficient techniques for evaluating the visual similarity between different video clips are necessary. We present a framework for measuring video similarity across different resolutions-both spatial and temporal. In particular, the video clips to be compared can be properly aligned through the use of suitable weighting functions and alignment constraints. Dynamic programming techniques are employed to obtain the video similarity measure with a reasonable computational cost. An application to searching MPEG compressed video by example is presented to demonstrate the potential use of the proposed video similarity measure.