Chinh T. Dang

Heterogeneity Image Patch Index and Its Application to Consumer Video Summarization

Automatic video summarization is indispensable for fast browsing and efficient management of large video libraries. In this paper, we introduce an image feature that we refer to as heterogeneity image patch (HIP) index. The proposed HIP index provides a new entropy-based measure of the heterogeneity of patches within any picture. By evaluating this index for every frame in a video sequence, we generate a HIP curve for that sequence. We exploit the HIP curve in solving two categories of video summarization applications: key frame extraction and dynamic video skimming. Under the key frame extraction framework, a set of candidate key frames is selected from abundant video frames based on the HIP curve. Then, a proposed patch-based image dissimilarity measure is used to create affinity matrix of these candidates. Finally, a set of key frames is extracted from the affinity matrix using a min-max based algorithm. Under video skimming, we propose a method to measure the distance between a video and its skimmed representation. The video skimming problem is then mapped into an optimization framework and solved by minimizing a HIP-based distance for a set of extracted excerpts. The HIP framework is pixel-based and does not require semantic information or complex camera motion estimation. Our simulation results are based on experiments performed on consumer videos and are compared with state-of-the-art methods. It is shown that the HIP approach outperforms other leading methods, while maintaining low complexity.

Key frame extraction from consumer videos using epitome

Key frame extraction algorithms select a subset of the most informative frames from videos. Key frame extraction finds applications in several broad areas of video processing research such as video summarization, video indexing, and prints from video. In this paper, an image epitome [1][2] based method to extract key frames from unstructured consumer videos is presented. In the proposed approach, we exploit image epitome to measure dissimilarity between frames of the input video. The dissimilarity scores are further analyzed using a min-max approach to extract the desired number of key frames from the input video. The proposed approach does not require shot(s) detection, segmentation, or semantic understanding. A comparison of the results obtained by this method with the ground truth agreed by multiple judges clearly indicates the feasibility of the proposed approach.

Image Super-Resolution via Local Self-Learning Manifold Approximation

This letter proposes a novel learning-based super-resolution method rooted in low dimensional manifold representations of high-resolution (HR) image-patch spaces. We exploit the input image and its different down-sampled scales to extract a set of training sample points using a min-max algorithm. A set of low dimensional tangent spaces is estimated from these samples using the l1 norm graph-based technique to cluster these samples into a set of manifold neighborhoods. The HR image is then reconstructed from these tangent spaces. Experimental results on standard images validate the effectiveness of the proposed method both quantitatively and perceptually.

RPCA-KFE: Key Frame Extraction for Video Using Robust Principal Component Analysis

Key frame extraction algorithms consider the problem of selecting a subset of the most informative frames from a video to summarize its content. Several applications, such as video summarization, search, indexing, and prints from video, can benefit from extracted key frames of the video under consideration. Most approaches in this class of algorithms work directly with the input video data set, without considering the underlying low-rank structure of the data set. Other algorithms exploit the low-rank component only, ignoring the other key information in the video. In this paper, a novel key frame extraction framework based on robust principal component analysis (RPCA) is proposed. Furthermore, we target the challenging application of extracting key frames from unstructured consumer videos. The proposed framework is motivated by the observation that the RPCA decomposes an input data into: 1) a low-rank component that reveals the systematic information across the elements of the data set and 2) a set of sparse components each of which containing distinct information about each element in the same data set. The two information types are combined into a single

Fast image super-resolution via selective manifold learning of high-resolution patches

\ell _{1}

Fast Single-Image Super-Resolution Via Tangent Space Learning of High-Resolution-Patch Manifold

-norm-based non-convex optimization problem to extract the desired number of key frames. Moreover, we develop a novel iterative algorithm to solve this optimization problem. The proposed RPCA-based framework does not require shot(s) detection, segmentation, or semantic understanding of the underlying video. Finally, experiments are performed on a variety of consumer and other types of videos. A comparison of the results obtained by our method with the ground truth and with related state-of-the-art algorithms clearly illustrates the viability of the proposed RPCA-based framework.

Image super-resolution via Dual-Manifold Clustering and Subspace Similarity

This paper considers the problem of single image super-resolution (SR). Previous example-based SR approaches mainly focus on analyzing the co-occurrence properties of low resolution (LR) and high resolution (HR) patches via dictionary learning. In our recent work [1], a novel approach (SR via sparse subspace clustering-based linear approximation of manifold or SLAM) has been proposed. In this paper, we further improve the SLAM method by considering and analyzing each tangent subspace as one point in a Grassmann manifold to select an optimal subset of tangent spaces. Furthermore, the optimal subset is clustered hierarchically, which helps in reducing the proposed algorithms complexity significantly while still preserving the quality of the reconstructed HR image.

Wind speed and direction estimation using manifold approximation

Manifold assumption has been used in example-based super-resolution (SR) reconstruction from a single frame. Previous manifold-based SR approaches (more generally example-based SR) mainly focus on analyzing the co-occurrence properties of low-resolution and high-resolution patches. This paper develops a novel single-image SR approach based on linear approximation of the high-resolution-patch space using a sparse subspace clustering algorithm. The approach exploits the underlying high-resolution patches nonlinear space by considering it as a low-dimensional manifold in a high-dimensional Euclidean space, and by considering each training high-resolution-patch as a sample from the manifold. We utilize the sparse subspace clustering algorithm to create the set of low-dimensional linear spaces that are considered, approximately, as tangent spaces at the high-resolution samples. Furthermore, we consider and analyze each tangent space as one point in a Grassmann manifold, which helps to compute geodesic pairwise distances among these tangent spaces. An optimal subset of these tangent spaces is then selected using a min-max algorithm. The optimal subset reduces the computational cost in comparison with using the full set of tangent spaces while still preserving the quality of the high-resolution image reconstruction. In addition, we perform hierarchical clustering on the optimal subset based on the geodesic distance, which helps to further achieve much faster SR algorithm. We also analytically prove the validity of the geodesic distance based clustering under the proposed framework. A comparison of the obtained results with other related methods in both high-resolution image quality and computational complexity clearly indicates the viability of the proposed framework.

Representative selection for big data via sparse graph and geodesic Grassmann manifold distance

In this paper, we consider the problem of example based single image super-resolution. Our main contribution is introducing a new framework that makes no assumption about the structural similarity between the high-resolution (HR) and low-resolution (LR) manifolds. Instead, we use a subspace affinity measure to exploit the similarity between each HR and LR subspace. First, we train both LR and HR manifolds independently, and then, by using subspace similarity we find the closest HR subspace to each LR subspace. Each patch from the LR test image is projected onto the LR trained manifold to find the closest LR subspace. Finally, the corresponding HR subspace is selected to reconstruct the HR version of the test patch. We refer to the proposed framework Dual-Manifold Clustering and Subspace Similarity (DMCSS). The experimental results showed that DMCSS achieves clear visual improvements and an average of 1dB improvement in PSNR over state-of-the-art algorithms in this field.

Single image super resolution via manifold linear approximation using sparse subspace clustering

In this paper, we describe a novel manifold-based interpolation method for sensed environmental data. Furthermore, we present initial results for applying the proposed method to estimate wind speed and direction around Lake Michigan. The proposed method is showing promising results based on the hypothesis that an environmental dataset (including longitude, latitude time, and measured parameters) can be mapped onto an underlying differential manifold. Our preliminary results show that the proposed manifold-based approach outperforms state-of-the-art interpolation and estimation methods.