Hongcheng Wang

Out-of-core tensor approximation of multi-dimensional matrices of visual data

Tensor approximation is necessary to obtain compact multilinear models for multi-dimensional visual datasets. Traditionally, each multi-dimensional data item is represented as a vector. Such a scheme flattens the data and partially destroys the internal structures established throughout the multiple dimensions. In this paper, we retain the original dimensionality of the data items to more effectively exploit existing spatial redundancy and allow more efficient computation. Since the size of visual datasets can easily exceed the memory capacity of a single machine, we also present an out-of-core algorithm for higher-order tensor approximation. The basic idea is to partition a tensor into smaller blocks and perform tensor-related operations blockwise. We have successfully applied our techniques to three graphics-related data-driven models, including 6D bidirectional texture functions, 7D dynamic BTFs and 4D volume simulation sequences. Experimental results indicate that our techniques can not only process out-of-core data, but also achieve higher compression ratios and quality than previous methods.

Rank-R approximation of tensors using image-as-matrix representation

We present a novel multilinear algebra based approach for reduced dimensionality representation of image ensembles. We treat an image as a matrix, instead of a vector as in traditional dimensionality reduction techniques like PCA, and higher-dimensional data as a tensor. This helps exploit spatio-temporal redundancies with less information loss than image-as-vector methods. The challenges lie in the computational and memory requirements for large ensembles. Currently, there exists a rank-R approximation algorithm which, although applicable to any number of dimensions, is efficient for only low-rank approximations. For larger dimensionality reductions, the memory and time costs of this algorithm become prohibitive. We propose a novel algorithm, for rank-R approximations of third-order tensors, which is efficient for arbitrary R but for the important special case of 2D image ensembles, e.g. video. Both of these algorithms reduce redundancies present in all dimensions. Rank-R tensor approximation yields the most compact data representation among all known image-as-matrix methods. We evaluated the performance of our algorithm vs. other approaches on a number of datasets with the following two main results. First, for a fixed compression ratio, the proposed algorithm yields the best representation of image ensembles visually as well as in the least squares sense. Second, proposed representation gives the best performance for object classification.

Compact representation of multidimensional data using tensor rank-one decomposition

This paper presents a new approach for representing multidimensional data by a compact number of bases. We consider the multidimensional data as tensors instead of matrices or vectors, and propose a tensor rank-one decomposition (TROD) algorithm by decomposing Nth-order data into a collection of rank-1 tensors based on multilinear algebra. By applying this algorithm to image sequence compression, we obtain much higher quality images with the same compression ratio as principal component analysis (PCA). Experiments with gray-level and color video sequences are used to illustrate the validity of this approach.

A Tensor Approximation Approach to Dimensionality Reduction

AbstractnDimensionality reduction has recently been extensively studied for computer vision applications. We present axa0novel multilinear algebra based approach to reduced dimensionality representation of multidimensional data, such as image ensembles, video sequences and volume data. Before reducing the dimensionality we do not convert it into axa0vector as is done by traditional dimensionality reduction techniques like PCA. Our approach works directly on the multidimensional form of the data (matrix in 2D and tensor in higher dimensions) to yield what we call axa0Datum-as-Is representation. This helps exploit spatio-temporal redundancies with less information loss than image-as-vector methods. An efficient rank-R tensor approximation algorithm is presented to approximate higher-order tensors. We show that rank-R tensor approximation using Datum-as-Is representation generalizes many existing approaches that use image-as-matrix representation, such as generalized low rank approximation of matrices (GLRAM)xa0(Ye, Y. in Mach. Learn. 61:167–191, 2005), rank-one decomposition of matrices (RODM)xa0(Shashua, A., Levin, A. in CVPR’01: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition, p.xa042, 2001) and rank-one decomposition of tensors (RODT)xa0(Wang, H., Ahuja, N. in ICPR ’04: ICPR ’04: Proceedings of the 17th international conference on pattern recognition (ICPR’04), vol.xa01, pp.xa044–47, 2004). Our approach yields the most compact data representation among all known image-as-matrix methods. In addition, we propose another rank-R tensor approximation algorithm based on slice projection of third-order tensors, which needs fewer iterations for convergence for the important special case of 2D image ensembles, e.g., video. We evaluated the performance of our approach vs. other approaches on axa0number of datasets with the following two main results. First, for axa0fixed compression ratio, the proposed algorithm yields the best representation of image ensembles visually as well as in the least squares sense. Second, proposed representation gives the best performance for object classification.n

Hierarchical Tensor Approximation of Multi-Dimensional Visual Data

Visual data comprise of multiscale and inhomogeneous signals. In this paper, we exploit these characteristics and develop a compact data representation technique based on a hierarchical tensor-based transformation. In this technique, an original multidimensional data set is transformed into a hierarchy of signals to expose its multiscale structures. The signal at each level of the hierarchy is further divided into a number of smaller tensors to expose its spatially inhomogeneous structures. These smaller tensors are further transformed and pruned using a tensor approximation technique. Our hierarchical tensor approximation supports progressive transmission and partial decompression. Experimental results indicate that our technique can achieve higher compression ratios and quality than previous methods, including wavelet transforms, wavelet packet transforms, and single-level tensor approximation. We have successfully applied our technique to multiple tasks involving multidimensional visual data, including medical and scientific data visualization, data-driven rendering, and texture synthesis.

Seamless video editing

This paper presents a new framework for seamless video editing in the gradient domain. The spatio-temporal gradient fields of target videos are modified or mixed to generate a new gradient field, which is usually not integrable. We propose a 3D video integration algorithm, which finds a potential function, whose gradient field is closest to the resulting gradient field in the sense of least squares. The video is reconstructed by solving a 3D Poisson equation. We use a fast and accurate 3D discrete Poisson solver using diagonal multigrids. A set of gradient operators is defined for user interaction. The resulting video has temporal coherency and no artifacts. We evaluate our algorithm using a variety of examples.

Gradient Adaptive Image Restoration and Enhancement

Various methods have been proposed for image enhancement and restoration. The main difficulty is how to enhance the structures uniformly while suppressing the noise without artifacts. In this paper, we tackle this problem in the gradient domain instead of the traditional intensity domain. By enhancing the gradient field, we can enhance the structure uniformly without overshooting at the boundary. Because the gradient field is very sensitive to noise, we apply an orientation-isotropy adaptive filter to the gradient field, suppressing the gradients in the noise regions while enhancing along the object boundaries. Thus we obtain a modulated gradient field, which is usually not integrable. We reconstruct the enhanced image from the modulated gradient field with least square errors by solving a Poisson equation. This method can enhance the object contrast uniformly, suppress the noise with no artifacts, and avoid setting stopping time as in PDE methods. Experiments on noisy images show the efficacy of our method.

Videoshop: A new framework for spatio-temporal video editing in gradient domain

This paper proposes a new framework for video editing in gradient domain. The spatio-temporal gradient fields of target videos are modified and/or mixed to generate a new gradient field which is usually not integrable. We compare two methods to solve this mixed gradient problem, i.e., the variational method and loopy belief propagation. We propose a 3D video integration algorithm, which uses the variational method to find the potential function whose gradient field is closest to the mixed gradient field in the sense of least squares. The video is reconstructed by solving a 3D Poisson equation. The main contributions of our framework lie in three aspects: first, we derive a straightforward extension of current 2D gradient technique to 3D space, thus resulting in a novel video editing framework, which is very different from all current video editing software; secondly, we propose using a fast and accurate 3D discrete Poisson solver which uses diagonal multigrids to solve the 3D Poisson equation, which is up to twice as fast as a simple conventional multigrid algorithm; finally, we introduce a set of new applications, such as face replacement and painting, high dynamic range video compression and graphcut based video compositing. A set of gradient operators is also provided to the user for editing purposes. We evaluate our algorithm using a variety of examples for image/video or video/video pairs. The resulting video can be seamlessly reconstructed.

Sparse lumigraph relighting by illumination and reflectance estimation from multi-view images

We present a novel relighting approach that does not assume that the illumination is known or controllable. Instead, we estimate the illumination and texture from given multi-view images captured under a single illumination setting, given the object shape. We rely on the viewpoint-dependence of surface reflectance to resolve the usual texture-illumination ambiguity. The task of obtaining the illumination and texture models is formulated as the decomposition of the observed surface radiance tensor into the product of a light transport tensor, and illumination and texture matrices. We estimate both the illumination and texture at the same time by solving a system of bilinear equations. To reduce estimation error due to imperfect input surface geometry, we also perform a multi-scale discrete search on the specular surface normal. Our results on synthetic and real data indicate that we can estimate the illumination, the diffuse as well as the specular components of the surface texture map (up to a global scaling ambiguity). Our approach allows more flexibilities in rendering novel images, such as view changing, and light and texture editing.

Videoshop: a new framework for spatio-temporal video editing in gradient domain

Our goal is to develop tools that go beyond frame-constrained manipulation such as resizing, color correction, and simple transitions, and provide object-level operations within frames. Some of our targeted video editing tasks includes transferring a motion picture to a new still picture, importing a moving object into a new background, and compositing two video sequences. The challenges behind this kind of complex video editing tasks lie in two constraints: 1) Spatial consistency: imported objects should blend with the background seamlessly. Hence pixel replacement, which creates noticeable seams, is problematic. 2) Temporal coherency: successive frames should display smooth transitions. Hence frame-by-frame editing, which results in visual flicker, is inappropriate. Our work is aimed at providing an easy-to-use video editing tool that maximally satisfies the spatial and temporal constraints mentioned above and requires minimum user interaction. We propose a new framework for video editing in gradient domain. The spatio-temporal gradient fields of target videos are modified and/or mixed to generate a new gradient field which is usually not integrable. We propose a 3D video integration algorithm, which uses the variational method, to find the potential function whose gradient field is closest to the mixed gradient field in the sense of least squares. The video is reconstructed by solving a 3D Poisson equation. We derive an extension of current 2D gradient technique to 3D space, yielding in a novel video editing framework, which is very different from all current video editing software.