Songze Tang

Edge and color preserving single image superresolution

Abstract. Most existing superresolution (SR) techniques focus primarily on improving the quality in the luminance component of SR images, while paying less attention to the chrominance component. We present an edge and color preserving image SR approach. First, for the luminance channel, a heavy-tailed gradient distribution of natural images is investigated as an image prior. Then, an efficient optimization algorithm is developed to recover the latent high-resolution (HR) luminance component. Second, for the chrominance channels, we propose a two-stage framework for luminance-guided chrominance SR. In the first stage, since most of the shape and structural information is contained in the luminance channel, a simple Markov random field formulation is introduced to search the optimal direction for color local interpolation guided by HR luminance components. To further improve the quality of the chrominance channels, in the second stage, a nonlocal auto regression model is utilized to refine the initial HR chrominance. Finally, we combine the SR reconstructed luminance components with the generated HR chrominance maps to get the final SR color image. Systematic experimental results demonstrated that our method outperforms some state-of-the-art methods in terms of the peak signal-to-noise ratio, structural similarity, feature similarity, and the mean color errors.

A New Geometry Enforcing Variational Model for Pan-Sharpening

In this paper, a new variational method for pan-sharpening is proposed to obtain a high-resolution multispectral (MS) image from a low-resolution MS image and a high-resolution panchromatic (PAN) image. In addition to using the data generative fidelity term and wavelet-based spectral information preserving term, we also associate the Hessian structural information of the PAN image with the desired pan-sharpened MS image to enforce geometry correspondence in the fusion process. More specifically, we introduce a new geometry enforcing term called “vectorial Hessian feature consistence” and combine it with the data generative fidelity term and wavelet-based spectral information preserving term to form an unified variational model for pan-sharpening. Then, the optimal solution of the proposed variational pan-sharpening model is effectively obtained using the fast iterative shrinkage thresholding algorithm (FISTA) method. In addition to well preserving spectral information, our algorithm is also able to eliminate some undesired blocky or blurry artifacts by incorporating the curvature information. Experimental results demonstrate that the proposed method outperforms various well-known pan-sharpening methods in terms of both excellent spatial and spectral qualities.

Partial least-squares regression on common feature space for single image superresolution

We proposed a superresolution (SR) method based on example-learning framework. In our framework, the relationship between the output high-resolution (HR) estimation and the HR training images is approximated by the relationship between the low-resolution (LR) test image and the HR training images. To effectively capture the strong correlation between LR and HR images, the LR and HR images are mapped onto a common feature space. Furthermore, in order to maintain their original two-dimensional (2-D) spatial structure, the original LR and HR patches are mapped onto the underlying common feature space using 2-D canonical correlation analysis. Later, the relationship between HR and LR features is established by partial least squares (PLS) with low regression errors on the derived feature space. In addition, a steering kernel regression (SKR) constraint is integrated into patch aggregation to improve the quality of the recovered images. Finally, the effectiveness of our approach is validated by extensive experimental comparisons with several SR algorithms for the natural image superresolution both quantitatively and qualitatively.

Pan-sharpening using 2D CCA

Pan-sharpening is a multisource fusion process which combines a low-resolution multispectral (LRM) image with a high-resolution panchromatic (HRP) image to fuse a high-resolution multispectral (HRM) image. However, the previous methods only focused on the original spatial space or simple feature space without considering the strong correlations between HRM and HRP images. In this paper, a novel pan-sharpening method on a common feature space is proposed based on two-dimensional canonical correlation analysis (2D CCA). 2D CCA is first used to train four projection matrices from the HRM training images, the HRP training images and their degraded ones. A common feature space is then established by maximizing the statistical correlations between intrinsic structures of low- and high-resolution images. Then the k-nearest neighbour selection of the input LRM image patches is conducted in the derived feature space to estimate the reconstruction weights. Finally, the pan-sharpened HRM is reconstructed by neighbourhood embedding method. Experimental results on both synthetic and real data demonstrate that our method outperforms most existing methods.

Video stabilisation with total warping variation model

This study proposes a robust approach to stabilise videos with a new variational minimising model. In video stabilisation, accumulation error often occurs in cascaded transformation chain-based methods. To alleviate accumulation error, a new total warping variation (TWV) model is proposed, which describes the smoothness of stabilised camera motion and calculates all the warping transformations efficiently. After estimating original motion parameters based on a 2D similarity transformation model, the corresponding warping parameters are calculated under the TWV minimising framework, where the separable property of the motion parameters is utilised to obtain a closed-form solution. The proposed method provides robust, smooth and precise motion trajectories after stabilisation. Furthermore, an iterative TWV method is introduced to reduce high-frequency jitters as well as low-frequency motions. Moreover, an online TWV method is presented for a long video sequence streaming by adopting a sliding windowed approach. Experimental results on various shaky video sequences show the effectiveness of the proposed method.

Fractional order variational pan-sharpening

In this paper, we propose a new fractional order variational method for pan-sharpening, which aims to obtain a high resolution multi-spectral (MS) image from a low resolution MS image and a high resolution panchromatic (PAN) image. On one hand, we use the data generative constraint for preserving the spectral information. More specifically, on the other hand, we exploit the fractional order gradient feature consistence between the high resolution MS image and PAN image for preserving the spatial information. Based on these assumptions, a new fractional order variational model is proposed and an efficient algorithm is designed to solve the proposed model. Experimental results show that the proposed method outperforms various well-known pan-sharpening methods in terms of higher spatial and spectral qualities.

Robust color demosaicking via vectorial hessian frobenius norm regularization

Single sensor camera captures scenes using a color filter array, such that each pixel samples only one of the three primary colors. A process called color demosaicking (CDM) is used to produce full color image. In this paper, we present a new variational model for high quality CDM. The robust data term is measured by Z1-norm to repress the heavy tailed artifacts. The regularization term is measured by vectorial Hessian Frobenius norm (VHFN) to capture the higher order edges as well as the intra-correlations across different channels simultaneously. To solve the proposed model, an efficient algorithm is designed using alternating direction method of multiplier (ADMM). Experimental results demonstrate that the proposed CDM method outperforms many state-of-the-art methods in reducing color artifacts, preserving the sharp edges and reconstructing fine details.

A new variational method for pan-sharpening

In this paper, we present a new variational method for pan-sharpening, which aims to obtain a high resolution multi-spectral (MS) image from a low resolution MS image and a high resolution panchromatic (PAN) image. Firstly, we assume that the desired high resolution MS image after down-sampling should be close to the low resolution MS image. More specifically, the intensity maps of PAN image and high resolution MS image bands are treated as three-dimensional (3D) differential surfaces. Then, we constrain that the surfaces of PAN image and high resolution MS image band should have the same bending directions at each point in 3D space. Based on these assumptions, a variational model is proposed and an efficient algorithm is designed to solve this variational model. Experimental results demonstrate that the proposed method outperforms various pan-sharpening methods in terms of both excellent spatial and spectral qualities.

Joint dictionary learning with ridge regression for pansharpening

A novel pansharpening method is proposed for creating a fused image of high spatial and spectral resolutions through merging a panchromatic (PAN) image with a multispectral (MS) image. To replace the patch pairs sampled from the images directly as the dictionary pairs, a joint learning model is proposed to learn a pair of compact dictionaries. Meanwhile, instead of restricting the coding coefficients of low resolution (LR) MS and high resolution (HR) MS image patches to be equal, ridge regression model is employed to describe their relation. Then, the fused MS image is calculated by combining the mapped sparse coefficients and the dictionary for the HR MS image. By comparing with some well-known methods in terms of several universal quality evaluation indexes, the simulated experimental results demonstrate the superiority of our method.

Coupled learning based on singular-values-unique and hog for face hallucination

This paper proposed a novel method for face hallucination based on a neighbor embedding technique. Traditional neighbor embedding approaches often offer counterintuitive results because consistency between high resolution images and low resolution images cannot be preserved without taking the intrinsic features of the image patches into account. In order to reinforce the consistency, on the one hand, we exploit the singular-values-unique (SVU) features inspired by singular values decomposition (SVD) successfully applied in image processing. On the other hand, we introduced the Histograms of Oriented Gradients (HOG) features to characterize the local geometric structure of the image patches to alleviate the effects of noise. At last, the learning space is extended to a coupled feature space that combines the SVU and HOG features. Simulation experiments show that this proposed approach could provide competitive results in simulation experiments in subjective and objective quality.