Kwok-Wai Hung

Robust Soft-Decision Interpolation Using Weighted Least Squares

Soft-decision adaptive interpolation (SAI) provides a powerful framework for image interpolation. The robustness of SAI can be further improved by using weighted least-squares estimation, instead of least-squares estimation in both of the parameter estimation and data estimation steps. To address the mismatch issue of “geometric duality” during parameter estimation, the residuals (prediction errors) are weighted according to the geometric similarity between the pixel of interest and the residuals. The robustness of data estimation can be improved by modeling the weights of residuals with the well-known bilateral filter. Experimental results show that there is a 0.25-dB increase in peak signal-to-noise ratio (PSNR) for a sample set of natural images after the suggested improvements are incorporated into the original SAI. The proposed algorithm produces the highest quality in terms of PSNR and subjective quality among sophisticated algorithms in the literature.

New motion compensation model via frequency classification for fast video super-resolution

A typical dynamic reconstruction-based super-resolution video involves three independent processes: registration, fusion and restoration. Fast video super-resolution systems apply translational motion compensation model for registration with low computational cost. Traditional motion compensation model assumes that the whole spectrum of pixels is consistent between frames. In reality, the low frequency component of pixels often varies significantly. We propose a translational motion compensation model via frequency classification for video super-resolution systems. A novel idea to implement motion compensation by combining the up-sampled current frame and the high frequency part of the previous frame through the SAD framework is presented. Experimental results show that the new motion compensation model via frequency classification has an advantage of 2dB gain on average over that of the traditional motion compensation model. The SR quality has 0.25dB gain on average after the fusion process which is to minimize error by making use of the new motion compensated frame.

Single image super-resolution using iterative Wiener filter

In this paper, we propose an iterative Wiener filter which can simultaneously perform interpolation and restoration by using non-local means to directly model the correlation between the desired high-resolution image and observed low-resolution image. A novel mechanism is proposed to control the decay speed of the correlation function while iteratively updating both estimated correlation and high-resolution image. During the iterations, the image is decomposed into patches with similar intensities at initial iterations and the patches are connected naturally with good convergence. Experimental results show that the proposed algorithm is able to produce natural image structures, and provides better PSNR and visual quality than the state-of-the-art algorithms using the sparse representation and natural image priors.

Improved image interpolation using bilateral filter for weighted least square estimation

New edge-directed interpolation (NEDI) consists two steps. The two steps are parameter and data estimation. The second step can be replaced by a recently proposed technique called soft-decision to consider the consistency of image structure during this data estimation. The original idea of both steps is to assume equal variances for all estimation errors, such that an ordinary least squares (OLS) estimator can be used. Due to the existence of noise, different object layers, changing in image structures, different spatial distance to the missing data, etc, we observe that the estimation errors of data samples have unequal variances. Hence, a weighted least square (WLS) estimator should be used for both steps. The bilateral filter, which can accurately remove noise and preserve image structure, has been used to model successfully the weights of squared residuals, such that we can apply it to both steps of the estimation. Experimental results show that the average PSNR of this improved interpolation method is 0.47 dB and 0.23 dB higher than two similar approaches, NEDI and Soft-decision Adaptive Interpolation (SAI) using 24 natural images from Kodak. The subjective results show improvement as well.

Novel DCT-Based Image Up-Sampling Using Learning-Based Adaptive k-NN MMSE Estimation

Image up-sampling in the discrete cosine transform (DCT) domain is a challenging problem because DCT coefficients are de-correlated, such that it is nontrivial to estimate directly high-frequency DCT coefficients from observed low-frequency DCT coefficients. In the literature, DCT-based up-sampling algorithms usually pad zeros as high-frequency DCT coefficients or estimate such coefficients with limited success mainly due to the nonadaptive estimator and restricted information from a single observed image. In this paper, we tackle the problem of estimating high-frequency DCT coefficients in the spatial domain by proposing a learning-based scheme using an adaptive k-nearest neighbor weighted minimum mean squares error (MMSE) estimation framework. Our proposed scheme makes use of the information from precomputed dictionaries to formulate an adaptive linear MMSE estimator for each DCT block. The scheme is able to estimate high-frequency DCT coefficients with very successful results. Experimental results show that the proposed up-sampling scheme produces the minimal ringing and blocking effects, and significantly better results compared with the state-of-the-art algorithms in terms of peak signal-to-noise ratio (more than 1 dB), structural similarity, and subjective quality measurements.

Hybrid DCT-Wiener-based interpolation via learnt Wiener filter

The hybrid DCT-Wiener-based (DCT-WB) interpolation scheme provides a powerful framework to interpolate an image by utilizing the information in both spatial and DCT domain. In this paper, we investigate the bottleneck of this hybrid scheme and propose a 2D non-separable block-based Wiener filter for the hybrid scheme. The Wiener filter is learnt using training image pairs through the minimum mean squares error estimation. The proposed Wiener filter resolves the quarter-pixel shift issue and provides much better performance over the original 1D 6-tap pixel-based Wiener filter. Experimental results show that incorporating the proposed Wiener filter into the hybrid scheme improves the PSNR (0.44 dB), SSIM and subjective quality for our extensive experimental work on testing images with various contents.

Single-image super-resolution using iterative Wiener filter based on nonlocal means

In this paper, we propose a single-frame super-resolution algorithm using a finite impulse response (FIR) Wiener-filter, where the correlation matrices are estimated using the nonlocal means filter. The major contribution of this work is to make use of the nonlocal means-based FIR Wiener filter to form a new iterative framework which alternately improves the estimated correlation and the estimated high-resolution (HR) image. To minimize the mean squared error of the estimated HR image, we have tried to optimize several parameters empirically, including the hyper-parameters of the nonlocal means filter by using an efficient offline training process. Experimental results show that the trained iterative framework performs better than the state-of-the-art algorithms using sparse representations and Gaussian process regression in terms of PSNR and SSIM values on a set of commonly used standard testing images. The proposed framework can be directly applied to other image processing tasks, such as denoising and restoration, and content-specific tasks such as face super-resolution. Nonlocal means filter as the correlation function of the Wiener filter.Iterative scheme to resolve drawbacks of the current non-iterative approaches.Offline training of parameters through minimizing the MSE of the training image.High competitive performance compared with state-of-the-art approaches in terms of PSNR and SSIM.Can be applied to content specific applications, e.g. face super-resolution.

Depth-assisted nonlocal means hole filling for novel view synthesis

In novel view synthesis using the depth image based rendering, there exist some unknown pixel intensities (holes) due to the unexposed area. In this paper, we propose a depth-assisted nonlocal means algorithm to fill the holes using the information in the current frame and other frames in the synthesized video. The nonlocal means has been successfully applied to the video super-resolution applications. In hole filling, the challenge is the irregular interpolation. However, there can be a relatively reliable depth map, such that we are able combine the irregular intensity information and the reliable depth information, and make use of the formulation of nonlocal means to fill the holes. Experimental results show that the proposed algorithm outperforms the conventional spatial and temporal approaches in terms of visual quality.

Real-time interpolation using bilateral filter for image zoom or video up-scaling/transcoding

In this paper, we propose a novel image interpolation technique using the small-kernel bilateral filter. The range distance of the bilateral filter is estimated using a novel maximum a posterior (MAP) estimation, in order to consider both the diagonal and vertical-horizontal correlations in image. The proposed method requires a few arithmetic operations, such that it is suitable for real time image zoom, video up-scaling and video transcoding applications in HDTV systems. Subjective evaluation shows that the proposed approach outperforms the conventional method in terms of edge sharpness and edge smoothness.

Fast video interpolation/upsampling using linear motion model

Recently, the probabilistic motion field was proposed for super-resolution reconstruction (SRR). In the interpolation step of SRR, a missing pixel can be estimated by the weighted average of neighboring pixels, which are weighted by the errors with the missing pixel. However, the errors are far from true values due to the approximated missing pixel in calculating the errors. Hence, in this paper, we propose a linear motion model to better approximate the errors, which results in a better interpolation quality. Experimental results show that a gain of 0.6 dB in PSNR is achievable using this linear motion model, and only a small number of neighboring pixels have to be used for fast interpolation/upsampling.