Jinjian Wu

Perceptual Quality Metric With Internal Generative Mechanism

Objective image quality assessment (IQA) aims to evaluate image quality consistently with human perception. Most of the existing perceptual IQA metrics cannot accurately represent the degradations from different types of distortion, e.g., existing structural similarity metrics perform well on content-dependent distortions while not as well as peak signal-to-noise ratio (PSNR) on content-independent distortions. In this paper, we integrate the merits of the existing IQA metrics with the guide of the recently revealed internal generative mechanism (IGM). The IGM indicates that the human visual system actively predicts sensory information and tries to avoid residual uncertainty for image perception and understanding. Inspired by the IGM theory, we adopt an autoregressive prediction algorithm to decompose an input scene into two portions, the predicted portion with the predicted visual content and the disorderly portion with the residual content. Distortions on the predicted portion degrade the primary visual information, and structural similarity procedures are employed to measure its degradation; distortions on the disorderly portion mainly change the uncertain information and the PNSR is employed for it. Finally, according to the noise energy deployment on the two portions, we combine the two evaluation results to acquire the overall quality score. Experimental results on six publicly available databases demonstrate that the proposed metric is comparable with the state-of-the-art quality metrics.

Reduced-Reference Image Quality Assessment With Visual Information Fidelity

Reduced-reference (RR) image quality assessment (IQA) aims to use less data about the reference image and achieve higher evaluation accuracy. Recent research on brain theory suggests that the human visual system (HVS) actively predicts the primary visual information and tries to avoid the residual uncertainty for image perception and understanding. Therefore, the perceptual quality relies to the information fidelities of the primary visual information and the residual uncertainty. In this paper, we propose a novel RR IQA index based on visual information fidelity. We advocate that distortions on the primary visual information mainly disturb image understanding, and distortions on the residual uncertainty mainly change the comfort of perception. We separately compute the quantities of the primary visual information and the residual uncertainty of an image. Then the fidelities of the two types of information are separately evaluated for quality assessment. Experimental results demonstrate that the proposed index uses few data (30 bits) and achieves high consistency with human perception.

Hyperspectral Image Super-Resolution via Non-Negative Structured Sparse Representation

Hyperspectral imaging has many applications from agriculture and astronomy to surveillance and mineralogy. However, it is often challenging to obtain high-resolution (HR) hyperspectral images using existing hyperspectral imaging techniques due to various hardware limitations. In this paper, we propose a new hyperspectral image super-resolution method from a low-resolution (LR) image and a HR reference image of the same scene. The estimation of the HR hyperspectral image is formulated as a joint estimation of the hyperspectral dictionary and the sparse codes based on the prior knowledge of the spatial-spectral sparsity of the hyperspectral image. The hyperspectral dictionary representing prototype reflectance spectra vectors of the scene is first learned from the input LR image. Specifically, an efficient non-negative dictionary learning algorithm using the block-coordinate descent optimization technique is proposed. Then, the sparse codes of the desired HR hyperspectral image with respect to learned hyperspectral basis are estimated from the pair of LR and HR reference images. To improve the accuracy of non-negative sparse coding, a clustering-based structured sparse coding method is proposed to exploit the spatial correlation among the learned sparse codes. The experimental results on both public datasets and real LR hypspectral images suggest that the proposed method substantially outperforms several existing HR hyperspectral image recovery techniques in the literature in terms of both objective quality metrics and computational efficiency.Hyperspectral imaging has many applications from agriculture and astronomy to surveillance and mineralogy. However, it is often challenging to obtain high-resolution (HR) hyperspectral images using existing hyperspectral imaging techniques due to various hardware limitations. In this paper, we propose a new hyperspectral image super-resolution method from a low-resolution (LR) image and a HR reference image of the same scene. The estimation of the HR hyperspectral image is formulated as a joint estimation of the hyperspectral dictionary and the sparse codes based on the prior knowledge of the spatial-spectral sparsity of the hyperspectral image. The hyperspectral dictionary representing prototype reflectance spectra vectors of the scene is first learned from the input LR image. Specifically, an efficient non-negative dictionary learning algorithm using the block-coordinate descent optimization technique is proposed. Then, the sparse codes of the desired HR hyperspectral image with respect to learned hyperspectral basis are estimated from the pair of LR and HR reference images. To improve the accuracy of non-negative sparse coding, a clustering-based structured sparse coding method is proposed to exploit the spatial correlation among the learned sparse codes. The experimental results on both public datasets and real LR hypspectral images suggest that the proposed method substantially outperforms several existing HR hyperspectral image recovery techniques in the literature in terms of both objective quality metrics and computational efficiency.

No-reference quality assessment of deblocked images

JPEG is the most commonly used image compression standard. In practice, JPEG images are easily subject to blocking artifacts at low bit rates. To reduce the blocking artifacts, many deblocking algorithms have been proposed. However, they also introduce certain degree of blur, so the deblocked images contain multiple distortions. Unfortunately, the current quality metrics are not designed for multiply distorted images, so they are limited in evaluating the quality of deblocked images. To solve the problem, this paper presents a no-reference (NR) quality metric for deblocked images. A DeBlocked Image Database (DBID) is first built with subjective Mean Opinion Score (MOS) as ground truth. Then a NR DeBlocked Image Quality (DBIQ) metric is proposed by simultaneously evaluating blocking artifacts in smooth regions and blur in textured regions. Experimental results conducted on the DBID database demonstrate that the proposed metric is effective in evaluating the quality of deblocked images, and it significantly outperforms the existing metrics. As an application, the proposed metric is further used for automatic parameter selection in image deblocking algorithms.

Just Noticeable Difference Estimation for Images With Free-Energy Principle

In this paper, we introduce a novel just noticeable difference (JND) estimation model based on the unified brain theory, namely the free-energy principle. The existing pixel-based JND models mainly consider the orderly factors and always underestimate the JND threshold of the disorderly region. Recent research indicates that the human visual system (HVS) actively predicts the orderly information and avoids the residual disorderly uncertainty for image perception and understanding. Thus, we suggest that there exists disorderly concealment effect which results in high JND threshold of the disorderly region. Beginning with the Bayesian inference, we deduce an autoregressive model to imitate the active prediction of the HVS. Then, we estimate the disorderly concealment effect for the novel JND model. Experimental results confirm that the proposed JND model outperforms the relevant existing ones. Furthermore, we apply the proposed JND model in image compression, and around 15% of bit rate can be reduced without jeopardizing the perceptual quality.

Image Quality Assessment with Degradation on Spatial Structure

In this letter, we introduce an improved structural degradation based image quality assessment (IQA) method. Most of the existing structural similarity based IQA metrics mainly consider the spatial contrast degradation but have not fully considered the changes on the spatial distribution of structures. Since the human visual system (HVS) is sensitive to degradations on both spatial contrast and spatial distribution, both factors need to be considered for IQA. In order to measure the structural degradation on spatial distribution, the local binary patterns (LBPs) are first employed to extract structural information. And then, the LBP shift between the reference and distorted images is computed, because noise distorts structural patterns. Finally, the spatial contrast degradation on each pair of LBP shifts is calculated for quality assessment. Experimental results on three large benchmark databases confirm that the proposed IQA method is highly consistent with the subjective perception.

Pattern Masking Estimation in Image With Structural Uncertainty

A model of visual masking, which reveals the visibility of stimuli in the human visual system (HVS), is useful in perceptual based image/video processing. The existing visual masking function mainly considers luminance contrast, which always overestimates the visibility threshold of the edge region and underestimates that of the texture region. Recent research on visual perception indicates that the HVS is sensitive to orderly regions that possess regular structures and insensitive to disorderly regions that possess uncertain structures. Therefore, structural uncertainty is another determining factor on visual masking. In this paper, we introduce a novel pattern masking function based on both luminance contrast and structural uncertainty. Through mimicking the internal generative mechanism of the HVS, a prediction model is firstly employed to separate out the unpredictable uncertainty from an input image. In addition, an improved local binary pattern is introduced to compute the structural uncertainty. Finally, combining luminance contrast with structural uncertainty, the pattern masking function is deduced. Experimental result demonstrates that the proposed pattern masking function outperforms the existing visual masking function. Furthermore, we extend the pattern masking function to just noticeable difference (JND) estimation and introduce a novel pixel domain JND model. Subjective viewing test confirms that the proposed JND model is more consistent with the HVS than the existing JND models.

Orientation selectivity based visual pattern for reduced-reference image quality assessment

Image quality assessment (IQA) is in great demand for high quality image selection in the big data era. The challenge of reduced-reference (RR) IQA is how to use limited data to effectively represent the visual content of an image in the context of IQA. Research on neuroscience indicates that the human visual system (HVS) exhibits obvious orientation selectivity (OS) mechanism for visual content extraction. Inspired by this, an OS based visual pattern (OSVP) is proposed to extract visual content for RR IQA in this paper. The OS arises from the arrangement of the excitatory and inhibitory interactions among connected cortical neurons in a local receptive field. According to the OS mechanism, the similarity of preferred orientations between two nearby pixels is first analyzed. Then, the orientation similarities of pixels in a local neighborhood are arranged, and the OSVP is built for visual information representation. With the help of OSVP, the visual content of an image is extracted and mapped into a histogram. By calculating the changes between the two histograms of reference and distorted images, a quality score is produced. Experimental results on five public databases demonstrate that the proposed RR IQA method has performance consistent with the human perception under a small amount of reference data (only 9 values).

Image Sharpness Assessment by Sparse Representation

Recent advances in sparse representation show that overcomplete dictionaries learned from natural images can capture high-level features for image analysis. Since atoms in the dictionaries are typically edge patterns and image blur is characterized by the spread of edges, an overcomplete dictionary can be used to measure the extent of blur. Motivated by this, this paper presents a no-reference sparse representation-based image sharpness index. An overcomplete dictionary is first learned using natural images. The blurred image is then represented using the dictionary in a block manner, and block energy is computed using the sparse coefficients. The sharpness score is defined as the variance-normalized energy over a set of selected high-variance blocks, which is achieved by normalizing the total block energy using the sum of block variances. The proposed method is not sensitive to training images, so a universal dictionary can be used to evaluate the sharpness of images. Experiments on six public image quality databases demonstrate the advantages of the proposed method.

Non-local spatial redundancy reduction for bottom-up saliency estimation

In this paper we present a redundancy reduction based approach for computational bottom-up visual saliency estimation. In contrast to conventional methods, our approach determines the saliency by filtering out redundant contents instead of measuring their significance. To analyze the redundancy of self-repeating spatial structures, we propose a non-local self-similarity based procedure. The result redundancy coefficient is used to compensate the Shannon entropy, which is based on statistics of pixel intensities, to generate the bottom-up saliency map of the visual input. Experimental results on three publicly available databases demonstrate that the proposed model is highly consistent with the subjective visual attention.