Hantao Liu

Visual Attention in Objective Image Quality Assessment: Based on Eye-Tracking Data

Since the human visual system (HVS) is the ultimate assessor of image quality, current research on the design of objective image quality metrics tends to include an important feature of the HVS, namely, visual attention. Different metrics for image quality prediction have been extended with a computational model of visual attention, but the resulting gain in reliability of the metrics so far was variable. To better understand the basic added value of including visual attention in the design of objective metrics, we used measured data of visual attention. To this end, we performed two eye-tracking experiments: one with a free-looking task and one with a quality assessment task. In the first experiment, 20 observers looked freely to 29 unimpaired original images, yielding us so-called natural scene saliency (NSS). In the second experiment, 20 different observers assessed the quality of distorted versions of the original images. The resulting saliency maps showed some differences with the NSS, and therefore, we applied both types of saliency to four different objective metrics predicting the quality of JPEG compressed images. For both types of saliency the performance gain of the metrics improved, but to a larger extent when adding the NSS. As a consequence, we further integrated NSS in several state-of-the-art quality metrics, including three full-reference metrics and two no-reference metrics, and evaluated their prediction performance for a larger set of distortions. By doing so, we evaluated whether and to what extent the addition of NSS is beneficial to objective quality prediction in general terms. In addition, we address some practical issues in the design of an attention-based metric. The eye-tracking data are made available to the research community .

A No-Reference Metric for Perceived Ringing Artifacts in Images

A novel no-reference metric that can automatically quantify ringing annoyance in compressed images is presented. In the first step a recently proposed ringing region detection method extracts the regions which are likely to be impaired by ringing artifacts. To quantify ringing annoyance in these detected regions, the visibility of ringing artifacts is estimated, and is compared to the activity of the corresponding local background. The local annoyance score calculated for each individual ringing region is averaged over all ringing regions to yield a ringing annoyance score for the whole image. A psychovisual experiment is carried out to measure ringing annoyance subjectively and to validate the proposed metric. The performance of our metric is compared to existing alternatives in literature and shows to be highly consistent with subjective data.

A perceptually relevant no-reference blockiness metric based on local image characteristics

A novel no-reference blockiness metric that provides a quantitative measure of blocking annoyance in block-based DCT coding is presented. The metric incorporates properties of the human visual system (HVS) to improve its reliability, while the additional cost introduced by the HVS is minimized to ensure its use for real-time processing. This is mainly achieved by calculating the local pixel-based distortion of the artifact itself, combined with its local visibility by means of a simplified model of visual masking. The overall computation efficiency and metric accuracy is further improved by including a grid detector to identify the exact location of blocking artifacts in a given image. The metric calculated only at the detected blocking artifacts is averaged over all blocking artifacts in the image to yield an overall blockiness score. The performance of this metric is compared to existing alternatives in literature and shows to be highly consistent with subjective data at a reduced computational load. As such, the proposed blockiness metric is promising in terms of both computational efficiency and practical reliability for real-life applications.

Studying the added value of visual attention in objective image quality metrics based on eye movement data

Current research on image quality assessment tends to include visual attention in objective metrics to further enhance their performance. A variety of computational models of visual attention are implemented in different metrics, but their accuracy in representing human visual attention is not fully proved yet. Thus, to provide more accurate evidence on whether and to what extent visual attention can be beneficial for objective quality prediction, the use of “ground truth” visual attention data is highly desired. In this paper, the data of an eye-tracking experiment are integrated in two objective metrics well-known in literature. Experimental results demonstrate that there is indeed a gain in performance including visual attention in objective metrics. The amount of gain in performance tends to depend on the type of objective metric and image distortion.

A no-reference perceptual blockiness metric

A novel no-reference blockiness metric that can automatically and perceptually quantify blocking artifacts of DCT coding is presented. The proposed metric is built upon the specific structure information of the artifact itself combined with the properties of the human visual system (HVS) by means of a simple and efficient model of visual masking. Investigations are conducted to reduce the additional cost introduced by the human vision model, without compromising its overall prediction ability. The proposed metric is validated through comparing its performance to state-of-the-art HVS model based blockiness metrics with respect to accuracy, reliability and computational complexity.

Interactions of visual attention and quality perception

Several attempts to integrate visual saliency information in quality metrics are described in literature, albeit with contradictory results. The way saliency is integrated in quality metrics should reflect the mechanisms underlying the interaction between image quality assessment and visual attention. This interaction is actually two-fold: (1) image distortions can attract attention away from the Natural Scene Saliency (NSS), and (2) the quality assessment task in itself can affect the way people look at an image. A subjective study was performed to analyze the deviation in attention from NSS as a consequence of being asked to assess the quality of distorted images, and, in particular, whether, and if so how, this deviation depended on the distortion kind and/or amount. Saliency maps were derived from eye-tracking data obtained during scoring distorted images, and they were compared to the corresponding NSS, derived from eye-tracking data obtained during freely looking at high quality images. The study revealed some structural differences between the NSS maps and the ones obtained during quality assessment of the distorted images. These differences were related to the quality level of the images; the lower the quality, the higher the deviation from the NSS was. The main change was identified as a shrinking of the region of interest, being most evident at low quality. No evident role for the kind of distortion in the change in saliency was found. Especially at low quality, the quality assessment task seemed to prevail on the natural attention, forcing it to deviate in order to better evaluate the impact of artifacts.

A Perceptually Relevant Approach to Ringing Region Detection

An efficient approach toward a no-reference ringing metric intrinsically exists of two steps: first detecting regions in an image where ringing might occur, and second quantifying the ringing annoyance in these regions. This paper presents a novel approach toward the first step: the automatic detection of regions visually impaired by ringing artifacts in compressed images. It is a no-reference approach, taking into account the specific physical structure of ringing artifacts combined with properties of the human visual system (HVS). To maintain low complexity for real-time applications, the proposed approach adopts a perceptually relevant edge detector to capture regions in the image susceptible to ringing, and a simple yet efficient model of visual masking to determine ringing visibility. The approach is validated with the results of a psychovisual experiment, and its performance is compared to existing alternatives in literature for ringing region detection. Experimental results show that our method is promising in terms of both reliability and computational efficiency.

How to apply spatial saliency into objective metrics for JPEG compressed images

This paper investigates how saliency obtained from eye-tracking data can be integrated into objective metrics for JPEG compressed images. The objective metrics used in this paper are both based on features, locally extracted from the images and serving as input to a neural network for the overall quality prediction. We compare various weighting functions to combine saliency with these objective metrics, taking into account the possible distraction due to artifacts that might affect the quality judgment. Experimental results indicate that including saliency into objective metrics in an appropriate way can further enhance their performance.

A simplified human vision model applied to a blocking artifact metric

A novel approach towards a simplified, though still reliable human vision model based on the spatial masking properties of the human visual system (HVS) is presented. The model contains two basic characteristics of the HVS, namely texture masking and luminance masking. These masking effects are implemented as simple spatial filtering followed by a weighting function, and are efficiently combined into a single visibility coefficient. This HVS model is applied to a blockiness metric by using its output to scale the blockedge strength. To validate the proposed model, its performance in the blockiness metric is determined by comparing it to the same blockiness metric having different HVS-based models embedded. The results show that the proposed model is indeed simple, without compromising its accuracy.

An efficient no-reference metric for perceived blur

This paper presents an efficient no-reference metric that quantifies perceived image quality induced by blur. Instead of explicitly simulating the human visual perception of blur, it calculates the local edge blur in a cost-effective way, and applies an adaptive neural network to empirically learn the highly nonlinear relationship between the local values and the overall image quality. Evaluation of the proposed metric using the LIVE blur database shows its high prediction accuracy at a largely reduced computational cost. To further validate the performance of the blur metric on its robustness against different image content, two additional quality perception experiments were conducted: one with highly textured natural images and one with images with an intentionally blurred background1. Experimental results demonstrate that the proposed blur metric is promising for real-world applications both in terms of computational efficiency and practical reliability.