Jari Korhonen

Attention modeling for video quality assessment: Balancing global quality and local quality

This paper proposes to evaluate video quality by balancing two quality components: global quality and local quality. The global quality is a result from subjects allocating their attention equally to all regions in a frame and all frames in a video. It is evaluated by image quality metrics (IQM) with averaged spatiotemporal pooling. The local quality is derived from visual attention modeling and quality variations over frames. Saliency, motion, and contrast information are taken into account in modeling visual attention, which is then integrated into IQMs to calculate the local quality of a video frame. The local quality of a video sequence is calculated by pooling local quality values over all frames with a temporal pooling scheme derived from the known relationship between perceived video quality and the frequency of temporal quality variations. The overall quality of a distorted video is a weighted average between the global quality and the local quality. Experimental results demonstrate that the combination of the global quality and local quality outperforms both sole global quality and local quality, as well as other quality models, in video quality assessment. In addition, the proposed video quality modeling algorithm can improve the performance of image quality metrics on video quality assessment compared to the normal averaged spatiotemporal pooling scheme.

Power-efficient streaming for mobile terminals

Wireless Network Interface (WNI) is one of the most critical components for power efficiency in multimedia streaming to mobile devices. A common strategy to save power is to switch WNI to active mode only when network activity is expected. In streaming systems, this approach is problematic because data are typically received continuously. One solution is to transmit data packets as bursts, which leaves WNI more time between bursts in standby mode. However, that subjects bursty transmission in high peak rates, which leaves it prone to congestion. In this paper, we study theoretically and empirically the impact of burst length and peak transmission rate for observed packet loss and delay characteristics as well as potential energy savings in a Wireless Local Area Network (WLAN) environment. We outline and implement a test system with adaptive burst length to achieve improved trade-off between power efficiency and congestion tolerance.

Balancing Attended and Global Stimuli in Perceived Video Quality Assessment

The visual attention mechanism plays a key role in the human perception system and it has a significant impact on our assessment of perceived video quality. In spite of receiving less attention from the viewers, unattended stimuli can still contribute to the understanding of the visual content. This paper proposes a quality model based on the late attention selection theory, assuming that the video quality is perceived via two mechanisms: global and local quality assessment. First we model several visual features influencing the visual attention in quality assessment scenarios to derive an attention map using appropriate fusion techniques. The global quality assessment as based on the assumption that viewers allocate their attention equally to the entire visual scene, is modeled by four carefully designed quality features. By employing these same quality features, the local quality model tuned by the attention map considers the degradations on the significantly attended stimuli. To generate the overall video quality score, global and local quality features are combined by a content adaptive linear fusion method and pooled over time, taking the temporal quality variation into consideration. The experimental results have been compared to results from appropriate eye tracking and video quality assessment experiments, demonstrating promising performance.

Modeling Power-Constrained Optimal Backlight Dimming for Color Displays

In this paper, we present a framework for modeling color liquid crystal displays (LCDs) having local light-emitting diode (LED) backlight with dimming capability. The proposed framework includes critical aspects like leakage, clipping, light diffusion and human perception of luminance and allows adjustable penalization of power consumption. Based on the framework, we have designed a set of optimization-based backlight dimming algorithms providing a perceptual optimal balance of clipping and leakage, if necessary. The novel algorithms are compared with several other schemes known from the literature, using both objective measures and subjective assessment. The results show that the novel algorithms provide better quality at a given energy level or lower energy at a given quality level.

Flexible forward error correction codes with application to partial media data recovery

Conventionally, linear block codes designed for packet erasure correction are targeted to recover all the lost source packets per block, when the fraction of lost data is smaller than the redundancy overhead. However, these codes fail to recover any lost packets, if the number of erasures just exceeds the limit for full recovery capability, while it can still be beneficial to recover part of the symbols. In addition, common linear block codes are not well suited for unequal error protection, since different block codes with different rates must be allocated for each priority class separately. These two problems motivate the design of more flexible forward error correction (FEC) codes for media streaming applications. We first review the performance of short and long linear block codes. Long block codes generally offer better error correction capabilities, but at the price of higher complexity and larger coding delay. Short block codes can be more appropriate in media streaming applications that require smooth performance degradation when the channel loss rate increases. We study a new class of linear block codes using sparse generator matrices that permit to optimize the performance of short block codes for partial recovery of the lost packets. In addition, the proposed codes are extended to the design of unequal erasure protection solutions. Simulations of practical video streaming scenarios demonstrate that the flexible sparse codes offer a promising solution with interesting error correction capabilities and small variance in the residual loss rate. They typically represent an effective trade-off between short block codes with limited flexibility, and long block codes with delay penalties.

Comparing apples and oranges: Subjective quality assessment of streamed video with different types of distortion

Video quality assessment is essential for the performance analysis of visual communication applications. Objective metrics can be used to estimate the relative quality differences, but they typically give reliable results only if the compared videos contain similar type of quality distortion. However, video compression typically produces different kinds of visual artifacts than transmission errors. In this paper, we propose a novel subjective quality assessment method that is suitable for comparing different types of quality distortions. The proposed method has been used to evaluate how well PSNR estimates the relative subjective quality levels for content with different types of quality distortions. Our conclusion is that PSNR is not a reliable metric for assessing the co-impact of compression artifacts and transmission errors on the subjective quality.

Modeling LCD displays with local backlight dimming for image quality assessment

Traditionally, algorithm-based (objective) image and video quality assessment methods operate with the numerical presentation of the signal, and they do not take the characteristics of the actual output device into account. This is a reasonable approach, when quality assessment is needed for evaluating the signal quality distortion related directly to digital signal processing, such as compression. However, the physical characteristics of the display device also pose a significant impact on the overall perception. In order to facilitate image quality assessment on modern liquid crystaldisplays (LCD) using light emitting diode (LED) backlight with local dimming, we present the essential considerations and guidelines for modeling the characteristics of displays with high dynamic range (HDR) and locally adjustable backlight segments. The representation of the image generated by the model can be assessed using the traditional objective metrics, and therefore the proposed approach is useful for assessing the performance of different backlight dimming algorithms in terms of resulting quality and power consumption in a simulated environment. We have implemented the proposed model in C++ and compared the visual results produced by the model against respective images displayed on a real display with locally controlled backlight units.

Spatial and temporal pooling of image quality metrics for perceptual video quality assessment on packet loss streams

Video streaming through bandwidth-limited channels often suffer from packet losses. Therefore, perceptual quality assessment on video sequences with packet losses is a critical issue in digital video communications. This paper analyzes several image quality metrics and evaluates their applications using spatial and temporal pooling schemes in perceptual video quality assessment for video streams with packet losses. Several approaches using Minkowski summation and averages over different distorted spatial regions and temporal frames to pool the spatial and temporal qualities are evaluated. The experimental results with respect to the subjective video quality measurements demonstrate that the subjects are more sensitive to the most annoying spatial regions and temporal segments when assessing the video quality of the lossy streams.

No-Reference Video Quality Assessment Using Codec Analysis

A no-reference (NR) video quality assessment (VQA) method is presented for videos distorted by H.264/Advanced Video Coding (AVC) and MPEG-2. The assessment is performed without access to the bitstream. Instead, we analyze and estimate coefficients based on decoded pixels. The approach involves distinguishing between the two types of videos, estimating the level of quantization used in the I-frames, and exploiting this information to assess the video quality. To do this for H.264/AVC, the distribution of the discrete cosine transform-coefficients after intra-prediction and deblocking are modeled. To obtain VQA features for H.264/AVC, we propose a novel estimation method of the quantization in H.264/AVC videos without bitstream access, which can also be used for peak signal-to-noise ratio estimation. The results from the MPEG-2 and H.264/AVC analysis are mapped to a perceptual measure of video quality by support vector regression. For validation purposes, the proposed method was tested on two databases. In both cases, a good performance compared with state of the art full, reduced, and NR VQA algorithms was achieved.

How to evaluate objective video quality metrics reliably

The typical procedure for evaluating the performance of different objective quality metrics and indices involves comparisons between subjective quality ratings and the quality indices obtained using the objective metrics in question on the known video sequences. Several correlation indicators can be employed to assess how well the subjective ratings can be predicted from the objective values. In this paper, we give an overview of the potential sources for uncertainties and inaccuracies in such studies, related both to the method of comparison, possible inaccuracies in the subjective data, as well as processing of subjective data. We also suggest some general guidelines for researchers to make comparison studies of objective video quality metrics more reliable and useful for the practitioners in the field.