Philippe Hanhart

Subjective quality evaluation of the upcoming HEVC video compression standard

High Effciency Video Coding (HEVC) is the latest attempt by ISO/MPEG and ITU-T/VCEG to define the next generation compression standard beyond H.264/MPEG-4 Part 10 AVC. One of the major goals of HEVC is to provide effcient compression for resolutions beyond HDTV. However, the subjective evaluations that led to the selection of technologies were bound to HDTV resolution. Moreover, performance evaluation metrics to report effciency results of this standard are mainly based on PSNR, especially for resolutions beyond HDTV. This paper provides subjective evaluation results to assess the performance of the current HEVC codec for resolutions beyond HDTV.

HDR image compression: A new challenge for objective quality metrics

High Dynamic Range (HDR) imaging is able to capture a wide range of luminance values, closer to what the human visual system can perceive. It is believed by many that HDR is a technology that will revolutionize TV and cinema industry similar to how color television did. However, the complexity of HDR requires reinvention of the whole chain from capture to display. In this paper, HDR images compressed with the upcoming JPEG XT HDR image coding standard are used to investigate the correlation between thirteen well known full-reference metrics and perceived quality of HDR content. The metrics are benchmarked using ground truth subjective scores collected during quality evaluations performed on a Dolby Pulsar HDR monitor. Results demonstrate that objective quality assessment of HDR image compression is challenging. Most of the tested metrics, with exceptions of HDR-VDP-2 and FSIM computed for luma component, poorly predict human perception of visual quality.

Quality assessment of a stereo pair formed from decoded and synthesized views using objective metrics

When a stereo pair is formed from a decoded view and a synthesized view, it is unclear how the overall quality of the stereo pair should be assessed through objective quality metrics. In this paper, this problem is addressed considering a 3D video represented in the format of multiview video plus depth. The performance of different state-of-the-art 2D quality metrics is analyzed in terms of correlation with subjective perception of video quality. A set of subjective data collected through formal subjective evaluation tests is used as benchmark. Results show that the measured quality of the decoded view has the highest correlation with perceived quality. If the objective quality assessment is based on the measured quality of the synthesized view, it is suggested to use VIF, VQM, MS-SSIM, or SSIM since they significantly outperform other objective metrics, including PSNR.

Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment

The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.

Calculation of average coding efficiency based on subjective quality scores

The Bjontegaard model is widely used to calculate the coding efficiency between different codecs. However, this model might not be an accurate predictor of the true coding efficiency as it relies on PSNR measurements. Therefore, in this paper, we propose a model to calculate the average coding efficiency based on subjective quality scores, i.e., mean opinion scores (MOS). We call this approach Subjective Comparison of ENcoders based on fItted Curves (SCENIC). To consider the intrinsic nature of bounded rating scales, a logistic function is used to fit the rate-distortion (R-D) values. The average MOS and bit rate differences are computed between the fitted R-D curves. The statistical property of subjective scores is considered to estimate corresponding confidence intervals on the calculated average MOS and bit rate differences. The proposed model is expected to report more realistic coding efficiency as PSNR is not always correlated with perceived visual quality.

Subjective Evaluation of Higher Dynamic Range Video

High dynamic range (HDR) imaging is able to capture a wide range of luminance values, closer to what the human eye can perceive. However, for capture and display technologies, it is important to answer the question on the significance of higher dynamic range for user preference. This paper answers this question by investigating the added value of higher dynamic range via a rigorous set of subjective experiments using paired comparison methodology. Video sequences at four different peak luminance levels were displayed side-by-side on a Dolby Research HDR RGB backlight dual modulation display (aka ‘Pulsar’), which is capable of reliably displaying video content at 4000 cd=m2 peak luminance. The results of the subjective experiment demonstrate that the preference of an average viewer increases logarithmically with the increase in the maximum luminance level at which HDR content is displayed, with 4000 cd=m2 being the most attractive option.

A quality assessment protocol for free-viewpoint video sequences synthesized from decompressed depth data

In this paper, the analysis of a subjective quality experiment consisting in assessing the quality of free-viewpoint video sequences generated from decompressed depth data is presented. In the absence of a dedicated subjective assessment protocol for the evaluation of such 3D systems, a subjective quality assessment methodology is proposed for the context of MVD compression. The proposed methodology includes the assessment of free-viewpoint video sequences generated from decompressed depth data and from view synthesis processes. The proposed methodology is meant to assess the performances of depth map compression and view synthesis algorithms.

Quality evaluation of HEVC and VP9 video compression in real-time applications

Video consumption over Internet has increased significantly over the recent years and occupies the majority of the overall data traffic. To decrease the load on the Internet infrastructure and reduce bandwidth taken by video, higher efficiency video codecs, such as H.265/HEVC and VP9, have been developed. The availability of these two new competing video coding formats raises the question of which is more efficient in terms of rate-distortion and by how much they outperform the current state-of-the-art coding standard, H.264/AVC. This paper provides an answer to this difficult question for low-delay video applications, e.g., real-time video streaming/conferencing or video surveillance. The benchmarking of HEVC and VP9 video compression was conducted by means of subjective evaluations, assuming web browser playback, an uncontrolled environment, and HD video content. Considering a wide range of bit rates from very low to high bit rates, corresponding to low quality up to transparent quality (when compared to the original video), results show a clear advantage of HEVC with average bit rate savings of 59.5% when compared to AVC and 42.4% when compared to VP9.

Benchmarking of quality metrics on ultra-high definition video sequences

The performance of objective quality metrics for high-definition (HD) video sequences is well studied, but little is known about their performance for ultra-high definition (UHD) video sequences. This paper analyzes the performance of several common objective quality metrics (PSNR, VSNR, SSIM, MS-SSIM, VIF, and VQM) on three different 4K UHD video sequences using subjective scores as ground truth. The findings confirm the content-dependent nature of most metrics (with VIF being the only exception), which has been reported previously for standard and high resolution video sequences. PSNR showed the lowest correlation with ground truth quality scores when the analysis was performed for all contents at once and thus is not recommended as a general metric for video quality, while VIF showed the highest Pearson (0.83) and Spearman (0.87) correlation coefficients and may be used as a general purpose metric. On the other hand, all studied metrics were accurate in distinguishing different quality levels for the same content. The results of several fittings between metric values and subjective ground truth scores demonstrated that logistic fitting provides the highest correlation. The results also indicated a shift in metrics values between synthetic and natural contents.

Subjective quality assessment database of HDR images compressed with JPEG XT

Recent advances in high dynamic range (HDR) capturing and display technologies attracted a lot of interest to HDR imaging. Many issues that are considered as being resolved for conventional low dynamic range (LDR) images pose new challenges in HDR context. One such issue is a lack of standards for HDR image compression. Another is the limited availability of suitable image datasets that are suitable for studying and evaluation of HDR image compression. In this paper, we address this problem by creating a publicly available dataset of 20 HDR images and corresponding versions compressed at four different bit rates with three profiles of the upcoming JPEG XT standard for HDR image compression. The images cover different scenes, dynamic ranges, and acquisition methods (fusion from several exposures, frame of an HDR video, and CGI generated images). The dataset also includes Mean Opinion Scores (MOS) for each compressed version of the images obtained from extensive subjective experiments using SIM2 HDR monitor.