David S. Hands

VQeg validation and ITU standardization of objective perceptual video quality metrics [Standards in a Nutshell]

For industry, the need to access accurate and reliable objective video metrics has become more pressing with the advent of new video applications and services such as mobile broadcasting, Internet video, and Internet Protocol television (IPTV). Industry-class objective quality- measurement models have a wide range of uses, including equipment testing (e.g., codec evaluation), transmission- planning and network-dimensioning tasks, head-end quality assurance, in- service network monitoring, and client-based quality measurement. The Video Quality Experts Group (VQEG) is the primary forum for validation testing of objective perceptual quality models. The work of VQEG has resulted in International Telecommunication Union (ITU) standardization of objective quality models designed for standard- definition television and for multimedia applications. This article reviews VQEGs work, paying particular attention to the groups approach to validation testing.

Subjective Multimedia Quality Assessment

The Video Quality Experts Group (VQEG) is preparing a programme of subjective multimedia quality tests. The results from these tests will be used to evaluate the performance of competing objective multimedia quality metrics. The reliability of the subjective test data is of great importance for VQEGs task. This paper provides an overview of VQEGs multimedia ad-hoc group. The work of this group will require subjective tests to be performed by laboratories located in Europe, Asia and North America. For VQEGs multimedia work to be successful, the subjective assessment methodology must be precisely defined and produce reliable and repeatable subjective quality data. Although international standards covering multimedia quality assessment methods are in force, there remains some uncertainty regarding the most effective approach to assessing the subjective quality of multimedia. A review of existing methods is provided. Two experiments are presented investigating the suitability of alternative subjective assessment methods (single-stimulus ACR and SAMVIQ). The results of these experiments are discussed within the context of the VQEG multimedia testing programme.

Accurate and Efficient Method for Smoothly Space-Variant Gaussian Blurring

This paper presents a computationally efficient algorithm for smoothly space-variant Gaussian blurring of images. The proposed algorithm uses a specialized filter bank with optimal filters computed through principal component analysis. This filter bank approximates perfect space-variant Gaussian blurring to arbitrarily high accuracy and at greatly reduced computational cost compared to the brute force approach of employing a separate low-pass filter at each image location. This is particularly important for spatially variant image processing such as foveated coding. Experimental results show that the proposed algorithm provides typically 10 to 15 dB better approximation of perfect Gaussian blurring than the blended Gaussian pyramid blurring approach when using a bank of just eight filters.

QoS of Video Delivered over 802.11e WLANs

Many home networks now use 802.ll wireless local area networks (WLAN)s. The 802.11e amendment has been designed to improve quality of service (QoS) over these networks allowing for multimedia applications such as IPTV to be better supported. The H.264 video compression standard is suitable for IPTV due to its high compression and error resilience. Video packets of different slice types are of varying importance to the decoded video quality so can be mapped into different priority queues using 802.11e enhanced distributed channel access (EDCA). We investigate several mapping schemes for a variety of video content to see how the quality of the decoded video is affected as the number of concurrent video connections is increased. The different mapping schemes exhibit different loss patterns in the video sequences and their impact on the video quality is content dependent. Subjective tests were therefore carried out which allow us to perform an accurate and valid assessment of the video quality. Packet loss rate is also reported. Our results show that as the number of concurrent videos approaches the network capacity some mapping schemes show a cliff-edge drop in quality while others offer a more acceptable gradual quality degradation. These schemes cause B-frames to be dropped and in effect reduce the video frame rate. These schemes are more successful for videos that have low or medium temporal activity.

Hybrid no-reference video quality prediction

This paper describes how the quality of H.264/AVC encoded standard resolution video sequences may be accurately estimated within a suitable video decoder. Results are presented for eighteen different content clips from bit-rates of 100 kbit/s up to 14 Mbit/s and show the technique to outperform traditional peak signal-to-noise ratio (PSNR) estimation schemes. The techniques low complexity and high accuracy, opens up the possibility of uses in video service monitoring and testing.

Quality assurance for IPTV

A tool designed to perform quality assurance on IPTV encoded assets is presented. The tool works on stored MPEG-2 transport stream files and combines a powerful graphical-user-interface with automatic no-reference detectors to handle video quality, audio and field-ordering issues. The tool provides automatic quality assurance of assets before they are made available for streaming and replaces the need for exhaustive viewing tests by human operators.

Perceptual Quality Measurement—Towards a More Efficient Process for Validating Objective Models [Standards in a Nutshell]

The Quality of Service Metrics (QoSM) Committee of the Alliance for Telecommunication Industry Standards (ATIS) Internet Protocol Television (IPTV) Interoperability Forum (ΠF) is tasked with defining how objective quality metrics can provide meaningful IPTV performance measures. This group has reviewed current objective quality models as well as the processes by which such models are validated. This article describes current practices in validating objective quality models and presents a new, streamlined process that can be implemented to achieve more efficient and effective model validation. Of main interest for IPTV are models for predicting video and audiovisual quality; however, the process also applies to the validation of perceptual quality models (PQMs) for other modalities. The proposed process offers vendors a fast route to validating objective PQMs while providing industry with the assurance of independent, unbiased model evaluation.

No reference perceptual quality metrics: approaches and limitations

To predict subjective quality it is necessary to develop and validate approaches that accurately predict video quality. For perceptual quality models, developers have implemented methods that utilise information from both the original and the processed signals (full reference and reduced reference methods). For many practical applications, no reference (NR) methods are required. It has been a major challenge for developers to produce no reference methods that attain the necessary predictive performance for the methods to be deployed by industry. In this paper, we present a comparison between no reference methods operating on either the decoded picture information alone or using a bit-stream / decoded picture hybrid analysis approach. Two NR models are introduced: one using decoded picture information only; the other using a hybrid approach. Validation data obtained from subjective quality tests are used to examine the predictive performance of both models. The strengths and limitations of the two NR methods are discussed.

Subjective assessment of HDTV content: comparison of quality across HDTV formats

This paper reports on a series of experiments designed to examine the subjective quality of HDTV encoded video. A common set of video sequences varying in spatial and temporal complexity will be encoded at different H.264 encoding profiles. Each video clip used in the test will be available in 720p, 1080i and 1080p HDTV formats. Each video clip will then be encoded in CBR mode using H.264 at bitrates ranging from 1 Mbit/s up to 15 Mbit/s with the other encoding parameters identical (e.g., identical key frame intervals and CABAC entropy mode). This approach has been chosen to enable direct comparison across bit-rates and formats most relevant to HDTV broadcast services. Three subjective tests will be performed, one test for each HDTV format. The single-stimulus subjective test method with the ACR rating scale will be employed. A total of 15 non-expert subjects will participate in each test. A different sample of subjects will participate in each test, making 45 subjects in total. The test results will be available by end of the summer 2009.

Subjective video quality evaluation for multimedia applications

Video quality can be measured using both subjective and objective assessment methods. Subjective experiments are crucially important as they constitute a benchmark for evaluating the performance of objective quality metrics. Subjective quality assessment of television pictures has received extensive attention from experts over the past decades. On the other hand, emerging applications such as PC-based video streaming and mobile video streaming require new subjective test methodologies. Although some recent studies have compared different test methodologies and procedures, most concerned television pictures. No studies for multimedia-type video really validated the repeatability and reliability of the assessment method and the experimental procedure. This paper outlines a methodology for conducting subjective evaluation of video quality for multimedia applications in a repeatable and reliable manner across different laboratories. Using video material at low-resolution, low-bit rate and low-frame rate, the same experiment was conducted by two different laboratories, i.e. test material was identical in both experiments. Laboratory set-up was different, i.e. different computers and display panels were used, and viewing distance was not fixed. Results show that quality ratings obtained in both experiments are statistically identical. This is an important validation step for the Video Quality Experts Group, which will conduct an ambitious campaign of subjective experiments using many different test laboratories.