Margaret H. Pinson

A new standardized method for objectively measuring video quality

The National Telecommunications and Information Administration (NTIA) General Model for estimating video quality and its associated calibration techniques were independently evaluated by the Video Quality Experts Group (VQEG) in their Phase II Full Reference Television (FR-TV) test. The NTIA General Model was the only video quality estimator that was in the top performing group for both the 525-line and 625-line video tests. As a result, the American National Standards Institute (ANSI) adopted the NTIA General Model and its associated calibration techniques as a North American Standard in 2003. The International Telecommunication Union (ITU) has also included the NTIA General Model as a normative method in two Draft Recommendations. This paper presents a description of the NTIA General Model and its associated calibration techniques. The independent test results from the VQEG FR-TV Phase II tests are summarized, as well as results from eleven other subjective data sets that were used to develop the method.

Comparing subjective video quality testing methodologies

International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. Some of these test methods are double stimulus where viewers rate the quality or change in quality between two video streams (reference and impaired). Others are single stimulus where viewers rate the quality of just one video stream (the impaired). Two examples of the former are the double stimulus continuous quality scale (DSCQS) and double stimulus comparison scale (DSCS). An example of the latter is single stimulus continuous quality evaluation (SSCQE). Each subjective test methodology has claimed advantages. For instance, the DSCQS method is claimed to be less sensitive to context (i.e., subjective ratings are less influenced by the severity and ordering of the impairments within the test session). The SSCQE method is claimed to yield more representative quality estimates for quality monitoring applications. This paper considers data from six different subjective video quality experiments, originally performed with SSCQE, DSCQS and DSCS methodologies. A subset of video clips from each of these six experiments were combined and rated in a secondary SSCQE subjective video quality test. We give a method for post-processing the secondary SSCQE data to produce quality scores that are highly correlated to the original DSCQS and DSCS data. We also provide evidence that human memory effects for time-varying quality estimation seem to be limited to about 15 seconds.

HDTV Subjective Quality of H.264 vs. MPEG-2, With and Without Packet Loss

The intent of H.264 (MPEG-4 Part 10) was to achieve equivalent quality to previous standards (e.g., MPEG-2) at no more than half the bit-rate. H.264 is commonly felt to have achieved this objective. This document presents results of an HDTV subjective experiment that compared the perceptual quality of H.264 to MPEG-2. The study included both the coding-only impairment case and a coding plus packet loss case, where the packet loss was representative of a well managed network (0.02% random packet loss rate). Subjective testing results partially uphold the commonly held claim that H.264 provides quality similar to MPEG-2 at no more than half the bit rate for the coding-only case. However, the advantage of H.264 diminishes with increasing bit rate and all but disappears when one reaches about 18 Mbps. For the packet loss case, results from the study indicate that H.264 suffers a large decrease in quality whereas MPEG-2 undergoes a much smaller decrease.

On the impact of policing and rate guarantees in DiffServ networks: a video streaming application perspective

Over the past few years, there have been a number of proposals aimed at introducing different levels of service in the Internet. One of the more recent proposals is the Differentiated Services (Diff-Serv) architecture, and in this paper we explore how the policing actions and associated rate guarantees provided by the Expedited Forwarding (EF) translate into perceived benefits for applications that are the presumed users of such enhancements. Specifically, we focus on video streaming applications that arguably have relatively strong service quality requirements, and which should, therefore, stand to benefit from the availability of some form of enhanced service. Our goal is to gain a better understanding of the relation that exists between application level quality measures and the selection of the network level parameters that govern the delivery of the guarantees that an EF based service would provide. Our investigation, which is experimental in nature, relies on a number of standard streaming video servers and clients that have been modified and instrumented to allow quantification of the perceived quality of the received video stream. Quality assessments are performed using a Video Quality Measurement tool based on the ANSI objective quality standard. Measurements were made over both a local Diff-Serv testbed and across the QBone, a QoS enabled segment of the Internet2 infrastructure. The paper reports and analyzes the results of those measurements.

The Relationship Among Video Quality, Screen Resolution, and Bit Rate

How much bandwidth is required for good quality video for a given screen resolution? Data acquired during two Video Quality Experts Group (VQEG) projects allow at least a partial answer to this question. This international subjective testing produced large amounts of mean opinion score (MOS) data for the screen resolutions QIF, CIF, VGA, and HD; for H.264 and similar modern codecs; and for many bit rates. Those data are assembled in the present report. For each screen resolution, MOS is plotted as a function of bit rate. A plot of all four data sets together shows the bit rate that would be required to achieve a given level of video quality for a given screen resolution. Relations among the four data sets are regular, suggesting that interpolation across screen resolutions might be reasonable. Based on these data, it would be reasonable to choose a bit rate, given a screen resolution; it would not be reasonable to choose a screen resolution given a bit rate.

An Objective Method for Combining Multiple Subjective Data Sets

International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. In addition to displaying the video sequences in different ways, subjective tests also have different rating scales, different words associated with these scales, and many other test variables that change from one laboratory to another (e.g., viewer expertise and criticality, cultural differences, physical test environments). Thus, it is very difficult to directly compare or combine results from two or more subjective experiments. The ability to compare and combine results from multiple subjective experiments would greatly benefit developers and users of video technology since standardized subjective data bases could be expanded upon to include new source material and past measurement results could be related to newer measurement results. This paper presents a subjective method and an objective method for combining multiple subjective data sets. The subjective method utilizes a large meta-test with selected video clips from each subjective data set. The objective method utilizes the functional relationships between objective video quality metrics (extracted from the video sequences) and corresponding subjective mean opinion scores (MOSs). The objective mapping algorithm, called the iterated nested least-squares algorithm (INLSA), relates two or more independent data sets that are themselves correlated with some common intermediate variables (i.e, the objective video quality metrics). We demonstrate that the objective method can be used as an effective substitute for the expensive and time consuming subjective meta-test.

Objective quality assessment of digitally transmitted video

The authors discuss an automated, objective video quality measurement system being developed for use in measuring end-to-end user performance. The method is based on extraction and classification of features from sampled input and output video. The extracted features quantify many of the distortions present in modern digital compression and transmission systems. These distortions include video compression artifacts resulting from spatial compression (blocking, edge blurring/smearing, etc.) and temporal compression (image persistence, jerky motion, etc.). Since the quality of the motion and still portions of a video scene can differ dramatically, an algorithm that segments each video frame into motion and still parts has been developed.<<ETX>>

Audiovisual Quality Components

The perceived quality of an audiovisual sequence is heavily influenced by both the quality of the audio and the quality of the video. The question then arises as to the relative importance of each factor and whether a regression model predicting audiovisual quality can be devised that is generally applicable.

Video Performance Requirements for Tactical Video Applications

The Public Safety Statement of Requirements (PS-SoR) for Communications & Interoperability focuses on the needs of first responders to communicate and share information as authorized, when it is needed, where it is needed, and in a mode or form that allows the practitioners to effectively use it. PS-SoR Volume I defined functional communication and interoperability requirements. Published in September, 2006, PS-SoR Volume II identifies quantitative performance metrics, including minimum video performance requirements for public safetys tactical video applications. The goal was not to identify what is achievable with current technology but rather, looking towards the future, to investigate the minimum level of performance that first responders need in order to effectively use their video equipment. On behalf of the SAFECOM Program and the Office of Law Enforcement Standards, the Institute for Telecommunication Sciences (ITS) conducted subjective video quality testing to estimate the level of video quality that first responders find acceptable for tactical video applications. This subjective testing utilized source video content that is typical of public safety operations in structured subjective viewing experiments with 35 first responders. The evaluations from these first responders, in viewing high quality video (original video) and purposefully degraded video (using video compression and transmission equipment), allowed determination of basic quality thresholds for public safety tactical video applications. These perceptual quality thresholds have been translated into technical parameters for use by video equipment designers, manufacturers, and customers. This paper summarizes those findings. Other testing to evaluate requirements for other public safety applications is underway.

The Consumer Digital Video Library [Best of the Web]

Video is a booming industry: content is embedded on many Web sites, delivered over the Internet, and streamed to mobile devices. Content providers own vast quantities of studio-quality video (i.e., produced to the quality standards of a television studio), but legal contracts between actors, producers, and owners limit how and where others can use such video. As a result, finding and getting rights to use relevant video remains an obstacle to addressing relevant research problems. The Consumer Digital Video Library (CDVL) Web site (http//:www.cdvl.org) attempts to address this obstacle. The CDVL makes high-quality, uncompressed video clips available for free download. Content owners have granted permission for use, and a use agreement protects owners? rights. The clips are ideal for use by the education, research, and product development communities. Developing objective video quality models and testing emergency telemedicine systems are two applications enabled by CDVL content.