Arthur A. Webster

Video Quality Experts Group: Current Results and Future Directions

The Video Quality Experts Group (VQEG) was formed in October 1997 to address video quality issues. The group is composed of experts from various backgrounds and affiliations, including participants from several internationally recognized organizations working int he field of video quality assessment. The first task undertaken by VQEG was to provide a validation of objective video quality measurement methods leading to recommendations in both the telecommunications and radiocommunication sectors of the International Telecommunications Union. To this end, VQEG designed and executed a test program to compare subjective video quality evaluations to the predictions of a number of proposed objective measurement methods for video quality in the bit rate range of 768 kb/s to 50 Mb/s. The results of this test show that there is no objective measurement system that is currently able to replace subjective testing. Depending on the metric used for evaluation, the performance of eight or nine models was found to be statistically equivalent, leading to the conclusion that no single model outperforms the others in all cases. The greatest achievement of this first validation effort is the unique data set assembled to help future development of objective models.

Objective quality assessment of digitally transmitted video

The authors discuss an automated, objective video quality measurement system being developed for use in measuring end-to-end user performance. The method is based on extraction and classification of features from sampled input and output video. The extracted features quantify many of the distortions present in modern digital compression and transmission systems. These distortions include video compression artifacts resulting from spatial compression (blocking, edge blurring/smearing, etc.) and temporal compression (image persistence, jerky motion, etc.). Since the quality of the motion and still portions of a video scene can differ dramatically, an algorithm that segments each video frame into motion and still parts has been developed.<<ETX>>

Validating Objective Measures of MPEG Video Quality

In 1996, the American National Standards Institute (ANSI) adopted ANSI T1.801.03, which presents a number of new objective video quality metrics for quantifying the effects of digital compression and transmission impairments. The measurements in ANSI T1.801.03 were selected based on an extensive multilaboratory quality assessment study that included video systems from bit rates of 64 kbits/sec to 45 Mbits/sec and video test scenes that spanned a wide range of spatial and temporal coding difficulties. The set of objective video quality measurements effectively accounted for subjective judgments by human viewers. While 25 video systems were tested, this multilaboratory study did not include MPEG video systems and did not cover any bit rates between 1.6 and 10 Mbits/sec. This paper presents the results from two MPEG studies designed to fill in the bit-rate gap in the previous multilaboratory study.

Audiovisual Quality Components

The perceived quality of an audiovisual sequence is heavily influenced by both the quality of the audio and the quality of the video. The question then arises as to the relative importance of each factor and whether a regression model predicting audiovisual quality can be devised that is generally applicable.

VQEG Evaluation of Objective Methods of Video Quality Assessment

This paper briefly discusses objective and subjective methods for video quality assessment by focusing on the current work plan and schedule of the Video Quality Experts Group (VQEG). Subjective assessment methods have been used reliably for many years to evaluate video image quality, yet, some issues arise with subjective assessment including the cost of conducting the evaluations and the fact that these methods cannot easily be used to monitor video image quality in realtime. Traditional analog objective measurements, while still necessary, are not sufficient to measure the quality of digitally compressed video systems. Thus, there is a need to develop new objective methods utilizing the characteristics of the human visual system. While several new objective methods have been developed, there is no internationally standardized method. The VQEG is comprised of experts from various fields and organizations working on the evaluation and standardization of methods for video quality assessment. The first and current task of VQEG is the evaluation of objective model performance against subjective results for video quality in the bit rate range of 768 kbits/sec to 50 Mbits/sec.

The history of video quality model validation

This paper describes objective video quality validation efforts conducted in the past two decades. Validation efforts to be examined include a validation test performed by the T1A1 committee in the early 1990s; five rounds of validation testing performed by the Video Quality Experts Group; and validation tests performed by ITU-T Study Group 12. Useful products that resulted from those efforts will be identified, including standards, datasets, and model validation techniques.

Video Quality Experts group : The quest for valid objective methods

Subjective assessment methods have been used reliably for many years to evaluate video quality. They continue to provide the most reliable assessments compared to objective methods. Some issues that arise with subjective assessment include the cost of conducting the evaluations and the fact that these methods cannot easily be used to monitor video quality in real time. Furthermore, traditional, analog objective methods, while still necessary, are not sufficient to measure the quality of digitally compressed video systems. Thus, there is a need to develop new objective methods utilizing the characteristics of the human visual system. While several new objective methods have been developed, there is to date no internationally standardized method. The Video Quality Experts Group (VQEG) was formed in October 1997 to address video quality issues. The group is composed of experts from various backgrounds and affiliations, including participants from several internationally recognized organizations working in the field of video quality assessment. The majority of participants are active in the International Telecommunications Union (ITU) and VQEG combines the expertise and resources found in several ITU Study Groups to work towards a common goal. The first task undertaken by VQEG was to provide a validation of objective video quality measurement methods leading to Recommendations in both the Telecommunications (ITU-T) and Radiocommunication (ITU-R) sectors of the ITU. To this end, VQEG designed and executed a test program to compare subjective video quality evaluations to the predictions of a number of proposed objective measurement methods for video quality in the bit rate range of 768 kb/s to 50 Mb/s. The results of this test show that there is no objective measurement system that is currently able to replace subjective testing. Depending on the metric used for evaluation, the performance of eight or nine models was found to be statistically equivalent, leading to the conclusion that no single model outperforms the others in all cases. The greatest achievement of this first validation effort is the unique data set assembled to help future development of objective models.