Vittorio Baroncini

Video Quality Experts Group: Current Results and Future Directions

The Video Quality Experts Group (VQEG) was formed in October 1997 to address video quality issues. The group is composed of experts from various backgrounds and affiliations, including participants from several internationally recognized organizations working int he field of video quality assessment. The first task undertaken by VQEG was to provide a validation of objective video quality measurement methods leading to recommendations in both the telecommunications and radiocommunication sectors of the International Telecommunications Union. To this end, VQEG designed and executed a test program to compare subjective video quality evaluations to the predictions of a number of proposed objective measurement methods for video quality in the bit rate range of 768 kb/s to 50 Mb/s. The results of this test show that there is no objective measurement system that is currently able to replace subjective testing. Depending on the metric used for evaluation, the performance of eight or nine models was found to be statistically equivalent, leading to the conclusion that no single model outperforms the others in all cases. The greatest achievement of this first validation effort is the unique data set assembled to help future development of objective models.

Video Quality Evaluation Methodology and Verification Testing of HEVC Compression Performance

The High Efficiency Video Coding (HEVC) standard (ITU-T H.265 and ISO/IEC 23008-2) has been developed with the main goal of providing significantly improved video compression compared with its predecessors. In order to evaluate this goal, verification tests were conducted by the Joint Collaborative Team on Video Coding of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29. This paper presents the subjective and objective results of a verification test in which the performance of the new standard is compared with its highly successful predecessor, the Advanced Video Coding (AVC) video compression standard (ITU-T H.264 and ISO/IEC 14496-10). The test used video sequences with resolutions ranging from 480p up to ultra-high definition, encoded at various quality levels using the HEVC Main profile and the AVC High profile. In order to provide a clear evaluation, this paper also discusses various aspects for the analysis of the test results. The tests showed that bit rate savings of 59% on average can be achieved by HEVC for the same perceived video quality, which is higher than a bit rate saving of 44% demonstrated with the PSNR objective quality metric. However, it has been shown that the bit rates required to achieve good quality of compressed content, as well as the bit rate savings relative to AVC, are highly dependent on the characteristics of the tested content.

Subjective evaluation of JPEG XR image compression

In this paper a procedure for subjective evaluation of the new JPEG XR codec for compression of still pictures is described in details. The new algorithm has been compared to the existing JPEG and JPEG 2000 standards when considering compression of high resolution 24 bpp pictures, by mean of a campaign of subjective quality assessment tests which followed the guidelines defined by the AIC JPEG ah-hoc group. Sixteen subjects took part in experiments at EPFL and each subject participated in four test sessions, scoring a total of 208 test stimuli. A detailed procedure for statistical analysis of subjective data is also proposed and performed. The obtained results show high consistency and allow an accurate comparison of codec performance.

Subjective evaluation of MPEG-4 video codec proposals: Methodological approach and test procedures

Abstract A new audio-visual coding standard, MPEG-4, is currently under development. MPEG-4 will address not only compression, but also completely new audio-video coding functionalities related to content-based interactivity and universal access. As part of the MPEG-4 standardization process, in November, 1995 assessments were performed on technologies proposed for incorporation in the standard. These assessments included formal subjective tests, as well as expert panel evaluations. This paper describes the MPEG-4 video formal subjective tests. Since MPEG-4 addresses new coding functionalities, and also operates at bit-rates lower than ever subjectively tested before on a large scale, standard ITU test methods were not directly applicable. These methods had to be adapted, and even new test methods devised, for the MPEG-4 video subjective tests. We describe here the test methods used in the MPEG-4 video subjective tests, how the tests were carried out, and how the test results were interpreted. We also evaluate the successes and shortcomings of the MPEG-4 video subjective tests, and suggest possible improvements for future tests. The MPEG-4 video subjective tests were successful, providing the MPEG community with critical information to guide in the selection of technologies for inclusion in the video part of the MPEG-4 standard.

Subjective testing methodology in MPEG video verification

The development of new video processing, new displays, and new modes of dissemination and usage enables a variety of moving picture applications intended for mobile and desktop devices as well as the more conventional platforms. These applications include multimedia as well as traditional video and require novel lighting environments and bit rates previously unplumbed in Moving Picture Experts Group (MPEG) video compression. The migration to new environments poses a methodological challenge to testers of video quality. Both the viewing environment and the display characteristics differ dramatically from those used in well-established subjective testing methods for television. The MPEG Test Committee has adapted the television-centric methodology to the new testing environments. The adaptations that are examined here include: (1) The display of progressive scan pictures in the Common Intermediate Format (CIF at 352x288 pixel/frame) and Quarter CIF (QCIF at 176x144 pixel/frame) as well as other, larger moving pictures requires new ways of testing the subjects including different viewing distances and altered ambient lighting. (2) The advent of new varieties of display technologies suggests there is a need for methods of characterizing them to assure the results of the testing do not depend strongly on the display. (3) The use of non-parametric statistical tests in test data analysis. In MPEG testing these appear to provide rigorous confidence statements more in line with testing experience than those provided by classical parametric tests. These issues have been addressed in a recent MPEG subjective test. Some of the test results are reviewed; they suggest that these adaptations of long-established subjective testing methodology for TV are capable of providing practical and reliable measures of subjective video quality for a new generation of technology.

Verification testing of HEVC compression performance for UHD video

This paper presents results and analysis of a compression performance test carried out within standardization activities for the HEVC standard (ITU-T H.265 and ISO/IEC 23008-2) with the focus on UHD video content. The analysis is based on subjective tests, and the performance of the new standard is compared with its predecessor AVC (ITU-T H.264 and ISO/IEC 14496-2). The Degradation Category Rating test methodology was used in a controlled laboratory environment. The test showed that a bit rate savings of more than 50 % can be achieved by HEVC for the same perceived video quality (with the tested encoding algorithms). In the high-quality coding range, an average measured bit rate savings of 60 % is reported. These results are higher than the bit rate savings measured using the PSNR objective quality metric. However, it has been shown that the bit rates required to achieve good quality of compressed content, as well as the bit rate savings relative to AVC, are highly dependent on the characteristics of the underlying UHD content. Additional analysis is presented to explain the measured results and show their dependency on the measurement metric, viewing distance, and the range of considered bit rates.

Comparison between objective and subjective measurements of quality of service over an Optical Wide Area network

The introduction of intelligence for management and control of the quality of service (QoS) are key issues for the evolution of the next generation IP optical network based on Ethernet technology. A test bed of a core-access network was implemented to investigate such issues and in this paper, the authors report measurements concerning the QoS of multimedia services for different traffic conditions. The network was implemented with a differentiated service over Multi-Protocol Label Switch (MPLS) architecture; it consists of a real optical network based on core routers with optical Gigabit Ethernet (GBE) interfaces connected by means of long single mode fibres (about 50\,km), contained in an installed cable between Rome and Pomezia. The network is based on different access devices, but in this paper we only refer to an access based on Fibre To The building (FTTB) architecture. The QoS was investigated both in terms of network (or objective) tests that include packet loss, jitter, one way delay and throughput measurements and perceptive (or subjective) tests that conversely are based on the evaluation of user perception. The services under test mainly consisted of video streams that circulated in the network according to the switching operations based in the IP differentiated service over MPLS technique that allowed us to guarantee the QoS for some class of service, also in the presence of network overload. Particular attention was given to the correlation between objective and subjective measurements. Furthermore, in order to have a complete analysis about QoS of real networks, measurements were performed also in the presence of restoration operations for link failures; in particular we compare the results in the case of conventional IP network restoration with a procedure proposed by us and based on a link switching activated by the loss of signal command coming from the routers. Copyright

Editorial: Special issue on Image and Video Quality Assessment

The aim of this special issue is to provide an overview of state-of-the-art IQA/VQA methods and to address new developments towards various directions. The issue starts with an invited paper that surveys an area of great current research interest, namely NR QA. It is followed by another survey paper, which examines audiovisual QA. The next two papers each propose a novel image quality metric, one based on gradient profiles, and the other on structural similarity. They are followed by a paper on perceptual deblocking. Video quality is the topic of the last two papers, one investigating spatio-temporal interactions in VQA, and the other the impact of visual attention on VQA.

Quality of Service Measurements over an Optical GMPLS Wide Area Access Network

The introduction of intelligence for management and control of optical networks based on internet protocol and the quality of service (QoS) are key issues for the evolution of the next generation network. In this paper the authors report experimental results on QoS measurements related to different real time multimedia services in an optical network based on a ring configuration obtained by means of the fibers contained in an installed cable between Rome and Pomezia, core routers with optical Gigabit Ethernet interfaces and configured with differentiated service over multi-protocol label switching architecture.

Variable frame rate video for mobile devices

In [1] we introduced a new real-time variable frame rate control scheme. It is based on the video jerkiness and it can be applied to both coding and transcoding. This scheme constantly traces the motion of the incoming video and automatically tunes the outgoing frame rate according to the level of jerkiness acceptable by the user. This scheme has been conceived in the framework of mobile communications, which calls for an optimum use of both the available bandwidth and terminal resources. In this paper we present a subjective assessment of our solution carried out in a professional laboratory suitably equipped. Towards this aim a group of non-expert users was asked to express their preference when watching side by side the same video coded at variable frame rate and at fixed frame rate. Results show that most of the times a variable frame rate control based on a dynamic bit/frame allocation scheme may substantially improve the video quality perceived by the users.