Gregory W. Cermak

HDTV Subjective Quality of H.264 vs. MPEG-2, With and Without Packet Loss

The intent of H.264 (MPEG-4 Part 10) was to achieve equivalent quality to previous standards (e.g., MPEG-2) at no more than half the bit-rate. H.264 is commonly felt to have achieved this objective. This document presents results of an HDTV subjective experiment that compared the perceptual quality of H.264 to MPEG-2. The study included both the coding-only impairment case and a coding plus packet loss case, where the packet loss was representative of a well managed network (0.02% random packet loss rate). Subjective testing results partially uphold the commonly held claim that H.264 provides quality similar to MPEG-2 at no more than half the bit rate for the coding-only case. However, the advantage of H.264 diminishes with increasing bit rate and all but disappears when one reaches about 18 Mbps. For the packet loss case, results from the study indicate that H.264 suffers a large decrease in quality whereas MPEG-2 undergoes a much smaller decrease.

The Relationship Among Video Quality, Screen Resolution, and Bit Rate

How much bandwidth is required for good quality video for a given screen resolution? Data acquired during two Video Quality Experts Group (VQEG) projects allow at least a partial answer to this question. This international subjective testing produced large amounts of mean opinion score (MOS) data for the screen resolutions QIF, CIF, VGA, and HD; for H.264 and similar modern codecs; and for many bit rates. Those data are assembled in the present report. For each screen resolution, MOS is plotted as a function of bit rate. A plot of all four data sets together shows the bit rate that would be required to achieve a given level of video quality for a given screen resolution. Relations among the four data sets are regular, suggesting that interpolation across screen resolutions might be reasonable. Based on these data, it would be reasonable to choose a bit rate, given a screen resolution; it would not be reasonable to choose a screen resolution given a bit rate.

Subjective Quality of Speech Over Packet Networks as a Function of Packet Loss, Delay and Delay Variation

An expanding proportion of voice traffic is being carried by packet networks. Speech quality can be impaired in qualitatively new ways in packet networks when packets are lost or the spacing between them is distorted. Three parameters that characterize the performance of packet networks were examined for their relative impact on speech quality as judged by human observers: network delay or latency, packet loss, and packet delay variation or “jitter.” We manipulated these variables via a network emulator made available by NIST. This report summarizes five laboratory experiments that examined the variables in a variety of experimental procedures for presenting and judging speech. The experiments agreed in showing that the relative importance of the variables for affecting speech quality was, in decreasing order: packet loss, jitter, delay. The effect on speech quality of 200 ms of network delay was shown to be equivalent to the effect of one percentage point of packet loss. Many consumers also traded off some speech quality for a free, added feature, unified messaging.

Multimedia Quality as a Function of Bandwidth, Packet Loss, and Latency

AbstractTwo experiments examined the judged quality of videoconferencing as a function of three measures of IP network performance: bandwidth, latency, and packet loss. The experiments were realized in a laboratory using a network emulator and a commercial videoconferencing system. Experiment 1 used a fractional factorial design and all three parameters: Bandwidth: 128 kbits/s, 384 kbits/s, 768 kbits/sLatency: 0, 150, 300 ms one-wayBursty packet loss: 0, 2%, 4% (using the “Gilbert-Eliot” method). Experiment 2 was designed (a) to use random packet loss rather than bursty packet loss, and (b) so that a statistical interaction between bandwidth and packet loss could be detected, if present. Bandwidth levels were the same as in Experiment 1, but packet loss was set to 0, 1 and 2%. The experimental design was full factorial.In both experiments, pairs of non-expert judges held five-minute videoconferences for each combination of parameters, then rated the quality of system performance immediately after each videoconference. In both experiments statistical analysis showed that packet loss was the most important network performance parameter in predicting subjective quality of videoconferencing. These results agree with results on VoIP quality. Bandwidth and latency also affected the judges’ ratings, but to a smaller extent. In Experiment 2, an interaction between packet loss and bandwidth was detected: At lower bandwidths and greater packet loss, the subjective ratings were not as low as would be expected, contrary to the idea that quality would degrade catastrophically when both bandwidth and packet loss were simultaneously unfavorable.

Verbal Descriptors for VoIP Speech Sounds

Laboratory studies with human observers were used to develop and apply a set of qualitative verbal descriptors for the sound of digital speech signals in a packet network under various loads of packet loss and jitter. The specific set of speech samples used were derived from 24 systems that were combinations of– G.729A vs. G.711 codecs,– Produced by two manufacturers,– With three levels of packet delay variation (“jitter”) introduced by a packet network emulator, and– Three levels of packet loss, also by means of the emulator.Each system was represented by a set of speech samples that had been recorded through it. A suite of verbal descriptors developed in a preliminary study was used to distinguish among the set of speech samples, and thereby the systems. Speech samples that had gone through one kind of system had a different profile of ratings than speech that had gone through other kinds of systems. Statistical analysis showed that the descriptors discriminated among all the experimental variables and most of their interactions. The system variables with the greatest effect on sound character were packet loss and a combination of codec and manufacturer. The qualitative descriptors also predicted overall subjective quality of the speech.

Speech quality as a function of IP network performance

Subjective evaluations of IP telephony were collected while network performance was varied using an IP network emulator. Packet loss was set at 0%, 1%, or 8%. Mean one-way delay was set at 8 ms, 46 ms, or 156 ms (in addition to system delay of 90 ms). Delay variation (standard deviation) was set to 1 ms, 8 ms, or 17 ms. For combinations of these network settings, a standard mean opinion score (MOS) of speech quality was collected from 31 human judges. MOS is reported as a function of the network emulator settings.

Subjective evaluation of MPEG-2 video with and without B frames

Two studies examined the effect of B-frames on subjective quality of MPEG-2 video. One study used consumer judgments in a variant of the standard CCIR Recommendation 500-5 procedure for collecting subjective evaluations. The other study used the judgments of a single expert in adjusting the bit rate necessary for MPEG-2 without B-frames to be subjectively equal to MPEG-2 with B-frames at a given bit rate. The results of the two studies were qualitatively similar. Summary of results: Picture quality improved with increase in bit rate until a saturation point was reached. The introduction of B-frames improved picture quality, especially for difficult source material. This was more noticeable at the lower bit rates (e.g., 3 Mb/s). The difference in bit rate between MPEG-2 with and without B-frames varied substantially with the source material. For example, for basketball, 5 Mb/s with B-frames was subjectively equal to 8 Mb/s without B-frames, but for other material the difference was near zero.