Philip J. Corriveau

Study of Rating Scales for Subjective Quality Assessment of High-Definition Video

With the constant evolution of video technology and the deployment of new video services, content providers and broadcasters always face the challenge of delivering an adequate video quality which meets end-users expectations. The development of reliable quality testing and quality monitoring tools that can be used by broadcasters ultimately requires reliable objective video quality metrics. In turn, the validation of these objective models requires reliable subjective assessment, the most accurate representation of the quality perceived by end-users. Many different subjective assessment methodologies exist, and each has its advantages and drawbacks. One important element in a subjective testing methodology is the choice of the rating scale. In this paper, we make a direct comparison between four scales, which are either included in existing international standards or proposed to be used in future standardization activities related to video quality. We examine the subjective data from the points of view of response behavior from participants, similarity and variability of subjective scores. We discuss these results within the context of the subjective quality assessment of high-definition video compressed and transmitted over error-prone networks. Our experimental data show no overall statistical differences between the different scales. Results also show that the single-stimulus presentation provides highly repeatable results even if different scales or groups of participants are used.

Psychovisual aspects of viewing stereoscopic video sequences

In comparison to conventional displays, 3D stereoscopic displays convey additional information about the 3D structure of a scene by providing information that can be used to extract depth. In the present study we evaluated the psychovisual impact of stereoscopic images on viewers. Thirty-three non-expert viewers rated sensation of depth, perceived sharpness, subjective image quality, and relative preference for stereoscopic over non-stereoscopic images. Rating methods were based on procedures described in ITU- Rec. 500. Viewers also rated sequences in which the left- and right-eye images were processed independently, using a generic MPEG-2 codec, at bit-rates of 6, 3, and 1 Mbits/s. The main finding was that viewers preferred the stereoscopic version over the non-stereoscopic version of the sequences, provided that the sequence did not contain noticeable stereo artifacts, such as exaggerated disparity. Perceived depth was rated greater for stereoscopic than for non-stereoscopic sequences, and perceived sharpness of stereoscopic sequences was rated the same or lower compared to non-stereoscopic sequences. Subjective image quality was influenced primarily by apparent sharpness of the video sequences, and less so by perceived depth.

Human perception of mismatched stereoscopic 3D inputs

The bandwidth required to transmit stereoscopic video images is nominally twice that required for standard, monoscopic images. One method of reducing the required bandwidth is to code the two video streams asymmetrically. We assessed the perceptual impact of this bandwidth-reduction technique for low-pass filtering, DCT-based quantization, and a combination of filtering and quantization. It was found that the binocular percept depended on the type of degradation: for low-pass filtering, the binocular percept was dominated by the high-quality image, whereas for quantization it corresponded to the average of the inputs to the two eyes. The results indicated that asymmetrical coding is a promising technique for reducing storage and transmission bandwidth of stereoscopic sequences.

Stereoscopic Viewing and Reported Perceived Immersion and Symptoms

Purpose. Stereoscopic 3D displays heighten perceived immersion but elevate viewing symptoms for some viewers. The present study measured prevalence and magnitude of perceived immersion and viewing symptoms in stereoscopic viewing, and related them to viewers characteristics and viewing position. Methods. Two hundred three teens and adults viewed a movie in 2D or 3D while sitting at different angles and distances. Their prior viewing symptoms, as well as visual and physical discomfort immediately before and after viewing, were measured with questionnaires. They were also asked to report their perceived immersion after the viewing. Results. Twelve percent and twenty-one percent of 2D and stereoscopic 3D participants reported increases of measured symptoms during and/or after viewing. Stereoscopic 3D viewing incurred greater and more frequent perception of blurred vision, double vision, dizziness, disorientation, and nausea than 2D viewing. Reported ocular and physical symptoms were negatively correlated to perceived immersion in 3D viewing. Older viewers (age 46 years or older) reported greater ocular, visual, and motion sickness symptoms in 2D viewing, and younger viewers (age 24–34 years) reported greater visual and motion sickness symptoms in 3D viewing. Sitting in an oblique position attenuated perceived immersion but also reduced motion symptoms in 3D viewing. Prior viewing symptoms in 2D tasks also predicted ocular and physical symptoms in 2D but less so in 3D viewing. Conclusions. Stereoscopic 3D viewing provides greater immersion, but it can also lead to heightened visual and motion sickness symptoms. Viewers with prior symptoms in viewing TV and computer screen are not more likely to have increased ocular and physical symptoms in 3D viewing. Young viewers incurred higher immersion but also greater visual and motion sickness symptoms in 3D viewing; both will be reduced if a farther distance and a wider viewing angle are adopted.

The Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study

Traditionally, audio quality and video quality are evaluated separately in subjective tests. Best practices within the quality assessment community were developed before many modern mobile audiovisual devices and services came into use, such as internet video, smart phones, tablets and connected televisions. These devices and services raise unique questions that require jointly evaluating both the audio and the video within a subjective test. However, audiovisual subjective testing is a relatively under-explored field. In this paper, we address the question of determining the most suitable way to conduct audiovisual subjective testing on a wide range of audiovisual quality. Six laboratories from four countries conducted a systematic study of audiovisual subjective testing. The stimuli and scale were held constant across experiments and labs; only the environment of the subjective test was varied. Some subjective tests were conducted in controlled environments and some in public environments (a cafeteria, patio or hallway). The audiovisual stimuli spanned a wide range of quality. Results show that these audiovisual subjective tests were highly repeatable from one laboratory and environment to the next. The number of subjects was the most important factor. Based on this experiment, 24 or more subjects are recommended for Absolute Category Rating (ACR) tests. In public environments, 35 subjects were required to obtain the same Students t-test sensitivity. The second most important variable was individual differences between subjects. Other environmental factors had minimal impact, such as language, country, lighting, background noise, wall color, and monitor calibration. Analyses indicate that Mean Opinion Scores (MOS) are relative rather than absolute. Our analyses show that the results of experiments done in pristine, laboratory environments are highly representative of those devices in actual use, in a typical user environment.

All subjective scales are not created equal: the effects of context on different scales

Abstract Subjective evaluation of image quality is an important tool in video research and broadcasting. Several methods have been recommended for the subjective evaluation of image quality: the double-stimulus continuous-quality scale method (DSCQS), the double stimulus impairment scale method variant II (DSIS II), and a newly proposed comparison method. In order to evaluate the effect of context on different scales, each scale was used to rate the same video sequences. The ratings from the scales were strongly correlated. The degree to which each method was affected by contextual effects was assessed. The results have revealed no contextual effects for the DSCQS method, moderate contextual effects for the comparison method and large contextual effects for the DSIS II method. It was concluded that the DSCQS method is the best method to use in order to minimize contextual effects for subjective picture quality assessment.

Objective Consumer Device Photo Quality Evaluation

We propose an approach to no-reference image quality evaluation that is consumer-centric and targets real consumer-type images with realistic distortions and realistic quality ranges. We show that the state-of-the-art approaches to no-reference image quality assessment do not perform as well on real consumer-type images, and propose an approach which is simple, efficient, easily interpretable, and achieves high prediction performance on a dataset of images with real non-simulated distortions.

Determining the Amount of Audio-Video Synchronization Errors Perceptible to the Average End-User

The media and acoustics perception lab (MAPL) designed a study to determine the minimum amount of audio-visual synchronization (a/v sync) errors that can be detected by end-users. Lip synchronization is the most noticeable a/v sync error, and was used as the testing stimuli to determine the perceptual threshold of audio leading errors. The results of the experiment determined that the average audio leading threshold for a/v sync detection was 185.19 ms, with a standard deviation of 42.32 ms. This threshold determination of lip sync error (with audio leading) will be widely used for validation and verification infrastructures across the industry. By implementing an objective pass/fail value into software, the system or network under test is held against criteria which were derived from a scientific subjective test.

Impact of camera pixel count and monitor resolution perceptual image quality

Traditional 35mm film cameras are no longer the main devices todays consumers use to capture images. Though the dominant technology has shifted to digital cameras and displays that differ widely in pixel count and resolution, our understanding of the quality impact of these variables lags. This paper explores the quality impact of resolution within this new paradigm. Images were collected from 23 cameras, ranging from a 1 megapixel (MP) mobile phone to a 20 MP digital single-lens reflex camera (DSLR). Subjective ratings from three labs were used to explore the relationship between the cameras pixel count, the display resolution, and the overall perceived quality. This dataset and subjective ratings will be made available on the Consumer Digital Video Library (CDVL, www.cdvl.org) when this paper is published. These images can be used royalty free for research and development purposes.

Depth perception from stationary and moving stereoscopic three-dimensional images

Purpose: The study evaluated the accuracy of depth perception afforded by static and dynamic stereoscopic three-dimensional (S3D) images with proportional (scaled to disparity) and constant size cues. Methods: Sixty adult participants, 18 to 40 years (mean, 24.8 years), with good binocular vision participated in the study. For static S3D trials, participants were asked to indicate the depth of stationary S3D images rendered with 36, 48 and 60 pixels of crossed disparity, and with either proportional or a constant size. For dynamic S3D trials, participants were asked to indicate the time when S3D images, moving at 27, 32 and 40 pixels/sec, matched the depth of a reference image which was presented with 36, 48 and 60 pixels of crossed image disparity. Results: Results show that viewers perceived S3D images as being closer than would be predicted by the magnitude of image disparity, and correspondingly they overestimated the depth in moving S3D images. The resultant depth perception and estimate of motion speed were more accurate for conditions with proportional and larger image size, slower motion-in-depth and larger image disparity. Conclusion: These findings possibly explain why effects such as looming are over stimulating in S3D viewing. To increase the accuracy of depth perception, S3D content should match image size to its disparity level, utilize larger depth separation (without inducing excessive discomfort) and render slower motion in depth.