Audrey C. Younkin

The Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study

Traditionally, audio quality and video quality are evaluated separately in subjective tests. Best practices within the quality assessment community were developed before many modern mobile audiovisual devices and services came into use, such as internet video, smart phones, tablets and connected televisions. These devices and services raise unique questions that require jointly evaluating both the audio and the video within a subjective test. However, audiovisual subjective testing is a relatively under-explored field. In this paper, we address the question of determining the most suitable way to conduct audiovisual subjective testing on a wide range of audiovisual quality. Six laboratories from four countries conducted a systematic study of audiovisual subjective testing. The stimuli and scale were held constant across experiments and labs; only the environment of the subjective test was varied. Some subjective tests were conducted in controlled environments and some in public environments (a cafeteria, patio or hallway). The audiovisual stimuli spanned a wide range of quality. Results show that these audiovisual subjective tests were highly repeatable from one laboratory and environment to the next. The number of subjects was the most important factor. Based on this experiment, 24 or more subjects are recommended for Absolute Category Rating (ACR) tests. In public environments, 35 subjects were required to obtain the same Students t-test sensitivity. The second most important variable was individual differences between subjects. Other environmental factors had minimal impact, such as language, country, lighting, background noise, wall color, and monitor calibration. Analyses indicate that Mean Opinion Scores (MOS) are relative rather than absolute. Our analyses show that the results of experiments done in pristine, laboratory environments are highly representative of those devices in actual use, in a typical user environment.

Subjective and objective evaluation of an audiovisual subjective dataset for research and development

In 2011, the Video Quality Experts Group (VQEG) ran subjects through the same audiovisual subjective test at six different international laboratories. That small dataset is now publically available for research and development purposes.

Determining the Amount of Audio-Video Synchronization Errors Perceptible to the Average End-User

The media and acoustics perception lab (MAPL) designed a study to determine the minimum amount of audio-visual synchronization (a/v sync) errors that can be detected by end-users. Lip synchronization is the most noticeable a/v sync error, and was used as the testing stimuli to determine the perceptual threshold of audio leading errors. The results of the experiment determined that the average audio leading threshold for a/v sync detection was 185.19 ms, with a standard deviation of 42.32 ms. This threshold determination of lip sync error (with audio leading) will be widely used for validation and verification infrastructures across the industry. By implementing an objective pass/fail value into software, the system or network under test is held against criteria which were derived from a scientific subjective test.

Organic experiences: (re)shaping interactions with deformable displays

Display technology developments mean the next generation of visual output devices will extend beyond the rigid, flat surfaces with which we are familiar to those that the user or the machine can deform. These will allow users to physically push, pull, bend, fold or flex the display and facilitate a range of self-deformation to better represent on-screen content or support new modes of interaction. This workshop will provide a forum to examine, discuss and shape the three primary themes of research in this area: prototyping and implementation, interaction and experience design, and evaluation. It will bring together an interdisciplinary group of academic and industrial researchers to define the current and future challenges of crafting organic user experiences with deformable displays.

Depth perception from stationary and moving stereoscopic three-dimensional images

Purpose: The study evaluated the accuracy of depth perception afforded by static and dynamic stereoscopic three-dimensional (S3D) images with proportional (scaled to disparity) and constant size cues. Methods: Sixty adult participants, 18 to 40 years (mean, 24.8 years), with good binocular vision participated in the study. For static S3D trials, participants were asked to indicate the depth of stationary S3D images rendered with 36, 48 and 60 pixels of crossed disparity, and with either proportional or a constant size. For dynamic S3D trials, participants were asked to indicate the time when S3D images, moving at 27, 32 and 40 pixels/sec, matched the depth of a reference image which was presented with 36, 48 and 60 pixels of crossed image disparity. Results: Results show that viewers perceived S3D images as being closer than would be predicted by the magnitude of image disparity, and correspondingly they overestimated the depth in moving S3D images. The resultant depth perception and estimate of motion speed were more accurate for conditions with proportional and larger image size, slower motion-in-depth and larger image disparity. Conclusion: These findings possibly explain why effects such as looming are over stimulating in S3D viewing. To increase the accuracy of depth perception, S3D content should match image size to its disparity level, utilize larger depth separation (without inducing excessive discomfort) and render slower motion in depth.

The effects of quantity and location of video playback errors on the average end-user’s experience

The end-user experience of a given platform or product with respect to video playback is an increasingly important aspect for engineers and product planners to understand. However, testing real world situations in an objective and repeatable fashion is complex. In an initial study, the video Gross Error Detector (GED) which is a tool that provides a quick and cost-efficient way to evaluate video playback, was mapped to end-users perception of video smoothness. Video errors such as dropped, repeated or out of sequence frames were evaluated in varying quantities to understand the impact on the average en-users perception of playback. A second study, discussed here, was done to understand the differences in the quantity and location of these playback errors. These end-user perception studies provide useful data so the video GED can be used to monitor errors in video for quality control, benchmark video processing and algorithms, and can be rooted into the video processing system to optimize algorithms and limit settings.

Discernible difference and change in object depth afforded by stereoscopic three-dimensional content

Purpose: Stereoscopic three-dimensional (S3D) viewing enhances depth perception of two-dimensional (2D) images. The present study measured viewer’s ability to discern depth differences and depth change afforded by image disparities presented on an S3D display. Methods: Sixty adults (age 24.8 +/- 3.4 years, 28% male) with binocular acuity better than 20/25 and stereoacuity better than 60 arcsec viewed test images presented on a 55” 3D TV (1920 x 1080 pixels) at 2.4m distance. In each trial, three of the four circles in the test image were with the same crossed baseline disparity of 12, 24, 36, 48, and 60 pixels, and the other (target) circle with added crossed disparity (delta disparity) of 2, 4, and 6 pixels. A subsequent change in delta disparity of the target circle (+/-2 pixels) was presented. Participants response time and accuracy for identifying the target circle and its direction of depth change, as well as their vergence eye position, was recorded. Results: Larger baseline disparity resulted in lower accuracy and longer response times in identifying the target circle. The change of delta disparity was more accurately discerned when the delta disparity was larger and the change resulted in increased crossed disparity. Direction of vergence change and the final vergence error reflected an averaging of screen and image depths. Conclusion: S3D-induced depth difference and motion in depth is more easily discerned with smaller crossed disparity for background objects (< 43.3 arcmin or 48 pixels) and larger separation between image disparities (>3.6 arcmin or 4 pixels).