Lark Kwon Choi

Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies

We introduce a new video quality database that models video distortions in heavily-trafficked wireless networks and that contains measurements of human subjective impressions of the quality of videos. The new LIVE Mobile Video Quality Assessment (VQA) database consists of 200 distorted videos created from 10 RAW HD reference videos, obtained using a RED ONE digital cinematographic camera. While the LIVE Mobile VQA database includes distortions that have been previously studied such as compression and wireless packet-loss, it also incorporates dynamically varying distortions that change as a function of time, such as frame-freezes and temporally varying compression rates. In this article, we describe the construction of the database and detail the human study that was performed on mobile phones and tablets in order to gauge the human perception of quality on mobile devices. The subjective study portion of the database includes both the differential mean opinion scores (DMOS) computed from the ratings that the subjects provided at the end of each video clip, as well as the continuous temporal scores that the subjects recorded as they viewed the video. The study involved over 50 subjects and resulted in 5,300 summary subjective scores and time-sampled subjective traces of quality. In the behavioral portion of the article we analyze human opinion using statistical techniques, and also study a variety of models of temporal pooling that may reflect strategies that the subjects used to make the final decision on video quality. Further, we compare the quality ratings obtained from the tablet and the mobile phone studies in order to study the impact of these different display modes on quality. We also evaluate several objective image and video quality assessment (IQA/VQA) algorithms with regards to their efficacy in predicting visual quality. A detailed correlation analysis and statistical hypothesis testing is carried out. Our general conclusion is that existing VQA algorithms are not well-equipped to handle distortions that vary over time. The LIVE Mobile VQA database, along with the subject DMOS and the continuous temporal scores is being made available to researchers in the field of VQA at no cost in order to further research in the area of video quality assessment.

Referenceless perceptual fog density prediction model

We propose a perceptual fog density prediction model based on natural scene statistics (NSS) and “fog aware” statistical features, which can predict the visibility in a foggy scene from a single image without reference to a corresponding fogless image, without side geographical camera information, without training on human-rated judgments, and without dependency on salient objects such as lane markings or traffic signs. The proposed fog density predictor only makes use of measurable deviations from statistical regularities observed in natural foggy and fog-free images. A fog aware collection of statistical features is derived from a corpus of foggy and fog-free images by using a space domain NSS model and observed characteristics of foggy images such as low contrast, faint color, and shifted intensity. The proposed model not only predicts perceptual fog density for the entire image but also provides a local fog density index for each patch. The predicted fog density of the model correlates well with the measured visibility in a foggy scene as measured by judgments taken in a human subjective study on a large foggy image database. As one application, the proposed model accurately evaluates the performance of defog algorithms designed to enhance the visibility of foggy images.

Flicker sensitive motion tuned video quality assessment

From a series of human subjective studies, we have found that large motion can strongly suppress flicker visibility. Based on the spectral analysis of flicker videos in frequency domain, we propose a full reference video quality assessment (VQA) framework that incorporates flicker sensitive temporal visual masking. The framework predicts perceptually silenced flicker visibility using a model of the responses of primary visual cortex to video flicker, a motion energy model, and divisive normalization. By incorporating perceptual flicker visibility into motion tuned video quality measurements as in the MOVIE framework, we augment VQA performance with sensitivity to flicker. Results show that the proposed VQA framework correlates well with human results and is highly competitive with recent state-of-the-art VQA algorithms tested on the LIVE VQA database.

Perceptual Flicker Visibility Prediction Model.

The mere presence of spatiotemporal distortions in digital videos does not have to imply quality degradation since distortion visibility can be strongly reduced by the perceptual phenomenon of visual masking. Flicker is a particularly annoying occurrence, which can arise from a variety of distortion processes. Yet flicker can also be suppressed by masking. We propose a perceptual flicker visibility prediction model which is based on a recently discovered visual change silencing phenomenon. The proposed model predicts flicker visibility on both static and moving regions without any need for content-dependent thresholds. Using a simple model of cortical responses to video flicker, an energy model of motion perception, and a divisive normalization stage, the system captures the local spectral signatures of flicker distortions and predicts perceptual flicker visibility. The model not only predicts silenced flicker distortions in the presence of motion, but also provides a pixel-wise flicker visibility index. Results show that the predicted flicker visibility model correlates well with human percepts of flicker distortions tested on the LIVE Flicker Video Database and is highly competitive with current flicker visibility prediction methods. Introduction Digital videos are increasingly pervasive due to the rapid proliferation of video streaming services, video sharing in social networks, and the global increase of mobile video traffic [1], [2]. The dramatic growth of digital videos and user demand for highquality video have necessitated the development of precise automatic perceptual video quality assessment (VQA) tools to help provide satisfactory levels of Quality of Experience (QoE) to the end user [3]. To achieve optimal video quality under limited bandwidth and power consumption, video coding technologies commonly employ lossy coding schemes, which cause compression artifacts that can lead to degradation of perceptual video quality [4]. In addition, compressed videos can suffer from transmission distortions, including packet losses and playback interruptions triggered by channel throughput fluctuations. Since humans are generally the ultimate arbiter of the received videos, predicting and reducing perceptual visual distortions of compressed digital videos is of great interest [5]. Researchers have performed a large number of subjective studies to understand essential factors that influence video quality by analyzing compression artifacts or transmission distortions of the compressed videos [6], by investigating dynamic time varying distortions [7], and by probing the time varying subjective quality of rate adaptive videos [8]. Substantial progress has also been made toward understanding and modeling low-level visual processes in the vision system extending from the retina to primary visual cortex and extra-striate cortex [9]. As a result, perceptual models of disruptions to natural scene statistics [10] and of visual masking [11] have been widely applied to predict perceptual visual quality. Spatial distortions are effectively predicted by VQA algorithms such as SSIM [12], VQM [13], MOVIE [14], STRRED [15], and Video-BLIINDS [16]. Spatial masking is well-modeled in modern perceptual image and video quality assessment tools, video compression, and watermarking. However, temporal visual masking is not well-modeled although one type of it has been observed to occur near scene changes [17], and been used in the context of early video compression methods [18-20]. Among temporal distortions, flicker distortion is particularly challenging to predict and often occurs on low bit-rate compressed videos. Flicker distortion is (spatially local or global) temporal fluctuation of luminance or chrominance in videos. Local flicker occurs mainly due to coarse quantization, varying prediction modes, mismatching of inter-frame blocks, improper deinterlacing, and dynamic rate changes caused by adaptive rate control methods [21-25]. Mosquito noise and stationary area fluctuations are also often categorized under local flicker. Mosquito noise is a joint effect of object motion and time-varying spatial artifacts such as ringing and motion prediction errors near high-contrast sharp edges or moving objects, while stationary area fluctuations result from different types of prediction, quantization levels, or a combination of these factors on static regions [4], [21]. Current flicker visibility prediction methods that operate on a compressed video measure the Sum of Squared Differences (SSD) between the block difference of an original video and the block difference of a compressed video. The block difference is obtained between successive frames on macroblocks. When the sum of squared block differences on an original video falls below a threshold, a static region is indicated [22]. The ratio between luminance level fluctuation in the compressed video and that in the original video has also been used [23]. To improve the prediction of flicker-prone blocks, a normalized fraction model was proposed [24], where the difference of SSDs between the original and compressed block differences is divided by the sum of the SSDs. These methods have the virtue of simplicity, but the resulting flicker prediction performance is limited and content-dependent. Another method included the influence of motion on flicker prediction, where motion compensation was applied prior to SSD calculation [25]. The mean absolute discrete temporal derivatives of the average DC coefficient of DCT blocks was used to measure sudden local changes (flicker) in a VQA model [16]. Current flicker prediction methods are limited to block-wise accuracy. Further, human visual system (HVS)-based perceptual flicker visibility e.g., considering temporal visual masking, has not yet been extensively studied. Recently, Suchow and Alvarez [26] demonstrated a striking “motion silencing” illusion, in the form of a powerful temporal visual masking phenomenon called change silencing, where the salient temporal changes of objects in luminance, color, size, and shape appear to cease in the presence of large object motions. This motion-induced failure to detect change not only suggests a tight coupling between motion and object appearance, but also reveals that commonly occurring temporal distortions such as flicker may be dramatically suppressed by the presence of motion. To understand the mechanism of motion silencing, physiologically plausible explanations have been proposed [26-29]. However, since the effect has only been studied on highly synthetic stimuli such as moving dots, we performed a series of human subjective studies on naturalistic videos, where flicker visibility is observed to be strongly reduced by large coherent object motions [30-33]. A consistent physiological and computational model that detects motion silencing might be useful to probe perceptual flicker visibility on compressed videos. In this paper, we propose a new perceptual flicker visibility prediction model based on motion silencing. The new perceptual flicker visibility prediction model is a significant step towards improving the performance of VQA models by making possible a model of temporal masking of temporal distortions. The new model measures the bandpass filter responses to a reference video and a corresponding flicker video using a localized multiscale 3D space time Gabor filter bank [34], [35], a spatiotemporal energy model of motion perception [36], and a divisive normalization model of nonlinear gain control in primary visual cortex [37]. We observed that flicker produces locally separated spectral signatures that almost lie along the same orientation as the motion tuned plane of the reference video but at a distance. The captured V1 responses for the flicker induced spectral signatures generally decreased when object speeds increase. Next, we measured the local difference of bandpass responses at each space-time frequency orientation and defined the sum of the magnitude responses as a perceptual flicker visibility index. The proposed model predicts temporal masking effects on flicker distortions and thereby shows highly competitive performance against previous flicker visibility prediction methods. Background: Motion Perception Motion perception is the process of inferring the speed and direction of moving objects. Since motion perception is important for understanding flicker distortions in videos, we model motion perception in the frequency domain. Watson and Ahumada [38] proposed a model of how humans sense the velocity of moving images, where the motion-sensing elements appear locally tuned to specific spatiotemporal frequencies. Assuming that complex motions of video without any scene changes can be constructed by piecing together spatiotemporally localized image patches undergoing translation, we can model the local spectral signatures of videos when an image patch moves [38]. An arbitrary space-time image patch can be represented by a function a(x, y, t) at each point x, y, and time t, and its Fourier transform by A(u, v, w) where u, v, and w are spatial and temporal frequency variables corresponding to x, y and t, respectively. Let λ and φ denote the image patch horizontal and vertical velocity components. When an image patch translates at constant velocity [λ, φ], the moving video sequence becomes b(x, y, t) = a(x – λt, y – φt, t). The spectrum of a stationary image patch lies on the u, v plane, while the Fourier transform shears into an oblique plane through the origin when the image patch moves. The orientation of this plane indicates the speed and direction of motion. Prediction of Perceptual Flicker Visibility Linear Decomposition Natural environments are inherently multi-scale and multiorientation, and objects move multi-directionally at diverse speeds. To efficiently encode visual signals, the vision system decomposes (a) (b) (c) Figure 1. Gabor filter bank in the frequency domain. (a) G

Referenceless perceptual image defogging

We propose a referenceless perceptual defog and visibility enhancement model based on multiscale “fog aware” statistical features. Our model operates on a single foggy image and uses a set of “fog aware” weight maps to improve the visibility of foggy regions. The proposed defog and visibility enhancer makes use of statistical regularities observed in foggy and fog-free images to extract the most visible information from three processed image results: one white balanced and two contrast enhanced images. Perceptual fog density, fog aware luminance, contrast, saturation, chrominance, and saliency weight maps smoothly blend these via a Laplacian pyramid. Evaluation on a variety of foggy images shows that the proposed model achieves better results for darker, denser foggy images as well as on standard defog test images.

Visibility prediction of flicker distortions on naturalistic videos

We conducted a series of human subjective studies where we found that the visibility of flicker distortions on naturalistic videos is strongly reduced when the speed of coherent object motion is large. Based on this, we propose a model of flicker visibility on naturalistic videos. The model predicts target-related activation levels in the excitatory layer of neurons for a video using spatiotemporal backward masking. The target-related activation level is then shifted and scaled according to video quality level changes that cause flicker distortions. Finally, flicker visibility is predicted based on neural flicker adaptation processes. Results show that the predicted flicker visibility using the model correlates well with human perception of flicker distortions on naturalistic videos.

Video quality assessment accounting for temporal visual masking of local flicker

Abstract An important element of the design of video quality assessment (VQA) models that remains poorly understood is the effect of temporal visual masking on the visibility of temporal distortions. The visibility of temporal distortions like local flicker can be strongly reduced by motion. Based on a recently discovered visual change silencing illusion, we have developed a full reference VQA model that accounts for temporal visual masking of local flicker. The proposed model, called Flicker Sensitive-MOtion-based Video Integrity Evaluation (FS-MOVIE), augments the well-known MOVIE Index by combining motion tuned video integrity features with a new perceptual flicker visibility/masking index. FS-MOVIE captures the separated spectral signatures caused by local flicker distortions, by using a model of the responses of neurons in primary visual cortex to video flicker, an energy model of motion perception, and a divisive normalization stage. FS-MOVIE predicts the perceptual suppression of local flicker by the presence of motion and evaluates local flicker as it affects video quality. Experimental results show that FS-MOVIE significantly improves VQA performance against its predecessor and is highly competitive with top performing VQA algorithms when tested on the LIVE, IVP, EPFL, and VQEGHD5 VQA databases.

Prediction of perceived fog density and defogging of natural foggy images

The perception of outdoor natural scenes is important for successfully participating in visual activities. When fog is present, however, visibility can be severely degraded. We have created a perceptual fog density index that seeks to predict the degree of visibility of a foggy scene using ‘fog aware’ features learned from a representative database of foggy and fog-free images. The features that define the fog density index derive from a spatial natural scene statistics (NSS) model and from observed characteristics of foggy images. We have found that the statistics of fog aware features consistently change as a function of fog density. A perceptual fog density model was derived by collecting patches from natural foggy and fog-free images, then computing pertinent NSS fog aware features from these patches to develop a perceptual fog density prediction model. A multivariate Gaussian (MVG) distribution is used to form a probabilistic feature model. In practice the perceived fog density of an arbitrary test image is predicted using a Mahalanobis-like distance measure between the ‘fog aware’ statistics of the test image and the MVG models obtained from the natural foggy and fog-free images. We also conducted a human study of perceived fog density. When applied to 100 natural foggy images, the predicted perceived fog density was found to correlate well with the human judgments of fog density. We have also found that the fog aware statistical features can be used to defog and thereby enhance the visibility of foggy scenes. We first produced white-balanced and contrast-enhanced images from a foggy image using the predicted visibility degree. We then selectively filtered these images with fog aware weighted maps representing distances from the statistics of fog-free images at each fog aware feature. Finally, we applied a Laplacian multiscale pyramidal refinement to achieve a halo-free defogged image.

Motion Silences the Perception of Changing Image Quality in Naturalistic Videos

Failure to detect changes in stimulus luminance (or color, etc.) termed “silencing” occurs in the presence of rapid motion (Suchow and Alvarez, 2011). It is possible that silencing is useful in naturalistic contexts, serving to suppress the perception of cast shadows on moving objects, say, or preventing video artifacts from being distracting. We conducted human experiments examining the perception of “flicker” caused by variation in image quality for moving objects in videos. 42 naive subjects evaluated the amount of perceived flicker of moving objects in the random ordered 36 test videos, which included six flicker-free reference videos and 30 degraded, flickering videos. The flicker was generated by alternating video frames of different quality levels (e.g.: bad, poor, good, and excellent). An eye and head tracker (faceLAB 5, Seeing Machines) was used to monitor gaze position (subjects’ heads were unrestrained), and subjects reported their percepts by moving a mouse continuously throughout the stimulus presentation. The results indicate that the reduction of the visibility of flickering in natural videos depends on both the overall video quality and the speed of motion. When the video quality was high, the perceptual visibility of flickering is low and less sensitive to motion, whereas when the video quality was poor, the impact is large. Furthermore, although subjects held their gaze on the moving objects, lessflicker was seen on fast-moving objects. We interpret this result to suggest that large coherent motions near the fixation point might silence the awareness of flickering on natural videos. The responsible mechanism might be useful in the real world, where the light coming from moving objects can change dramatically (due to passing through areas of light and shadow, e.g.) even for a rigid object with a constant trajectory.