Randi Fu

Binocular energy response based quality assessment of stereoscopic images

Abstract Perceptual quality assessment in three-dimensional (3D) is challenging. In this research, we propose a binocular energy response based quality assessment metric of stereoscopic images. To be more specific, we first construct binocular energy responses of the original and distorted images, and measure the similarity between them as Image Quality Metric (IQM). Then, the binocular response and the binocular masking components are used to modulate the IQM, respectively. Finally, two evaluation results are nonlinearly integrated into an overall score by considering the importance of each component. Experimental results show that compared with the relevant existing metrics, the proposed metric can achieve higher consistency with the subjective assessment of stereoscopic image.

New method of ray-space interpolation for free viewpoint video

Ray-space representation has superiority in rendering arbitrary viewpoint images of complicated scene in real-time. Ray-space interpolation is one of the key techniques to make ray-space based free viewpoint video (FVV) feasible. This paper presents a directionality based interpolation method for ray-space based FW system. Characteristic pixels near edges or within areas with fine texture are first extracted from sparse ray-space slice, and their directionalities are determined by multi-stage matching technique, so as to speed up the directionality searching and obtain more reliable directionalities. Pixels to be interpolated in dense ray-space slice are linear interpolated according to the directionalities of these characteristic pixels. Experimental results show that the proposed method improves visual quality as well as PSNRs of rendered intermediate viewpoint image greatly.

Stereoscopic Visual Attention Guided Seam Carving for Stereoscopic Image Retargeting

Stereoscopic image retargeting plays an important role in adaptive 3D stereoscopic displays. It aims to fit displays with various resolutions while preserving visually salient content and geometric consistency. We propose a stereoscopic image retargeting method based on stereoscopic visual attention guided seam carving. Firstly, stereoscopic saliency map is generated by combining 2D saliency and depth saliency maps, and significant energy map is generated by considering binocular disparity binocular and binocular just-noticeable-difference (BJND). Then, seam selection is applied to the left image based on stereoscopic saliency and energy maps, and seam replacement is performed for the occluded regions to prevent the geometry inconsistency. Finally, according to the matched left and right seams, the retargeted stereoscopic image is generated. In the experiments, subjective and objective analysis on three stereoscopic image databases shows that the proposed approach produces better seam carving results than the related existing methods.

Local and Global Sparse Representation for No-Reference Quality Assessment of Stereoscopic Images

Abstract No-reference/blind quality assessment of stereoscopic 3D images is much more challenging than 2D images due to the poor understanding of binocular vision. In this paper, we propose a BLind Quality Evaluator for stereoscopic 3D images by learning Local and Global Sparse Representations (BLQELGSR). Specifically, at the training stage, we first construct a large-scale training set by simulating some common distortions that are likely encountered by stereoscopic images, and propose a multi-modal sparse representation framework to characterize the relationship between the feature and quality spaces for all sources of information from left, right and cyclopean views in local and global manners. At the testing stage, based on the derived 3D quality prediction framework, the local and global quality scores from different sources are predicted and combined to drive a final 3D quality score. Experimental results on three 3D image quality databases show that in comparison with the existing methods, the devised BLQELGSR can achieve better prediction performance to be in line with subjective assessment.

Optimizing visual comfort for stereoscopic 3D display based on color-plus-depth signals

Visual comfort is a long-facing problem in stereoscopic 3D (S3D) display. In this paper, targeting to produce S3D content based on color-plus-depth signals, a general framework for depth mapping to optimize visual comfort for S3D display is proposed. The main motivation of this work is to remap the depth range of color-plus-depth signals to a new depth range that is suitable to comfortable S3D display. Towards this end, we first remap the depth range globally based on the adjusted zero disparity plane, and then present a two-stage global and local depth optimization solution to solve the visual comfort problem. The remapped depth map is used to generate the S3D output. We demonstrate the power of our approach on perceptually uncomfortable and comfortable stereoscopic images.

New Sampling Strategy in Asymmetric Stereoscopic Video Coding for Mobile Devices

Asymmetric stereoscopic coding can save bitrate and maintain the overall visual quality of stereoscopic perception, so it is an alternative for wireless environment in which the bandwidth is limited. In asymmetric stereoscopic coding, the right-view images are down-sampled to half resolution before coding, while the direction of sampling influences the rate-distortion performance of the asymmetric stereoscopic coding. In this paper, a new scheme to prejudge the down-sampling direction for the asymmetric stereoscopic coding is proposed. The sum of absolute transformed difference(SATD) of 16×16 macroblocks intra-predicated vertically and horizontally in the left-view image are calculated so as to judge the relativities in vertical and horizontal directions. Finally, the direction with higher relativity is chosen as the down-sampling direction for right-view images. The experimental results show that the proposed scheme can prejudge the sampling direction correctly.

StereoEditor: controllable stereoscopic display by content retargeting

We report a stereoscopic editing method, named as StereoEditor, to achieve controllable stereoscopic display. By taking content-aware, depth-adaptive and object-selective retargeting into account, the resolution, depth and object of stereoscopic input are simultaneously edited to create flexible stereoscopic output. The retargeting semantics from retinal disparity limits, viewing distance, and object’s scaling factors can be adjusted flexibly to achieve particular stereoscopic display, which can be controlled by user’s preference and practical viewing conditions. Compared with previous methods, our method can achieve a better tradeoff among shape preservation, depth control and object layout, leading to promising viewing experience.

Optimizing multiview video plus depth retargeting technique for stereoscopic 3D displays

In a multiview video plus depth (MVD) based three-dimensional (3D) video system, the generation of the contents with simultaneous resolution and depth adjustments is very challenging. In this paper, we have presented a Multiview Video plus Depth ReTargeting (MVDRT) technique for stereoscopic 3D (S3D) displays. The main motivation of this work is to optimize the resolution and depth of original MVD data so that it is suitable for view synthesis. Our method takes shape preservation, line bending and visual comfort constraints into account, and simultaneously optimizes the horizontal, vertical and depth coordinates in display space. The retargeted MVD data is used to generate the contents for S3D displays. Experimental results demonstrate our method can achieve a better view synthesis performance than other approaches that still preserve the original depth information after retargeting, leading to promising S3D experience.

An Energy-Constrained Video Retargeting Approach for Color-Plus-Depth 3D Video

Color-plus-depth 3D video is booming up with the growing popularity of depth-sensing camera. This imposes new challenges on video resizing since the information embedded in the depth and color video should be taken into consideration simultaneously. In this paper, we propose an energy-constrained video retargeting approach for color-plus-depth 3D video. We first build a 3D video visual attention (3DV-VA) computation model by combining gradient, saliency, motion and depth information. Then, an energy function is generated using spatial, temporal and 3DV-VA constraints. Finally, seam selection and carving operation is conducted to generate a retargeted 3D video. Experimental results on two types of 3D video sequences shows that the proposed approach produces better seam carving results than the related existing methods.

Decoding Path Selection for Multi-View Video System

To improve random access performance in multi-view video system, predication structure for multi-view video coding is represented with weighted graph. Then the optimal decoding path with respect to random access of user is computed by referring to the idea of the shortest path in graph theory. Experimental results show that the proposed scheme can obtain the shortest decoding path, and can effectively reduce the response time of random access and bandwidth in multi-view video system.