S. Worrall

Quality Evaluation of Color Plus Depth Map-Based Stereoscopic Video

In the near future, many conventional video applications are likely to be replaced by immersive video to provide a sense of ldquobeing there.rdquo This transition is facilitated by the recent advancement of 3D capture, coding, transmission, and display technologies. Stereoscopic video is the simplest form of 3D video available in the literature. ldquoColor plus depth maprdquo based stereoscopic video has attracted significant attention, as it can reduce storage and bandwidth requirements for the transmission of stereoscopic content over communication channels. However, quality assessment of coded video sequences can currently only be performed reliably using expensive and inconvenient subjective tests. To enable researchers to optimize 3D video systems in a timely fashion, it is essential that reliable objective measures are found. This paper investigates the correlation between subjective and objective evaluation of color plus depth video. The investigation is conducted for different compression ratios, and different video sequences. Transmission over Internet protocol (IP) is also investigated. Subjective tests are performed to determine the image quality and depth perception of a range of differently coded video sequences, with packet loss rates ranging from 0% to 20%. The subjective results are used to determine more accurate objective quality assessment metrics for 3D color plus depth video.

Frame concealment algorithm for stereoscopic video using motion vector sharing

Stereoscopic video is one of the simplest forms of multi view video, which can be easily adapted for communication applications. Much current research is based on colour and depth map stereoscopic video, due to its reduced bandwidth requirements and backward compatibility. Existing immersive media research is more focused on application processing than aspects related to transfer of immersive content over communication channels. As video over packet networks is affected by missing frames, caused by packet loss, this paper proposes a frame concealment method for colour and depth map based stereoscopic video. The proposed method exploits the motion correlation of colour and depth map image sequences. The colour motion information is reused for prediction during depth map coding. The redundant motion information is then used to conceal transmission errors at the decoder. The experimental results show that the proposed frame concealment scheme performs better than applying error concealment for colour and depth map video separately in a range of packet error conditions.

Display Dependent Preprocessing of Depth Maps Based on Just Noticeable Depth Difference Modeling

This paper addresses the sensitivity of human vision to spatial depth variations in a 3-D video scene, seen on a stereoscopic display, based on an experimental derivation of a just noticeable depth difference (JNDD) model. The main target is to exploit the depth perception sensitivity of humans in suppressing the unnecessary spatial depth details, hence reducing the transmission overhead allocated to depth maps. Based on the JNDD model derived, depth map sequences are preprocessed to suppress the depth details that are not perceivable by the viewers and to minimize the rendering artefacts that arise due to optical noise, where the optical noise is triggered by the inaccuracies in the depth estimation process. Theoretical and experimental evidences are provided to illustrate that the proposed depth adaptive preprocessing filter does not alter the 3-D visual quality or the view synthesis quality for free-viewpoint video applications. Experimental results suggest that the bit rate for depth map coding can be reduced up to 78% for the depth maps captured with depth-range cameras and up to 24% for the depth maps estimated with computer vision algorithms, without affecting the 3-D visual quality or the arbitrary view synthesis quality.

Content Adaptive Enhancement of Multi-View Depth Maps for Free Viewpoint Video

Depth map estimation is an important part of the multi-view video coding and virtual view synthesis within the free viewpoint video applications. However, computing an accurate depth map is a computationally complex process, which makes real-time implementation challenging. Alternatively, a simple estimation, though quick and promising for real-time processing, might result in inconsistent multi-view depth map sequences. To exploit this simplicity and to improve the quality of depth map estimation, we propose a novel content adaptive enhancement technique applied to the previously estimated multi-view depth map sequences. The enhancement method is locally adapted to edges, motion and depth-range of the scene to avoid blurring the synthesized views and to reduce the computational complexity. At the same time, and very importantly, the method enforces consistency across the spatial, temporal and inter-view dimensions of the depth maps so that both the coding efficiency and the quality of the synthesized views are improved. We demonstrate these improvements in the experiments, where the enhancement method is applied to several multi-view test sequences and the obtained synthesized views are compared to the views synthesized using other methods in terms of both numerical and perceived visual quality.

Sensitivity Analysis of the Human Visual System for Depth Cues in Stereoscopic 3-D Displays

Three-dimensional (3-D) displays provide a more realistic experience of entertainment by providing its viewers an added sensation of depth by artificially exploiting light rays to stimulate certain depth cues in the human visual system, especially binocular stereopsis. Due to its close relationship with human visual perception, mass market deployment of 3-D displays will be significantly dependant upon addressing the related perceptual factors such as visual comfort. In order to address the perceptual factors, it is very important to understand how humans experience depth on 3-D displays and how sensitive they are for different depth cues. In this paper, the sensitivity of humans for different depth cues is analyzed as applicable to 3-D viewing on stereoscopic displays. Mathematical models are derived to explain the just noticeable difference in depth (JNDD) for three different depth cues, namely binocular disparity, retinal blur, and relative size. Extensive subjective assessments are performed on a stereoscopic display with passive polarized glasses and on an auto-stereoscopic display to validate the mathematical models for JNDD. It is expected that the proposed models will have important use cases in 3-D display designing as well as 3-D content production.

Prioritisation of data partitioned MPEG-4 video over mobile networks

Despite much research in the field of mobile multimedia, delivery of real-time interactive video over noisy wireless channels remains a challenging problem. Two of the major issues in providing true end-to-end mobile multimedia capability are interoperability between platforms and networks and the poor performance of video compression algorithms in error-prone environments. This paper presents a method for prioritising data partitioned MPEG-4 video in a way suitable for transmission over a mobile network. The effectiveness of the technique is demonstrated by examining the performance when transmitted using the Real-Time Transport protocol over GPRS (General Packet Radio Service) data channels under varying channel conditions.

Bit-Rate Adaptive Downsampling for the Coding of Multi-View Video with Depth Information

In this paper, the potential for improving the compression efficiency of multi-view video coding with depth information is explored. The proposed technique uses downsampling prior to encoding, for arbitrary views and depth maps. A bit-rate adaptive downscaling-ratio decision approach is proposed for certain views and depth maps prior to encoding. Colour and depth videos are considered separately due to their different characteristics and effects on synthesized free view-point videos. The inter-view references, if present, are downsampled to the same resolution as the input video to be coded. The results for several multi-view with depth sequences indicate that using bit-rate adaptive mixed spatial resolution coding for both views and depth maps can achieve savings in bit-rate, compared to full resolution and fixed depth-to-colour ratio multi-view coding when the quality of synthesized viewpoints are considered. The computational complexity in the encoder is significantly reduced at the same time, since the number of blocks coded is reduced, and hence the number of block mode decisions carried out is reduced.

Utilisation of edge adaptive upsampling in compression of depth map videos for enhanced free-viewpoint rendering

In this paper we propose a novel video object edge adaptive upsampling scheme for application in video-plus-depth and Multi-View plus Depth (MVD) video coding chains with reduced resolution. Proposed scheme is for improving the rate-distortion performance of reduced-resolution depth map coders taking into account the rendering distortion induced in free-viewpoint videos. The inherent loss in fine details due to downsampling, particularly at video object boundaries causes significant visual artefacts in rendered free-viewpoint images. The proposed edge adaptive upsampling filter allows the conservation and better reconstruction of such critical object boundaries. Furthermore, the proposed scheme does not require the edge information to be communicated to the decoder, as the edge information used in the adaptive upsampling is derived from the reconstructed colour video. Test results show that as much as 1.2 dB gain in free-viewpoint video quality can be achieved with the utilization of the proposed method compared to the scheme that uses the linear MPEG re-sampling filter. The proposed approach is suitable for video-plus-depth as well as MVD applications, in which it is critical to satisfy bandwidth constraints while maintaining high free-viewpoint image quality.

Scalable multiple description video coding for stereoscopic 3D

Scalable multiple description video coding (MDC) provides adaptability to bandwidth variations and receiving device characteristics, and can improve error robustness in multimedia networks. In this paper, a scalable MDC scheme is proposed for stereoscopic 3D video. Scalable MDC has previously been applied to 2D video but not to 3D video. The proposed algorithm enhances the error resilience of the base layer of the scalable video coding (SVC) standard using even and odd frame based MDC. The performance of the algorithm is evaluated in error free and error prone environments. Simulation results show improved performance using the proposed scalable MDC at high error rates compared to the original SVC. To improve the rate distortion performance, down-sampling of the depth information is proposed for SVC and scalable MDC. The proposed method reduces the overall bit rates and consequently: (1) improves their rate distortion, particularly at lower bit rates in error free channels; and (2) improves their performance in error prone channels.

3D video assessment with Just Noticeable Difference in Depth evaluation

The ability to provide a realistic perception of depth is the core added functionality of modern 3D video display systems. At present, there is no standard method to assess the perception of depth in 3D video. Existence of such methods would immensely enhance the progression of 3D video research. This paper focuses on the depth perception assessment in color plus depth representation of 3D video. In this paper, we subjectively evaluate the depth perceived by the users on an auto stereoscopic display, and analyze its variation with the impairments introduced during the compression of the depth images. The variation of the subjective perception of depth is explained based on another evaluation that is carried out to identify the Just Noticeable Difference in Depth (JNDD) perceived by the subjects. The JNDD corresponds to the sensitivity of the observers to the changes in depth in a 3D video scene. Even though only the effects of compression artifacts are considered in this paper, the proposed assessment technique, based on the JNDD values can be used in any future depth perception assessment work.