D.V.S.X. De Silva

Display Dependent Preprocessing of Depth Maps Based on Just Noticeable Depth Difference Modeling

This paper addresses the sensitivity of human vision to spatial depth variations in a 3-D video scene, seen on a stereoscopic display, based on an experimental derivation of a just noticeable depth difference (JNDD) model. The main target is to exploit the depth perception sensitivity of humans in suppressing the unnecessary spatial depth details, hence reducing the transmission overhead allocated to depth maps. Based on the JNDD model derived, depth map sequences are preprocessed to suppress the depth details that are not perceivable by the viewers and to minimize the rendering artefacts that arise due to optical noise, where the optical noise is triggered by the inaccuracies in the depth estimation process. Theoretical and experimental evidences are provided to illustrate that the proposed depth adaptive preprocessing filter does not alter the 3-D visual quality or the view synthesis quality for free-viewpoint video applications. Experimental results suggest that the bit rate for depth map coding can be reduced up to 78% for the depth maps captured with depth-range cameras and up to 24% for the depth maps estimated with computer vision algorithms, without affecting the 3-D visual quality or the arbitrary view synthesis quality.

Object based coding of the depth maps for 3D video coding

The emergence of three dimensional (3D) video applications, based on depth image based rendering (DIBR) has brought up more requirements of bandwidth, due to the need of depth information. This additional bandwidth requirement need to be tackled to enable the widespread of 3D video applications based on DIBR. Exploiting visual correlations between the color image and the depth image, in depth image coding, will reduce the requirement of high bandwidth required to transmit the additional depth information. In this paper, an object based depth image coding technique is presented which is suitable for low bit rate 3D-TV applications that are based on depth image based rendering. The proposed method achieves at most 50% bit rate reduction at low bit rates.

Intra mode selection for depth map coding to minimize rendering distortions in 3D video

Depth image based rendering (DIBR) is a technique that generates virtual views for multiview video applications from 3D video represented in color plus depth format. The depth map is not viewed by end users. However, it helps to generate different views as required by the application. Therefore, the depth maps need to be compressed in a way that it minimizes distortions in the rendered views. By doing so, it would be possible to generate high quality virtual views, using compressed depth maps. This paper presents two mode selection techniques based on genetic algorithms for encoding depth maps. In the proposed techniques, the encoding modes are decided so that the distortion in the rendered views is minimized. Simulation results illustrate that proposed techniques improve the objective quality of the rendered virtual views by up to 2 dB over the Lagrange Optimization based mode selection technique that considers the distortions only in the depth map.

3D video assessment with Just Noticeable Difference in Depth evaluation

The ability to provide a realistic perception of depth is the core added functionality of modern 3D video display systems. At present, there is no standard method to assess the perception of depth in 3D video. Existence of such methods would immensely enhance the progression of 3D video research. This paper focuses on the depth perception assessment in color plus depth representation of 3D video. In this paper, we subjectively evaluate the depth perceived by the users on an auto stereoscopic display, and analyze its variation with the impairments introduced during the compression of the depth images. The variation of the subjective perception of depth is explained based on another evaluation that is carried out to identify the Just Noticeable Difference in Depth (JNDD) perceived by the subjects. The JNDD corresponds to the sensitivity of the observers to the changes in depth in a 3D video scene. Even though only the effects of compression artifacts are considered in this paper, the proposed assessment technique, based on the JNDD values can be used in any future depth perception assessment work.

A new mode selection technique for coding Depth maps of 3D video

Compression of Depth maps that are used in 3D video systems based on Depth Image Based Rendering (DIBR) poses a new challenge in video coding, since it is not a sequence of images for final viewing by end users rather an aid for rendering. Therefore, compressing depth maps using existing video coding techniques yields unacceptable distortions while rendering virtual views. In this paper we propose a novel mode selection method for offline compression of depth maps by selecting modes collaboratively considering an entire row of macroblocks together. For selecting these modes while encoding, we propose a novel distortion criteria that incorporates rendering distortions instead of distortion of depth map itself. A genetic algorithm based optimization technique is used for the mode selection. The simulation results suggest that the proposed technique can improve the PSNR up to 1.6dB in the rendered stereoscopic views in comparison to the block wise mode selection method based on Lagrange Optimization and the distortion of the depth map itself.

Just noticeable difference in depth model for stereoscopic 3D displays

The most distinguishing feature of 3D display systems, compared to their traditional 2D counterparts, is their ability to provide an additional perception of depth to its viewers. Thus, the mechanisms behind human depth perception play a significant role in 3D video systems. While there has been significant amounts of research carried out to understand human depth perception, in the areas of physiology and psychology, its applicability to 3D display systems is seldom spoken. Understanding the mechanisms of depth perception is of utmost importance to the development of 3D video technologies that are heavily based on exploitation of human perception. In this paper, it is explained with the aid of existing physiological and psychological models how humans perceive depth in 3D video displays. Based on these explanations a mathematical model is derived to explain the just noticeable difference in depth (JNDD) as perceived by a viewer, watching 3D video. The derived model is experimentally validated on an auto-stereoscopic display. This model is expected to be useful in both 3D content productions as well as in 3D content processing and compression.

A compound depth and image quality metric for measuring the effects of packet loss on 3D video

Interest in 3D video has surged in recent years. However, efforts to improve the quality of compression and transmission schemes are severely hampered by a lack of effective quality evaluation metrics. This is a particularly severe problem for researchers trying to improve the robustness of video transmission to packet loss. Subjective tests for evaluating error robustness present huge requirements in terms of time and resources. To solve this problem, this paper presents a quality metric for 3D video, and evaluates its effectiveness for the measurement of quality in the presence of packet loss. A key feature of the work is the use of depth planes to enable the metric to better model how the Human Visual System (HVS) perceives 3D video. The quality metric results are compared with subjective test results. The correlation between the proposed quality metric and the subjective test results is shown to be stronger than standard quality metrics, such as Video Quality Metric (VQM).

A novel depth map quality metric and its usage in depth map coding

While the depth maps of 3D video are represented as luminance images, they are used to aid rendering of novel views and are not viewed by an end user. Therefore, metrics that measure the quality of images that are for end user viewing does not necessarily reflect the quality of depth maps in terms of its ability to render views. This paper investigates the relationship between the quality of the rendered views and different quality measures of the depth map. A novel depth map quality metric is proposed based on a distortion model that approximates rendering errors due to pixel errors in the depth map. The proposed depth map quality metric correlates very well with the quality of the rendered views, as compared to the PSNR and SSIM of the depth map. The application of the proposed depth map quality metric is further illustrated by incorporating the metric at the encoding mode selection stage of a video encoder. Experimental results suggest that with the proposed encoding mode selection scheme bit rate savings of up to 30% can be achieved compared to traditional encoding mode selection scheme based on sum of squared errors.

Error resilience technique for multi-view coding using redundant disparity vectors

Research on error resilience in multi-view coding is currently receiving considerable interest. While there is a multitude of literature concerning error recovery in 2D video, due to the statistical difference in motion compensation among temporal frames and disparity compensation among view points, such methods are inadequate to cater to the requirements of multiview video transmission. This paper addresses the above issue by transmission of redundant disparity vectors for error recovery purposes. The proposed system, which is implemented using the Joint Scalable Video Model (JSVM) codec and tested using a simulated Internet Protocol (IP) packet network environment, can be used along with a suitable error concealment scheme to provide robust multi-view video transmission. The experimental results suggest that the proposed algorithm experiences a slight degradation of quality in error free environments due to the inclusion of redundant data. However, it improves the reconstructed picture quality significantly in error prone environments, specifically for Packet Loss Rates (PLRs) greater than 7%.

A study on the perceived quality of 3D video subject to packet losses

3D video quality evaluation received much attention from researchers due to the demand from the industry and to the complex nature of true 3D video perception. The availability of public 3D video databases provides an opportunity for researchers and developers to evaluate novel objective 3D video quality evaluation metrics. In this paper, a stereoscopic 3D video database for the evaluation of visual quality assessment metrics is described. The effects of random packet losses on the overall 3D perception (i.e., distortions due to different packet loss rates) are considered in this research. The database presented here contains 54 test stimuli from 9 reference test video sequences corresponding to 6 different packet loss rates, including the uncorrupted 3D video sequence. In order to obtain true 3D video perception, about 1730 individual human quality observations (Opinion Scores (OS) of subjects) are considered for this database. The obtained Differential Mean Opinion Scores (DMOS) can be effectively used for evaluating 3D video quality metrics, as well as for designing new 3D video quality evaluation methods. Together with DMOS values, we provide the corresponding objective quality measurements using several objective quality metrics. The designed 3D video database will be freely available for download and use in scientific research.