Ahmet M. Kondoz

Quality Evaluation of Color Plus Depth Map-Based Stereoscopic Video

In the near future, many conventional video applications are likely to be replaced by immersive video to provide a sense of ldquobeing there.rdquo This transition is facilitated by the recent advancement of 3D capture, coding, transmission, and display technologies. Stereoscopic video is the simplest form of 3D video available in the literature. ldquoColor plus depth maprdquo based stereoscopic video has attracted significant attention, as it can reduce storage and bandwidth requirements for the transmission of stereoscopic content over communication channels. However, quality assessment of coded video sequences can currently only be performed reliably using expensive and inconvenient subjective tests. To enable researchers to optimize 3D video systems in a timely fashion, it is essential that reliable objective measures are found. This paper investigates the correlation between subjective and objective evaluation of color plus depth video. The investigation is conducted for different compression ratios, and different video sequences. Transmission over Internet protocol (IP) is also investigated. Subjective tests are performed to determine the image quality and depth perception of a range of differently coded video sequences, with packet loss rates ranging from 0% to 20%. The subjective results are used to determine more accurate objective quality assessment metrics for 3D color plus depth video.

Quality analysis for 3D video using 2D video quality models

The 3D (3-dimensional) video technologies are emerging to provide more immersive media content compared to conventional 2D (2-dimensional) video applications. More often 3D video quality is measured using rigorous and time-consuming subjective evaluation test campaigns. This is due to the fact that 3D video quality can be described as a combination of several perceptual attributes such as overall image quality, perceived depth, presence, naturalness and eye strain, etc. Hence this paper investigates the relationship between subjective quality measures and several objective quality measures like PSNR, SSIM, and VQM for 3D video content. The 3D video content captured using both stereo camera pair (two cameras for left and right views) and colour-and-depth special range cameras are considered in this study. The results show that, VQM quality measures of individual left and right views (rendered left and right views for colour-and-depth sequences) can be effectively used in predicting the overall image quality and statistical measures like PSNR and SSIM of left and right views illustrate good correlations with depth perception of 3D video.

Analysis and improvement of a statistical model-based voice activity detector

From an investigation of a statistical model-based voice activity detector (VAD), it is found that the likelihood ratio defined in the VAD has a fundamental problem at the offset regions of the speech signals. Thus, we analyze the behavioural mechanism of the likelihood ratio, identify the reason for the unwanted phenomenon, and propose a solution based on a smoothed likelihood ratio. Objective test results show that the proposed method gives a significant improvement to the original VAD. Additionally, the improved VAD results in a performance that is even superior to G.729B and comparable to AMR VAD option 2.

Error-resilient video transcoding for robust internetwork communications using GPRS

A novel fully comprehensive mobile video communications system is proposed. The system exploits the useful rate management features of video transcoders and combines them with error resilience for the transmission of coded video streams over general packet radio service (GPRS) mobile-access networks. The error-resilient video transcoding operation takes place at a centralized point, referred to as a video proxy, which provides the necessary output transmission rates with the required amount of robustness. With the use of this proposed algorithm, error resilience can be added to an already compressed video stream at an intermediate stage at the edge of two or more different networks through two resilience schemes, namely the adaptive intra refresh (AIR) and feedback control signaling (FCS) methods. Both resilience tools impose an output rate increase which can also be prevented with the proposed novel technique. Thus, the presented scheme gives robust video outputs at near target transmission rates that only require the same number of GPRS timeslots as non-resilient schemes. Moreover, an ultimate robustness is also accomplished with the combination of the two resilience algorithms at the video proxy. Extensive computer simulations demonstrate the effectiveness of the proposed system.

Improved voice activity detection based on a smoothed statistical likelihood ratio

This paper presents the behavioural mechanism of a statistical model-based voice activity detector (VAD), featuring a likelihood ratio test for the activity decision. From investigation of the VAD, it is found that detection errors could occur frequently at speech offset regions because of the delay term in the decision-directed parameter estimator, employed for the estimation of an unknown parameter of the likelihood ratio. Hence, this paper proposes a smoothed likelihood ratio so as to alleviate the detection errors at the offset region. Objective test results show that the proposed scheme is useful for achieving a considerable performance improvement for the VAD. Additionally, the proposed VAD gives detection performances superior to G.729B VAD and comparable with the AMR VAD option 2.

Toward an Impairment Metric for Stereoscopic Video: A Full-Reference Video Quality Metric to Assess Compressed Stereoscopic Video

The quality assessment of impaired stereoscopic video is a key element in designing and deploying advanced immersive media distribution platforms. A widely accepted quality metric to measure impairments of stereoscopic video is, however, still to be developed. As a step toward finding a solution to this problem, this paper proposes a full reference stereoscopic video quality metric to measure the perceptual quality of compressed stereoscopic video. A comprehensive set of subjective experiments is performed with 14 different stereoscopic video sequences, which are encoded using both the H.264 and high efficiency video coding compliant video codecs, to develop a subjective test results database of 116 test stimuli. The subjective results are analyzed using statistical techniques to uncover different patterns of subjective scoring for symmetrically and asymmetrically encoded stereoscopic video. The subjective result database is subsequently used for training and validating a simple but effective stereoscopic video quality metric considering heuristics of binocular vision. The proposed metric performs significantly better than state-of-the-art stereoscopic image and video quality metrics in predicting the subjective scores. The proposed metric and the subjective result database will be made publicly available, and it is expected that the proposed metric and the subjective assessments will have important uses in advanced 3D media delivery systems.

Frame concealment algorithm for stereoscopic video using motion vector sharing

Stereoscopic video is one of the simplest forms of multi view video, which can be easily adapted for communication applications. Much current research is based on colour and depth map stereoscopic video, due to its reduced bandwidth requirements and backward compatibility. Existing immersive media research is more focused on application processing than aspects related to transfer of immersive content over communication channels. As video over packet networks is affected by missing frames, caused by packet loss, this paper proposes a frame concealment method for colour and depth map based stereoscopic video. The proposed method exploits the motion correlation of colour and depth map image sequences. The colour motion information is reused for prediction during depth map coding. The redundant motion information is then used to conceal transmission errors at the decoder. The experimental results show that the proposed frame concealment scheme performs better than applying error concealment for colour and depth map video separately in a range of packet error conditions.

Perceptual Video Quality Metric for 3D video quality assessment

One method of evaluating the quality of stereoscopic video is the use of conventional two dimensional (2D) objective metrics. Metrics with good representation of the Human Visual System (HVS) will present more accurate evaluation. In this paper we propose a perceptual based objective metric for 2D videos for 3D video quality evaluation. The proposed Perceptual Quality Metric (PQM) shows better results for 3D video quality evaluation and outperforms the Video Quality Metric (VQM); as it is sensitive to slight changes in image degradation and error quantification starts at pixel level right up to the sequence level. Verifications are done through series of subjective tests to show the level of correlation of PQM and user scores.

3D motion estimation for depth image coding in 3D video coding

In this paper, a new solution is introduced for the efficient compression of 3D video based on color and depth maps. While standard video codecs are designed for coding monoscopic videos, their application to depth maps is found to be suboptimal. With regard to the special properties of the depth maps, we propose an extension to conventional video coding in order to take advantage of object motion in the depth direction. Instead of performing a 2D motion search, as is common in conventional video codecs, we propose the use of a 3D motion search that is able to better exploit the temporal correlations of 3D content. In this new framework, the motion of blocks in depth maps are described using 3D motion vectors (x, y, z), representing the horizontal, vertical, and depth directions respectively. This leads to more accurate motion prediction and a smaller residual. The experimental results show that the proposed technique delivers an improvement in motion compensation, which leads to gains in compression efficiency.

Sensitivity Analysis of the Human Visual System for Depth Cues in Stereoscopic 3-D Displays

Three-dimensional (3-D) displays provide a more realistic experience of entertainment by providing its viewers an added sensation of depth by artificially exploiting light rays to stimulate certain depth cues in the human visual system, especially binocular stereopsis. Due to its close relationship with human visual perception, mass market deployment of 3-D displays will be significantly dependant upon addressing the related perceptual factors such as visual comfort. In order to address the perceptual factors, it is very important to understand how humans experience depth on 3-D displays and how sensitive they are for different depth cues. In this paper, the sensitivity of humans for different depth cues is analyzed as applicable to 3-D viewing on stereoscopic displays. Mathematical models are derived to explain the just noticeable difference in depth (JNDD) for three different depth cues, namely binocular disparity, retinal blur, and relative size. Extensive subjective assessments are performed on a stereoscopic display with passive polarized glasses and on an auto-stereoscopic display to validate the mathematical models for JNDD. It is expected that the proposed models will have important use cases in 3-D display designing as well as 3-D content production.