Cagdas Bilen

End-to-end stereoscopic video streaming with content-adaptive rate and format control

We address efficient compression and real-time streaming of stereoscopic video over the current Internet. We first propose content-adaptive stereo video coding (CA-SC), where additional coding gain, over that can be achieved by exploiting only inter-view correlations, is targeted by down-sampling one of the views spatially or temporally depending on the content, based on the well-known theory that the human visual system can perceive high frequencies in three-dimensional (3D) from the higher quality view. We also developed stereoscopic 3D video streaming server and clients by modifying available open source platforms, where each client can view the video in mono or stereo mode depending on its display capabilities. The performance of the end-to-end stereoscopic streaming system is demonstrated using subjective quality tests.

A Multi-View Video Codec Based on H.264

H.264 is the current state-of-the-art monoscopic video codec providing almost twice the coding efficiency with the same quality comparing the previous codecs. With the increasing interest in 3D TV, multi-view video sequences that are provided by multiple cameras capturing the three dimensional objects and/or scene are more widely used. Compressing multi-view sequences independently with H.264 (simulcast) is not efficient since the redundancy between the closer cameras is not exploited. In order to reduce these redundancies, we propose a multi-view video codec based on H.264 using disparity estimation/compensation as well as motion estimation/compensation. In order to effectively search for disparity/motion without increasing computational complexity, we modified the buffering structure of H.264 and implemented several referencing modes. Our results show that for closely located cameras, our codec outperforms simulcast H.264 coding. For sparsely located cameras, our method can still improve coding gain depending on the video characteristics.

A standards-based, flexible, end-to-end multi-view video streaming architecture

In this paper we propose a novel framework for the streaming of 3-D representations in the form of Multi- View Videos (MVV). The proposed streaming system is completely standards based, flexible and backwards compatible in order to support monoscopic streaming to legacy clients. We demonstrate compatibility of the proposed system with various possible encoding schemes and operating scenarios. In the current implementation, the MVVs in the server are compressed using a simplified form of MVC with negligible loss of compression efficiency and streamed using Real Time Streaming Protocol (RTSP), Session Description Protocol (SDP) and Real Time Protocol (RTP) to the clients. We describe our extensions to SDP and discuss a preliminary RTP payload format for MVC. The clients in this implementation perform basic error concealment to reduce the effects of packet losses and decode MVC in near-real-time. The modular clients can display decoded 3-D content on a multitude of 3-D display systems.

Error Resilient Layered Stereoscopic Video Streaming

In this paper, error resilient stereoscopic video streaming problem is addressed. Two different forward error correction (FEC) codes namely systematic LT and RS codes are utilized to protect the stereoscopic video data against transmission errors. Initially, the stereoscopic video is categorized in 3 layers with different priorities. Then, a packetization scheme is used to increase the efficiency of error protection. A comparative analysis of RS and LT codes are provided via simulations to observe the optimum packetization and UEP strategies.

Schemes for multiple description coding of stereoscopic video

This paper presents and compares two multiple description schemes for coding of stereoscopic video, which are based on H.264. The SS-MDC scheme exploits spatial scaling of one view. In case of one channel failure, SS-MDC can reconstruct the stereoscopic video with one view low-pass filtered. SS-MDC can achieve low redundancy (less than 10%) for video sequences with lower inter-view correlation. MS-MDC method is based on multi-state coding and is beneficial for video sequences with higher inter-view correlation. The encoder can switch between these two methods depending on the characteristics of video.

Rate-distortion optimized layered stereoscopic video streaming with raptor codes

A near optimal streaming system for stereoscopic video is proposed. Initially, the stereoscopic video is separated into three layers and the approximate analytical model of the Rate-Distortion (RD) curve of each layer is calculated from sufficient number of rate and distortion samples. The analytical modeling includes the interdependency of the defined layers. Then, the analytical models are used to derive the optimal source encoding rates for a given channel bandwidth. The distortion in the quality of the stereoscopic video that is caused by losing a NAL unit from the defined layers is estimated to minimize the average distortion of a single NAL unit loss. The minimization is performed over protection rates allocated to each layer. Raptor codes are utilized as the error protection scheme due to their novelty and suitability in video transmission. The layers are protected unequally using Raptor codes according to the parity ratios allocated to the layers. Comparison of the defined scheme with two other protection allocation schemes is provided via simulations to observe the quality of stereoscopic video.

End-To-End Stereoscopic Video Streaming System

Today, stereoscopic and multi-view video are among the popular research areas in the multimedia world. In this study, we have designed a platform consisting of stereo-view capturing, real time transmission and display. At the display stage, end users view video in 3D by using polarized glasses. Stereoscopic video is compressed in an efficient way by using stereoscopic video coding techniques and streamed using real time protocols on the sender side. Receiver can view the content of the video built from multiple channels as mono or stereo depending on its display and bandwidth capabilities. The entire system is built by modifying available open source systems whenever possible

Motion and Disparity Aided Stereoscopic Full Frame Loss Concealment Method

Stereoscopic video is one of the emerging research areas especially among the video coding community. Along with the studies for efficiently compressing the stereoscopic and multiview video, new error concealment and error protection methods are also necessary to overcome the problems due to erroneous channel conditions in practical applications. In this paper we propose a full frame loss concealment algorithm for stereoscopic sequences. The proposed method uses redundancy and disparity between the two views and motion information between the previously decoded frames to estimate the lost frame. The results show that, the proposed algorithm outperforms the monoscopic methods when they are applied to the same view as they are simulcast coded.

End-to-End Stereoscopic Video Streaming System

Today, stereoscopic and multi-view video are among the popular research areas in the multimedia world. In this study, we have designed and built a platform consisting of stereo-view capturing, real-time transmission and display. At the display stage, end users view video in 3D by using polarized glasses. Multi-view video is compressed in an efficient way by using multi-view video coding techniques and streamed using standard real-time transport protocols. The entire system is built by modifying available open source systems whenever possible. Receiver can view the content of the video built from multiple channels as mono or stereo depending on its display and bandwidth capabilities

Unequal Error Protection for Stereoscopic Video Streaming

The utilization of forward error correction (FEC) schemes for stereo video streaming is investigated. Stereo video is categorized in 3 layers and each layer is protected with different protection ratios for efficient streaming. Systematic Reed-Solomon (RS) and Luby Transform (LT) codes are utilized as the error protection schemes. Detailed simulations are performed in order to observe the optimum unequal error protection (UEP) strategies for the defined video layers. Moreover, as a result of these simulations the performance comparison of RS and LT codes for video streaming is provided.