Shipeng Li

Distributed Multi-view Video Coding

There are mainly two key points which can affect the efficiency of multi-view video capture and transmission system largely: communication between cameras and computing complexity of encoder. In this paper, we propose a practical framework of distributed multi-view video coding, in which inter-camera communication is avoided and the large computing complexity is moved from encoder to decoder. In this scheme, multi-camera video sources are encoded separately and decoded dependently, and the traditional inter frame is replaced by Wyner-Ziv frame. To reach this goal, Wyner-Ziv theory on source coding with side information is employed as the basic coding principle. A Wyner-Ziv coding method based on wavelet transform and turbo codes is used as the core of the scheme. To further improve the coding performance, we also consider exploiting the large redundancy between adjacent views. A more flexible prediction method that can jointly use temporal and view correlations is proposed to generate the side information at the decoder. The experimental results show that the coding performance of proposed DMVC scheme is very promising compared to the traditional intra coding.

Exploiting temporal correlation with adaptive block-size motion alignment for 3D wavelet coding

This paper proposes an adaptive block-size motion alignment technique in 3D wavelet coding to further exploit temporal correlations across pictures. Similar to B picture in traditional video coding, each macroblock can motion align from forward and/or backward for temporal wavelet de-composition. In each direction, a macroblock may select its partition from one of seven modes - 16x16, 8x16, 16x8, 8x8, 8x4, 4x8 and 4x4 - to allow accurate motion alignment. Furthermore, the rate-distortion optimization criterions are proposed to select motion mode, motion vectors and partition mode. Although the proposed technique greatly improves the accuracy of motion alignment, it does not directly bring the coding efficiency gain because of smaller block size and more block boundaries. Therefore, an overlapped block motion alignment is further proposed to cope with block boundaries and to suppress spatial high-frequency components. The experimental results show the proposed adaptive block-size motion alignment with the overlapped block motion alignment can achieve up to 1.0 dB gain in 3D wavelet video coding. Our 3D wavelet coder outperforms the MC-EZBC for most sequences by 1~2dB and we are doing up to 1.5 dB better than H.264.

Seamless switching of scalable video bitstreams for efficient streaming

Proposes a seamless switching scheme for scalable video bitstreams that fully takes advantage of both the high coding efficiency of non-scalable bitstreams and the flexibility of scalable bitstreams. Small bandwidth fluctuations are accommodated by the scalability of the bitstreams, while large bandwidth fluctuations are tolerated by switching between scalable bitstreams. The major contribution of this paper is a flexible and effective scheme for seamless switching between scalable bitstreams that significantly improves the efficiency of scalable video coding over a broad bit rate range. When the channel bandwidth drops below the effective range of a scalable bitstream operated at higher rates, the proposed scheme can switch at any frame from the current scalable bitstream to one operated at lower rates without sending any overhead bits. Additional bits are only necessary when switching from a scalable bitstream operated at lower rates to one operated at higher rates. Experimental results show that the proposed scheme significantly outperforms both the approach with a single scalable bitstream and the approach of switching among multiple non-scalable bitstreams.

Motion vector prediction with reference frame consideration

In this paper, we introduce a new motion vector prediction method that could be used within multiple picture reference codecs, such as the H.264 (MPEG-4 AVC) video coding standard. Our method considers for each candidate motion vector the temporal distance of its corresponding reference picture compared to the current one for the generation of the predictor motion vector. This allows for more accurate motion vector prediction, and better exploitation of the temporal correlation that may exist within a video sequence. Furthermore, we also introduce a modification to the SKIP motion vector macroblock mode, according to which not only the motion vectors but also the reference indices are adaptively generated. Simulation results suggest that our proposed methods, combined with an improved Rate Distortion optimization strategy, if implemented within the existing H.264 codec, can allow for a considerable performance improvement of up to 8.6% bitrate reduction compared to the current H.264 standard.

Real-time video coding under power constraint based on H.264 codec

In this paper, we propose a joint power-distortion optimization scheme for real-time H.264 video encoding under the power constraint. Firstly, the power constraint is translated to the complexity constraint based on DVS technology. Secondly, a computation allocation model (CAM) with virtual buffers is proposed to facilitate the optimal allocation of constrained computational resource for each frame. Thirdly, the complexity adjustable encoder based on optimal motion estimation and mode decision is proposed to meet the allocated resource. The proposed scheme takes the advantage of some new features of H.264/AVC video coding tools such as early termination strategy in fast ME. Moreover, it can avoid suffering from the high overhead of the parametric power control algorithms and achieve fine complexity scalability in a wide range with stable rate-distortion performance. The proposed scheme also shows the potential of a further reduction of computation and power consumption in the decoding without any change on the existing decoders.

Macroblock-based progressive fine granularity scalable (PFGS) video coding with flexible temporal-SNR scalablilities

We proposed a flexible and efficient architecture for scalable video coding, namely, the macroblock (MB)-based progressive fine granularity scalable video coding with temporal-SNR scalabilities (PFGST). The proposed architecture can provide not only much improved coding efficiency but also simultaneous SNR scalability and temporal scalability. Building upon the original frame-based progressive fine granularity scalable (PFGS) coding approach, the MB-based PFGS scheme is first proposed. Three INTER modes and the corresponding mode selection mechanism are presented for coding the SNR enhancement MBs in order to make a good trade-off between low drifting errors and high compression efficiency. Furthermore, temporal scalability is introduced into the MB-based PFGS, which forms the MB-based PFGST scheme. Two coding modes are proposed for coding the temporal enhancement MBs. Since it would not cause any error propagation if using the high quality reference in the temporal enhancement MB coding, the coding efficiency of the PFGST is highly improved by always choosing the most suitable reference for the temporal scalable coding. Experimental results show that the MB-based PFGST video coding scheme can significantly improve the coding efficiency up to 2.8 dB compared with the FGST scheme adopted in MPEG-4, while supporting full SNR, full temporal, and hybrid SNR-temporal scalabilities according to the different requirements from the channels, the clients or the servers.

Direct macroblock coding for predictive (P) pictures in the H.264 standard

In this paper we introduce a new Inter Macroblock type within the H.264 (or MPEG-4 AVC) video coding standard that can further improve coding efficiency by exploiting the temporal correlation of motion within a sequence. This leads to a reduction in the bits required for encoding motion information, while retaining or even improving quality under a Rate Distortion Optimization Framework. An extension of this concept within the skip macroblock type of the same standard is also presented. Simulation results show that the proposed semantic changes can lead to up to 7.6% average bitrate reduction or equivalently 0.39dB quality improvement over the current H.264 standard.

Free viewpoint switching in multi-view video streaming using Wyner-Ziv video coding

The free viewpoint switching is one of the most important features of multi-view video streaming. The key problem lies in how to achieve the best performance when the camera processing capability and the network bandwidth are limited. In this paper, we propose a novel free viewpoint switching scheme for multi-view video scenario, in which the distributed video coding technique is employed. In this scheme, the multi-camera video sources are encoded separately with the traditional hybrid video coding scheme, and meanwhile an alternative bitstream is produced for every frame based on the Wyner-Ziv coding method for the purpose of error correction when the viewpoint switching occurs. When switching happens, the Wyner-Ziv bits corresponding to the actual reference frame at the switching point is transmitted and used to recover the true reference. Instead of completely removing the mismatch, the proposed switching scheme tries to reduce the mismatch to an acceptable level so as to save the bits for the switching frame. A wavelet transform domain Wyner-Ziv coding method is proposed to produce the Wyner-Ziv bits for the switching frame. Conclusively, with the proposed scheme, the inter-camera communication can be avoided and the drifting error can be controlled efficiently when the viewpoint switching occurs.

Studies on spatial scalable frameworks for motion aligned 3D wavelet video coding

This paper makes a comparative study on the various spatial scalable coding frameworks. The frameworks with multiple image-domain motion aligned temporal filtering at various spatial resolutions, named as multi-T+2D, are mainly investigated. First we investigate a multi-T+2D scheme based on redundant frame representation. The cross spatial layer redundancy and prediction methods are discussed. The redundancy brings significant performance loss for schemes providing wide range SNR spatial scalability. To remove the redundancy produced in the multi-resolution temporal filtering while retaining the advantage of spatial-domain motion compensation, a novel non-redundant multi-T+2D scheme is proposed. Performance comparison is given among the discussed frameworks and it shows that the proposed non-redundant multi-T+2D framework has a good performance for fully scalable video coding. We also verify that the redundant multi-T+2D framework with cross spatial layer reconstruction feedback is practical in providing narrow range SNR scalability for each spatial layer.

Transcoding to FGS Streams from H.264/AVC Hierarchical B-Pictures

This paper presents a transcoder which transcodes to FGS streams from H.264/AVC hierarchical B-pictures. First, the DCT-domain architecture is designed for fast FGS transcoding. Then, we propose a mode decision method in DCT domain to achieve a trade-off between the performances at low bit-rate and high bit-rate. Experimental results demonstrated that our method can improve the coding performance up to 1 dB at low rate and only lose at worst 0.5 dB at high rate.