James Bankoski

An overview of new video coding tools under consideration for VP10: the successor to VP9

Google started an opensource project, entitled the WebM Project, in 2010 to develop royaltyfree video codecs for the web. The present generation codec developed in the WebM project called VP9 was finalized in mid2013 and is currently being served extensively by YouTube, resulting in billions of views per day. Even though adoption of VP9 outside Google is still in its infancy, the WebM project has already embarked on an ambitious project to develop a next edition codec VP10 that achieves at least a generational bitrate reduction over the current generation codec VP9. Although the project is still in early stages, a set of new experimental coding tools have already been added to baseline VP9 to achieve modest coding gains over a large enough test set. This paper provides a technical overview of these coding tools.

A dynamic motion vector referencing scheme for video coding

Video codecs exploit temporal redundancy in video signals, through the use of motion compensated prediction, to achieve superior compression performance. The coding of motion vectors takes a large portion of the total rate cost. Prior research utilizes the spatial and temporal correlation of the motion field to improve the coding efficiency of the motion information. It typically constructs a candidate pool composed of a fixed number of reference motion vectors and allows the codec to select and reuse the one that best approximates the motion of the current block. This largely disconnects the entropy coding process from the blocks motion information, and throws out any information related to motion consistency, leading to sub-optimal coding performance. An alternative motion vector referencing scheme is proposed in this work to fully accommodate the dynamic nature of the motion field. It adaptively extends or shortens the candidate list according to the actual number of available reference motion vectors. The associated probability model accounts for the likelihood that an individual motion vector candidate is used. A complementary motion vector candidate ranking system is also presented here. It is experimentally shown that the proposed scheme achieves about 1.6% compression performance gains on a wide range of test clips.

Deep Neural Network Based Frame Reconstruction for Optimized Video Coding

Video coding has served as a key enabling technology to the explosion in online video sharing and consumption. This includes live video streaming, online video sharing, video conferencing, video surveillance, remote medicine, online education, online gaming, video broadcasting, cloud video services, and many others. The recently released open source royalty-free video coding standard known as AV1, designed and developed by the Alliance of Open Media (AOM), achieves a 30%–40% data rate reduction from previous generational video coding standards, which includes VP9 and HEVC. This paper aims to outline paradigms that may provide further coding performance gains over AV1. Image restoration has demonstrated significant effectiveness in video coding performance enhancement in AV1. This paper describes techniques in the same vein effectively optimizing frame reconstruction through the use of the Deep Neural Networks (DNN) to further improve coding performance. Initial explorations of our proposed approach have demonstrated promising results.

A staircase transform coding scheme for screen content video coding

Demand for screen content videos that contain computer generated text and graphics is growing. They are very different from natural videos, because they include much sharper edge transitions and very repetitive patterns. On this type of material, the efficacy of the conventional discrete cosine transform (DCT) is questionable because it relies on the assumption that a Gauss-Markov model leads to a base-band signal. However, the assumption may not hold true for screen content material. This work exploits a class of staircase transforms. Unlike the DCT whose bases are samplings of sinusoidal functions, the staircase transforms have their bases sampled from staircase functions, which better approximate the sharp transitions often encountered in the context of screen content. The staircase transform is integrated into a hybrid transform coding scheme, in conjunction with DCT. It is experimentally shown that the proposed approach provides an average of 2.9% compression performance gains in terms of BD-rate reduction. A perceptual comparison further demonstrates that the use of staircase transform achieves substantial reduction in ringing artifact due to the Gibbs phenomenon.