Madhukar Budagavi

Improvements on Intra Block Copy in natural content video coding

The Intra Block Copy (IntraBC) is a newly adopted tool in the HEVC extension for the screen content video coding. The IntraBC tool efficiently encodes repeating patterns in a picture. The current IntraBC scheme achieves about 1.0% bit-rate reduction on average and up to 4.3% bitrate reduction on natural content video for a database consisting of 2K, 4K, and 8K sequences. In this paper, we propose to improve the IntraBC with a template matching block vector and a fractional search IntraBC. With these two tools, the gain on natural content video coding can be further improved by 0.5% on average and up to 2.0%.

Motion estimation and compensation for fisheye warped video

Video captured through fisheye lens is becoming very prevalent due to applications such as surveillance, automotive driver assistance systems, and recreational sports. Virtual reality applications such as 360 degrees video also use fish-eye lens video cameras to capture the 360 degree view. The main advantage of fisheye lenses is that they increases the field of view, however they also introduce warping distortion in the captured video. Due to the warping, the motion in the video is typically non-translational and traditional block motion compensation techniques are not fully effective for such video. This paper presents a warping motion compensation technique that models the fish-eye lens distortion to efficiently code warped video. Given the fisheye lens parameters (e.g., transmitted at a video sequence level), the warping model is implicitly derived in both the encoder and decoder and no block-level warping parameters are transmitted. The proposed approach was integrated into HEVC HM-14.0 and initial results on simulated fish-eye lens distorted video show promise especially when the global motion in the video is fast.

Improving Intra Prediction in High-Efficiency Video Coding

Intra prediction is an important tool in intra-frame video coding to reduce the spatial redundancy. In current coding standard H.265/high-efficiency video coding (HEVC), a copying-based method based on the boundary (or interpolated boundary) reference pixels is used to predict each pixel in the coding block to remove the spatial redundancy. We find that the conventional copying-based method can be further improved in two cases: 1) the boundary has an inhomogeneous region and 2) the predicted pixel is far away from the boundary that the correlation between the predicted pixel and the reference pixels is relatively weak. This paper performs a theoretical analysis of the optimal weights based on a first-order Gaussian Markov model and the effects when the pixel values deviate from the model and the predicted pixel is far away from the reference pixels. It also proposes a novel intra prediction scheme based on the analysis that smoothing the copying-based prediction can derive a better prediction block. Both the theoretical analysis and the experimental results show the effectiveness of the proposed intra prediction method. An average gain of 2.3% on all intra coding can be achieved with the HEVC reference software.

Coding efficiency improvements beyond HEVC with known tools

In this paper, several coding tools are evaluated on top of the HEVC version 1. Among them there are straightforward extension of HEVC coding tools (such as Coding Unit size enlarging, fine granularity of Intra prediction angles) and algorithms that have been studied during HEVC development (such as secondary transform, multi-hypothesis CABAC, multi-parameter Intra prediction, bidirectional optical flow). Most of them improve performance of Intra coding. Minor adjustment to the final version of HEVC standard was done for efficient harmonization of the proposed coding tools with HEVC. Performance improvement observed from investigated tools is up to 7,1%, 9,9%, 4,5% and 5,7% in all-intra, random access, low-delay B and low-delay P test scenario (using HEVC common test conditions).

Dual-fisheye lens stitching for 360-degree imaging

Dual-fisheye lens cameras have been increasingly used for 360-degree immersive imaging. However, the limited overlapping field of views and misalignment between the two lenses give rise to visible discontinuities in the stitching boundaries. This paper introduces a novel method for dual-fisheye camera stitching that adaptively minimizes the discontinuities in the overlapping regions to generate full spherical 360-degree images. Results show that this approach can produce good quality stitched images for Samsung Gear 360 - a dual-fisheye camera, even with hard-to-stitch objects in the stitching borders.

Predictive View Generation to Enable Mobile 360-degree and VR Experiences

As 360-degree videos and virtual reality (VR) applications become popular for consumer and enterprise use cases, the desire to enable truly mobile experiences also increases. Delivering 360-degree videos and cloud/edge-based VR applications require ultra-high bandwidth and ultra-low latency [22], challenging to achieve with mobile networks. A common approach to reduce bandwidth is streaming only the field of view (FOV). However, extracting and transmitting the FOV in response to user head motion can add high latency, adversely affecting user experience. In this paper, we propose a predictive view generation approach, where only the predicted view is extracted (for 360-degree video) or rendered (in case of VR) and transmitted in advance, leading to a simultaneous reduction in bandwidth and latency. The view generation method is based on a deep-learning-based viewpoint prediction model we develop, which uses past head motions to predict where a user will be looking in the 360-degree view. Using a very large dataset consisting of head motion traces from over 36,000 viewers for nineteen 360-degree/VR videos, we validate the ability of our viewpoint prediction model and predictive view generation method to offer very high accuracy while simultaneously significantly reducing bandwidth.

Low-complexity separable multiplier-less loop filter for video coding.

In this paper, we present a low-complexity loop filter for video coding. We begin by presenting a set of non-Wiener based loop filters that can complement the Wiener-based adaptive loop filter which was considered as a tool for possible adoption in the HEVC standard. We devise filters which can be adaptively operated on different regions in an image. We devise a quad-tree based signaling for the filters, and present various loop filters: such as bilateral and Gaussian, as well as a separable 3-tap filter which can be implemented by just shifts and adds. The proposed 3-tap filter is thus hardware friendly with minimal complexity. In terms of compression performance, the proposed 3-tap filter can approach other sophisticated filters, albeit at a substantially reduced complexity; and can provide compression gains of 2.3% on average; and upto 7.0% for Low-Delay-P configuration over a data-set of 19 diverse HD, and UHD sequences of upto 8K resolution when implemented on top of the HM14.0 software for HEVC. Finally, we also present results for combining our proposed filters with the Wiener-based adaptive loop filter considered in HEVC, and illustrate that there is a significant amount of compression gain that can be achieved by loop filters for the next generation of video coding standard beyond HEVC.