Edson M. Hung

On Macroblock Partition for Motion Compensation

In the H.264/AVC video coding standard, motion compensation can be performed by partitioning macroblocks into square or rectangular sub-macroblocks in a quadtree decomposition. This paper studies a motion compensation method using wedges, i.e. partitioning macroblocks or sub-macroblocks into two regions by an arbitrary line segment. This technique allows the shapes of the divided regions to better match the boundaries between moving objects. However, there are a large number of ways to slice a block and searching exhaustively over all of them would be an extremely computer-intensive task. Thus, we propose a fast algorithm which detects the predominant edge orientations within a block in order to pre-select candidate wedge lines. Finally a comparison among macroblock partition methods is performed, which points to the higher performance of the wedge partition method.

Video Super-Resolution Using Codebooks Derived From Key-Frames

Example-based super-resolution (SR) is an attractive option to Bayesian approaches to enhance image resolution. We use a multiresolution approach to example-based SR and discuss codebook construction for video sequences. We match a block to be super-resolved to a low-resolution version of the reference high-resolution image blocks. Once the match is found, we carefully apply the high-frequency contents of the chosen reference block to the one to be super-resolved. In essence, the method relies on “betting” that if the low-frequency contents of two blocks are very similar, their high-frequency contents also might match. In particular, we are interested in scenarios where examples can be picked up from readily available high-resolution images that are strongly related to the frame to be super-resolved. Hence, they constitute an excellent source of material to construct a dynamic codebook. Here, we propose a method to super-resolve a video using multiple overlapped variable-block-size codebooks. We implemented a mixed-resolution video coding scenario, where some frames are encoded at a higher resolution and can be used to enhance the other lower-resolution ones. In another scenario, we consider the framework where the camera captures a video at a lower resolution and also takes periodic snapshots at a higher resolution. Results indicate substantial gains over interpolation and fixed-codebook SR, and significant gains over previous works as well.

Loss-Resilient Coding of Texture and Depth for Free-Viewpoint Video Conferencing

Free-viewpoint video conferencing allows a participant to observe the remote 3D scene from any freely chosen viewpoint. An intermediate virtual viewpoint image is typically synthesized using two pairs of transmitted texture and depth maps from two neighboring captured viewpoints via depth-image-based rendering (DIBR). To maintain high quality of synthesized images, it is imperative to contain the adverse effects of network packet losses that may arise during texture and depth video transmission. Towards this goal, we develop an integrated approach that exploits the representation redundancy inherent in the multiple streamed videos-a voxel in the 3D scene visible to two captured views is sampled and coded twice in the two views. In particular, at the receiver we first develop an error concealment strategy that adaptively blends corresponding pixels in the two captured views during DIBR, so that pixels from the more reliable transmitted view are weighted more heavily. We then couple it with a sender-side optimization of reference picture selection (RPS) during real-time video coding, so that blocks containing pixel samples of voxels that are visible in both views are more error-resiliently coded in one view only, given adaptive blending will mitigate errors in the other view. Further, synthesized view distortion sensitivities to texture versus depth errors are analyzed, so that relative importance of texture and depth code blocks can be computed for system-wide RPS optimization. Finally, quantization parameter (QP) is adaptively selected per frame, optimally trading off source distortion due to compression with channel distortion due to potential packet losses. Experimental results show that the proposed scheme can outperform previous work by up to 2.9 dB at 5% packet loss rate.

Efficiency improvements for a geometric-partition-based video coder

H.264/AVC has brought an important increase in coding efficiency in comparison to previous video coding standards. One of its features is the use of macroblock partitioning in a tree-based structure. The use of macroblock partitions based in arbitrary line segments, like wedge partitions, has been reported to increase coding gains. The main problem of these non-standard new partitions schemes is the increase in computational complexity. Thus, our work proposes improvements in this extension to the H.264/AVC standard. First, we present a motion vector prediction scheme based on directional partitions. Second, we present a method for complexity reduction based on the most frequent partitions. The results show that it is possible to still produce good coding gains with lower complexity than previous approaches.

An H.264/AVC to HEVC video transcoder based on mode mapping

The emerging video coding standard, HEVC, was developed to replace the current standard, H.264/AVC. However, in order to promote inter-operability with existing systems using the H.264/AVC, transcoding from H.264/AVC to the HEVC codec is highly needed. This paper presents a transcoding solution that uses machine learning techniques in order to map H.264/AVC macroblocks into HEVC coding units (CUs). Two alternatives to build the machine learning model are evaluated. The first uses a static training, where the model is built offline and used to transcode any video sequence. The other uses a dynamic training, with two well-defined stages: a training stage and a transcoding stage. In the training stage, full re-encoding is performed while the H.264/AVC and the HEVC information are gathered. This information is then used to build a model, which is used in the transcoding stage to classify the HEVC CU partitioning. Both solutions are tested with well-known video sequences and evaluated in terms of rate-distortion (RD) and complexity. The proposed method is on average 2.26 times faster than the trivial transcoder using fast motion estimation, while yielding a RD loss of only 3.6% in terms of bitrate.

Reference frame selection for loss-resilient texture & depth map coding in multiview video conferencing

In a free-viewpoint video conferencing system, the viewer can choose any desired viewpoint of the 3D scene for observation. Rendering of images for arbitrarily chosen viewpoint can be achieved through depth-image-based rendering (DIBR), which typically employs “texture-plus-depth” video format for 3D data exchange. Robust and timely transmission of multiple texture and depth maps over bandwidth-constrained and loss-prone networks is a challenging problem. In this paper, we optimize transmission of multiview video in texture-plus-depth format over a lossy channel for free viewpoint synthesis at decoder. In particular, we construct a recursive model to estimate the distortion in synthesized view due to errors in both texture and depth maps, and formulate a rate-distortion optimization problem to select reference pictures for macroblock encoding in H.264 in a computation-efficient way, in order to provide unequal protection to different macroblocks. Results show that the proposed scheme can outperform random insertion of intra refresh blocks by up to 0.73 dB at 5% loss.

Fast H.264/AVC to HEVC transcoding based on machine learning

Since the HEVC codec has become an ITU-T and ISO/IEC standard, efficient transcoding from previous standards, such as the H.264/AVC, to HEVC is highly needed. In this paper, we build on our previous work with the goal to develop a faster transcoder from H.264/AVC to HEVC. The transcoder is built around an established two-stage transcoding. In the first stage, called the training stage, full re-encoding is performed while the H.264/AVC and the HEVC information are gathered. This information is then used to build a CU classification model that is used in the second stage (called the transcoding stage). The solution is tested with well-known video sequences and evaluated in terms of rate-distortion and complexity. The proposed method is 3.4 times faster, on average, than the trivial transcoder, and 1.65 times faster than a previous transcoding solution.

A fast HEVC transcoder based on content modeling and early termination

In this paper, a fast transcoding solution from H.264/AVC to HEVC bitstreams is presented. This solution is based on two main modules: a coding unit (CU) classification module that relies on a machine learning technique in order to map H.264/AVC macroblocks into HEVC CUs; and an early termination technique that is based on statistical modeling of the HEVC rate-distortion (RD) cost in order to further speed-up the transcoding. The transcoder is built around an established two-stage transcoding. In the first stage, called the training stage, full re-encoding is performed while the H.264/AVC and the HEVC information are gathered. This information is then used to build both the CU classification model and the early termination sieves, that are used in the second stage (called the transcoding stage). The solution is tested with well-known video sequences and evaluated in terms of RD and complexity. The proposed method is 3.83 times faster, on average, than the trivial transcoder, and 1.8 times faster than a previous transcoding solution, while yielding a RD loss of 4% compared to this solution.

Context adaptive mode sorting for fast HEVC mode decision

Typical H.265/High Efficiency Video Coding (HEVC) encoder implementations test a variety of prediction modes and select the optimal configuration for each block in terms of Rate-Distortion (RD) cost. A fast HEVC mode decision algorithm is proposed here referred to as Context Adaptive Mode Sorting (CAMS). The frequency of selection of modes and their RD costs are collected while encoding the training frames based on local parameters (the context). This information is then used to sort and restrict the prediction modes to test for each context, while the optimal mode found using CAMS on each CU is validated based on the RD cost distributions found during the training. Experimental results show that the method reduces total encoding time of fast HEVC implementations on average by 29.3%, at modest efficiency losses.

Video super-resolution based on local invariant features matching

This paper presents an algorithm for video super-resolution based on scale-invariant feature transform (SIFT) matching. SIFT features are known to be a robust method for locating keypoints. The matching of these keypoints from different frames in a video allows us to infer high-frequency information in order to perform example-based super-resolution. We first apply a block constrained keypoint detection for a more precise superposition of features. Later, we extract high-frequency information with a gradient-based matching scheme. Our results indicate gains over interpolation and previous example-based super-resolution approaches.