Wenjing Zhu

Local saliency detection based fast mode decision for HEVC intra coding

The High Efficiency Video Coding (HEVC) is the next generation video coding standard beyond H.264/AVC. Compared with only up to 9 modes for intra prediction in H.264/AVC, HEVC provides 35 intra prediction modes (IPM) to improve coding efficiency, which inevitably poses a huge complexity burden to the encoder. To speed up the HEVC encoder, a novel fast mode decision (FMD) algorithm for HEVC intra prediction is proposed. In the proposed algorithm, we analyzed the costs generated by rough mode decision (RMD), which has already been incorporated in the HM software. We found that the RMD costs listed by mode number generally follow the same trend with the rate-distortion optimization (RDO) costs. Further, the local salient modes, whose RMD costs have a significant drop compared with adjacent modes, tend to be promising competitors for the optimal mode. Based on these observations, we further reduced the number of the candidates for the RDO process. Experimental results show that our proposed algorithm achieves averagely 19.0% (up to 33.6%) encoding time saving whilst causing negligible RD performance loss (0.4% BD-Rate increase on average) compared with HM 7.0 anchor.

Color clustering matting

Natural image matting refers to the problem of extracting regions of interest such as foreground object from an image based on user inputs like scribbles or trimap. More specifically, we need to estimate the color information of background, foreground and the corresponding opacity, which is an ill-posed problem inherently. Inspired by closed-form matting and KNN matting, in this paper, we extend the local color line model which is based on the assumption of linear color clustering within a small local window, to nonlocal feature space neighborhood. New affinity matrix is defined to achieve better clustering. Further, we demonstrate that good clustering ensures better prediction of alpha matte. Experimental evaluations on benchmark datasets and comparisons show that our matting algorithm is of higher accuracy and better visual quality than some state-of-the-art matting algorithms.

SSIM-BASED RATE-DISTORTION OPTIMIZATION IN H.264

In the current video coding standards, rate-distortion optimization (RDO) plays an important role in achieving best tradeoff between the perceived distortion and transmission rate. It is widely used in all kinds of encoder decisions, including block mode decision, motion vector selection and so on. Generally, the sum of absolute difference (SAD) or the sum of square difference (SSD) is used as the distortion measurement. However, it is well known that both of them cannot always reflect the perceptual quality of the encoded video. In this paper, an objective quality measurement structural similarity (SSIM) index is proposed as the distortion measurement in the RDO framework for video coding standards. By fully exploiting the relationship between SSIM and mean square error (MSE), the SSIM-based RDO framework can be approximated by the original SSD-based RDO framework with only a scaling of the Lagrange multiplier. Experimental results show that the proposed method outperforms the latest H.264 codec and also the state-of-the-art SSIM-based RDO video codec.

A tutorial on image/video coding standards

The field of image and video compression has gone through rapid growth during the past thirty years, leading to various coding standards. The main goal of continuous efforts on image/video coding standardization is to achieve low bit rate for data storage and transmission, while maintaining acceptable distortion. In this paper, various developmental stages of image and video compression standards are reviewed, including JPEG and JPEG 2000 image standards, MPEG-1, MPEG-2, MPEG-4, H.261, H.263, H.264/MPEG-4 AVC, and the latest international video standard HEVC as well as Chinese video coding standard AVS. Key features and major applications of the standards will be briefly introduced and the compression performance of the standards at each stage will be compared and discussed.

Palette-based compound image compression in HEVC by exploiting non-local spatial correlation

Non-camera captured images (also known as compound image) contain a mixture of camera-captured natural images and computer-generated graphics and texts. Nowadays, there are more and more applications calling for non-camera captured image/video compression scheme. However, current video coding standards, which are designed for natural video, treat non-camera captured video less carefully. For example, the state-of-the-art video coding standard High Efficiency Video Coding (HEVC) may blur or even remove edges in text/graphic region. A lot of schemes are proposed to preserve direction property of texts and graphics, such as palette-based intra coding. In this paper, a novel palette coding scheme is proposed for palette-based intra coding in HEVC. The palette in a block is predicted from an adaptive palette template, which records the statistical non-local spatial correlation of an image. Every block chooses its own palette using the palette template as the prediction in a rate-distortion optimized manner. Experimental results show that the proposed scheme can achieve up to 5.2% bit-rate saving compared to the state-of-the-art palette-based coding scheme in HEVC.

A robust interpolation-free approach for sub-pixel accuracy motion estimation

Motion estimation (ME) is one of the key elements in video coding standard which eliminates the temporal redundancy by using a motion vector (MV) to indicate the best match between the current frame and reference frame. A coarse to fine process is taken to find the best MV. First of all, integer-pixel ME finds a coarse MV and followed by the sub-pixel ME around the best integer-pixel point. The sub-pixel ME plays an important role in improving the coding efficiency. However, the computational complexity of searching one sub-pixel point is much higher than the integer-pixel point searching because of the interpolation and Hadamard transform operation. In this paper, an accurate optimal sub-pixel position prediction algorithm is presented. With the information of the 8 neighboring integer-pixel points, the optimal sub-pixel position is predicted directly without explicitly solving model parameters. Moreover, an outlier rejection scheme is applied to improve the robustness of the proposed algorithm. Experimental results show that the proposed algorithm outperforms the state of the art interpolation-freesub-pixel ME algorithms.

Improved sample adaptive offset for HEVC

High-Efficiency Video Coding (HEVC) is the newest video coding standard which can significantly reduce the bit rate by 50% compared with existing standards. One new efficient tool is sample adaptive offset (SAO), which classifies reconstructed samples into different categories, and reduces the distortion by adding an offset to samples of each category. Two SAO types are adopted in HEVC: edge offset (EO) and band offset (BO). Four 1-D directional edge patterns are used in edge offset type, and only one is selected for each CTB. However, single directional pattern cannot remove artifacts effectively for the CTBs, which contain edges in different directions. Therefore, we analyze the performance of each edge pattern applied on this kind of CTB, and propose to take advantage of existing edge classes and combine some of the them as a new edge offset class, which can adapt to multiple edge directions. All the combinations are tested, and the results show that for Low Delay P condition, they can achieve 0.2% to 0.5% bit rate reduction.

Simplified generalized residual prediction in scalable extension of HEVC

Scalable video coding (SVC), which is an extension of H.264/AVC video coding standard, was introduced to provide scalability in different dimensions for adaptation to heterogeneous network and terminals. After the finalization of the new video coding standard called High Efficiency Video Coding (HEVC), the effort of the standardization committee has been redirected to the investigation of the scalable extension of HEVC. In addition to the basic inter-layer texture prediction mechanism, several other coding tools were proposed for coding performance improvement. Among those coding tools, the one called generalized residual prediction (GRP) scheme achieves most significant coding gain while the consumption of computational power is also huge. In this paper, the GRP mechanism is formulated and analyzed in detail. In addition, the combination of GRP mechanism with merge mode in HEVC is proposed for simplification of the existing GRP mechanism. Results show a better trade-off could be achieved by greatly reducing computational complexity while maintaining most of the coding gain.

Reconfigurable hardware-friendly CU-group based merge/skip mode for high efficient video coding

Merge/skip mode is one of the most important inter prediction tools adopted in the High Efficiency Video Coding (HEVC) standard which is the state-of-the-art video coding standard. It is very efficient in reducing the side information for the blocks within the same object. However, it is difficult for parallel encoding and decoding due to the data dependency problem between neighboring prediction units (PU). Furthermore, different shapes and positions of PUs would result in different definition of the merge/skip candidate list (MCL), which would lead to potentially extra hardware cost and is not easy to be efficiently implemented by the hardware. To deal with this problem, two reconfigurable hardware-friendly MCL construction schemes are proposed in this paper. The first scheme which is called unified MCL (UMCL) uses one candidate list for all PUs inside the motion estimation region (MER), which is regarded as the basic parallel processing unit for the hardware realization. The second scheme which is named boundary MCL (BMCL) allows different candidate lists for the PUs on the boundary of MER. Both of the two schemes can have flexible parallel degree based on the requirement specification. Experimental results show that UMCL reduces the hardware complexity significantly with little coding performance degradation and BMCL achieves significant coding gain while maintaining the hardware complexity.

BDCT compressed image deblocking usingweighted adaptive total variation

Images encoded at low bit rate usually exhibit visually annoying coding artifacts, which are commonly referred as blocking artifacts. In this paper, a novel weighted adaptive total variation method is proposed to remove the blocking artifacts. Based on the analysis of image coding process and block-based discrete cosine transform (BDCT) image properties, image deblocking is formulated as an optimization problem which is solved through approximating the objective function by a set of convex functions. Experimental results show that the proposed method can achieve better objective and subjective quality performance compared to other deblocking algorithms.