Wen-Hsiao Peng

Advances in the scalable amendment of H.264/AVC

To support clients with diverse capabilities, ISO/IEC MPEG and ITU-T form a joint video team (JVT) to develop a scalable video coding (SVC) technology that uses single bitstream to provide multiple spatial, temporal, and quality (SNR) resolutions, thus satisfying low-complexity and low-delay constraints. It is an amendment of the emerging standard H.264/AVC and it provides an H.264/AVC-compatible base layer and a fully scalable enhancement layer, which can be truncated and extracted on-the-fly to obtain a preferred spatio-temporal and quality resolution. An overview of the adopted key technologies in the SVC and a comparison in coding efficiency with H.264/AVC are presented

A platform-based MPEG-4 advanced video coding (AVC) decoder with block level pipelining

We present a baseline MPEG-4 AVC (advanced video coding) decoder based on an optimized platform-based design methodology. With this methodology, we jointly optimize the software and hardware design of the decoder. Overall decoding throughput is increased by synchronizing the software and the dedicated co-processors. The synchronization is achieved at macroblock-level pipelining. In addition, we optimize the decoder software by enhancing the frame buffer management, boundary padding, and content aware inverse transform. To speed up motion compensation and inverse transform, which are the most computationally intensive modules, two dedicated acceleration modules are realized. For comparison, the proposed prototype decoder and MPEG-4 AVC reference decoder are evaluated on an ARM platform, which is one of most popular portable devices. Our experiments show that the throughput of the MPEG-4 reference decoder can be improved by 6 to 7 times. On an ARM966 board, the optimized software without hardware acceleration can achieve a decoding rate up to 5 frames per second (fps) for QCIF video sequences. With the dedicated accelerators, the overall throughput is increased by about 30% to reach 6.6 fps on the average and is up to 10.3 fps for slow motion video sequences.

A platform based bus-interleaved architecture for de-blocking filter in H.264/MPEG-4 AVC

In this paper, we proposed a platform based bus-interleaved architecture for the de-blocking filter in H.264. Specifically, to efficiently use the bus bandwidth, we classify the filtering mode into 8 types and use an adaptive transmission scheme to avoid redundant data transfer. Moreover, to reduce the processing latency, we use a bus-interleaved architecture for conducting data transmission and parallel filtering. As compared to the state-of-the-art designs, our scheme offers 1.6x to 7x performance improvement. While clocking at 100 MHz, our design can support 2560/spl times/1280 @ 30 Hz processing throughput. The proposed design is suitable for low cost and real-time applications. Moreover, it can be easily applied in system-on-chip design.

Layer-Adaptive Mode Decision and Motion Search for Scalable Video Coding with Combined Coarse Granular Scalability (CGS) and Temporal Scalability

In this paper, we propose a layer-adaptive mode decision algorithm and a motion search scheme for the scalable video coding (SVC) with combined coarse granular scalability (CGS) and temporal scalability. To speed up the encoder while minimizing the loss in coding efficiency, our layer-adaptive mode decision recursively refers to the prediction modes and quantization parameter of the reference/base layer to minimize the number of modes tested at the enhancement layer. Moreover, our motion search scheme adaptively reuses the reference frame indices of the base layer and determines the initial search point using the motion vector at the base layer or the motion vector predictor at the enhancement layer. As compared with JSVM 8, the proposed algorithms provide up to 75% overall time saving and more than 85% time reduction for encoding enhancement layers with negligible loss in coding efficiency.

A Software-Hardware Co-Implementation of MPEG-4 Advanced Video Coding (AVC) Decoder with Block Level Pipelining

We present a baseline MPEG-4 Advanced Video Coding (AVC) decoder based on the methodology of joint optimization of software and hardware. The software is first optimized with algorithm improvements for frame buffer management, boundary padding, content-aware inverse transform and context-based entropy decoding. The overall decoding throughput is further enhanced by pipelining the software and the dedicated hardware at macroblock level. The decoder is partitioned into the software and hardware modules according to the target frame rate and complexity profiles. The hardware acceleration modules include motion compensation, inverse transform and loop filtering. By comparing the optimized decoder with the committee reference decoder of Joint Video Team (JVT), the experimental results show improvement on the decoding throughput by 7 to 8 times. On an ARM966 board, the optimized software without hardware acceleration can achieve a decoding rate up to 5.9 frames per second (fps) for QCIF video source. The overall throughput is improved by another 27% to 7.4 fps on the average and up to 11.5 fps for slow motion video sequences. Finally, we provide a theoretical analysis of the ideal performance of the proposed decoder.

Fast Context-Adaptive Mode Decision Algorithm for Scalable Video Coding With Combined Coarse-Grain Quality Scalability (CGS) and Temporal Scalability

To speed up the H.264/MPEG scalable video coding (SVC) encoder, we propose a layer-adaptive intra/inter mode decision algorithm and a motion search scheme for the hierarchical B-frames in SVC with combined coarse-grain quality scalability (CGS) and temporal scalability. To reduce computation but maintain the same level of coding efficiency, we examine the rate-distortion (R-D) performance contributed by different coding modes at the enhancement layers (EL) and the mode conditional probabilities at different temporal layers. For the intra prediction on inter frames, we can reduce the number of Intra4×4/Intra 8×8 prediction modes by 50% or more, based on the reference/base layer intra prediction directions. For the EL inter prediction, the look-up tables containing inter prediction candidate modes are designed to use the macroblock (MB) coding mode dependence and the reference/base layer quantization parameters (Qp). In addition, to avoid checking all motion estimation (ME) reference frames, the base layer (BL) reference frame index is selectively reused. And according to the EL MB partition, the BL motion vector can be used as the initial search point for the EL ME. Compared with Joint Scalable Video Model 9.11, our proposed algorithm provides a 20× speedup on encoding the EL and an 85% time saving on the entire encoding process with negligible loss in coding efficiency. Moreover, compared with other fast mode decision algorithms, our scheme can demonstrate a 7-41% complexity reduction on the overall encoding process.

A rate-distortion optimization model for SVC inter-layer encoding and bitstream extraction

The Scalable Video Coding (SVC) standard enables viewing devices to adapt their video reception using bitstream extraction. Since SVC offers spatial, temporal, and quality combined scalability, extracting proper bitstreams for different viewing devices can be a non-trivial task, and naive choices usually produce poor playback quality. In this paper, we propose a two-prong approach to achieve rate-distortion (R-D) optimal extraction of SVC bitstreams. For SVC encoding, we developed a set of adaptation rules for setting the quantization parameters and the inter-layer dependencies among the SVC coding layers. A well-adapted SVC bitstream thus produced manifests good R-D trade-offs when its scalable layers are extracted along extraction paths consisting of successive refinement steps. For extracting R-D optimized bitstreams for different viewing devices, we formalized the notion of optimal and near-optimal extraction paths and devised computationally efficient strategies to search for the extraction paths. Experiment results demonstrated that our R-D optimized adaptation schemes and extraction strategies offer significant improvement in playback picture quality among heterogeneous viewing devices. Particularly, our adaptation rules promise R-D convexity along optimal extraction paths and permit the use of steepest-descent strategy to discover the optimal/near-optimal paths. This simple search strategy performs only half of the computation necessary for an exhaustive search.

Design of Memory Sub-System with Constant-Rate Bumping Process for H.264/AVC Decoder

In this paper, we propose an efficient memory sub-system and a constant-rate humping process for a H.264/AVC decoder conforming to High profile@Level 4. To efficiently utilize the throughput of external DRAM, a synchronization buffer is employed as a bridge for reformatting the read/write data exchanged between the on-chip hardware and the off-chip DRAM. In addition, we optimize the issues of read/write commands and adaptively enable the auto-precharge function by monitoring the motion information of a submacroblock. Furthermore, a regulation buffer with size comparable to the decoded picture buffer is created to ensure a constant output rate of decoded pictures for any conformed prediction structures. Along with other modules, the proposed scheme is verified at system level using transaction level modeling (TLM) technique. Statistical results show that synchronization buffer of larger block size provides higher memory efficiency, less access cycles and power dissipation. However, the granularity of 8times8 block size provides better trade-off among cost, efficiency, power, and real-time requirement

A Synthesis-Quality-Oriented Depth Refinement Scheme for MPEG Free Viewpoint Television (FTV)

This paper addresses the problem of refining depth information from the received reference and depth images within the MPEG FTV framework. An analytical model is first developed to approximate the per-pixel synthesis distortion (caused by depth-image compression) as a function of depth-error variances, intensity variations, ground-truth depth and virtual camera locations. We then follow the model to detect unreliable depth pixels by inspecting intensity gradients and to refine their values with a candidate-based block disparity search. Additional side information is transmitted to make both operations robust against compression effects. Experimental results show that our scheme offers an average PSNR improvement of 1.2 dB over MPEG FTV and consistently outperforms the state-of-the-art methods. Moreover, it can remove synthesis artifacts to a great extent, producing a result that is very close in appearance to the ground-truth view image.

Error drifting reduction in enhanced fine granularity scalability

We incorporate fading and reset mechanisms in an enhanced fine granularity scalability algorithm to reduce the drifting error at low bit rate while still maintaining 1.5dB PSNR gain at high bit rate over the current MPEG-4 fine granularity scalability. Many previous works use enhancement layers to predict enhancement layers so as to increase the compression efficiency. Drifting error occurs because the enhancement layer, the predictor, is not received as expected. Our fading mechanism linearly combines the current reconstructed base layer and previously reconstructed enhancement layer with fading factors between 0 and 1. Our reset mechanism sets the reference frame for prediction to be the base layer periodically. Our theoretical formulation and experimental results show that drifting error can be distributed more uniformly and maximum accumulated mismatch error is significantly reduced while our mechanisms are turned on. Around 1dB can be improved at low bit rate comparing to the one without any drifting reduction mechanism.