Demin Wang

Motion-Compensated Frame Rate Up-Conversion—Part II: New Algorithms for Frame Interpolation

Motion-compensated frame rate up-conversion (MC-FRUC) consists of two key elements: motion estimation and motion-compensated frame interpolation. The motion estimation algorithm presented in , which is used in the MC-FRUC method proposed in this paper, provides unidirectional motion trajectories. The advantage of this motion estimation algorithm, besides its accuracy, is that it provides information on occlusions. However, motion compensation along unidirectional motion trajectories yields overlaps, holes, and blocking artifacts. To solve these problems, this paper presents two new algorithms for unidirectional motion-compensated frame interpolation: irregular-grid expanded-block weighted motion compensation (IEWMC) and block-wise directional hole interpolation (BDHI). The IEWMC is used to reduce the blocking artifacts and solve the problem of overlapping blocks. The BDHI preserves local texture and edges while filling holes. Experimental results show that the IEWMC outperforms conventional motion compensation, and the BDHI is better than the repeated median filter that is often used to fill holes. The performance of the proposed MC-FRUC, that uses the two new algorithms and the unidirectional motion estimation algorithm, is evaluated against three existing MC-FRUC techniques: a typical bi-directional algorithm, an object-based algorithm, and a commercial plug-in product. Experimental results show that the quality of the pictures interpolated using the proposed MC-FRUC method is much higher than those interpolated using the three existing MC-FRUC techniques.

Motion-Compensated Frame Rate Up-Conversion—Part I: Fast Multi-Frame Motion Estimation

Motion-compensated frame rate up-conversion is used to convert video/film materials of low frame rates to a higher frame rate so that the materials can be displayed with smooth motion and high-perceived quality. It consists of two key elements: motion estimation and motion-compensated frame interpolation. It requires accurate motion trajectories to ensure quality results and low computational cost to ensure practical applications. This paper presents a novel motion estimation algorithm that combines the accuracy of maximum a posteriori probability (MAP) estimation with the speed of hierarchical block-matching algorithm (BMA). This MAP estimation uses three consecutive pictures, instead of the conventional two, and one previously estimated motion field to exploit the temporal correlation between motion fields and to determine motion in occluded areas. The optimization of the MAP estimation is performed using full-search and implemented by means of look-up tables. The full search ensures that the optimization converges to the global minimum, while the look-up tables dramatically reduce the computational cost. Experimental results show that the proposed algorithm provides motion trajectories that are much more accurate than those obtained using either the full-search BMA or hierarchical BMA alone. Also, it is much faster than the full-search BMA.

Wyner-Ziv video coding with region adaptive quantization and progressive channel noise modeling

In this paper, we propose a region-adaptive quantization scheme and progressive side information/channel noise modeling for Wyner-Ziv video coding. The proposed region-adaptive quantization scheme can enhance coding performance and provide an additional functionality of region of interest (ROI) coding. It also improves the accuracy of channel noise modeling by using previously decoded bit-plane information. Simulation results show that the proposed region-adaptive quantization scheme is especially suitable for static sequences, and the progressive side information/channel noise modeling works well in dynamic scenes.

Distributed video coding without channel codes

Distributed video coding allows the compression of video frames in a distributed fashion leading to a rather simple computational encoding, but requiring a complex decoding. The main drawback however is the decoding complexity making some practical applications difficult. Nowadays, most distributed video coding schemes are based on efficient channel codes such as LDPC and Turbo codes. This is the main cause of the decoders high complexity. This paper proposes a new distributed video coding scheme that can avoid the use of such codes. It is based on an adaptive representation of the source frames combined with a DC-guided scheme. This combination can reduce the data that need to be transmitted from the encoder to the decoder. This subsequently allows complex channel coding to be replaced by simple entropic coding methods, such as arithmetic source coding, with little performance degradation. Experimental results show that the proposed scheme can significantly improve the performance compared to conventional distributed video coding schemes, while enabling a much lower computational complex decoder.

Adaptive source representation for distributed video coding

In the last few years distributed video coding (DVC) has become a new paradigm for video compression where the encoding process needs to be simple. DVC allows for the development of new applications where the computational complexity and the amount of memory inside the video encoder are limited. In this paper, we introduce a new data representation and coding method for DVC systems. The proposed method consists of an adaptive binary representation based on the maximum difference between the source picture and the side information. This adaptive representation provides a reduction of bitrate without any loss of information, while maintaining the low complexity of the encoder. Experimental results show a significant increase in performance over conventional DVC methods.

Segmentation of Source Symbols for Adaptive Arithmetic Coding

Adaptive arithmetic coding is a general technique for coding source symbols of a stochastic process based on an adaptive model. The adaptive model provides measures of the statistics of source symbols and is updated, along with encoding/decoding processes, when more encoded/decoded symbols are fed as samples to the adaptive model. The coding performance depends on how well the adaptive model fits the statistics of source symbols. If the number of source symbols is large and the number of samples is small, the adaptive model may not be able to provide valid measures of the statistics, which results in an inefficient coding performance of the adaptive arithmetic coder. To this end, this paper presents segmentation of source symbols to improve the performance of the adaptive arithmetic coder. Each source symbol is divided into several segments. Each segment is separately coded with an adaptive arithmetic coder. With this division, possible values of each segment are concentrated within a small range. Given the limited number of samples, this concentration leads to a better fit of the adaptive model to the statistics of source symbols and therefore to an improvement of the coding efficiency. The proposed coding algorithm is applied to lossless motion vector coding for video transmission as an application example to show its performance improvement and coding gains.

DC-guided compression scheme for distributed video coding

This paper presents a new distributed video coding (DVC) scheme where the DC and the AC coefficients of DCT transform are separately encoded and transmitted. The DC coefficients are first transmitted to the decoder and compared with their corresponding DC coefficients found in the side information derived within the decoder. The result of the comparison is then transmitted back to the encoder and used to decide whether the AC coefficients of a block need to be transmitted. Experimental results show that the proposed scheme can significantly improve the performance, while maintaining a low computational complexity at the encoder.

A high performance hardware architecture for multi-frame hierarchical motion estimation

This paper presents the architecture design and FPGA implementation of a multi-frame hierarchical motion estimation (MFHME) circuit. The target application of the circuit is high quality motion-compensated video frame rate up-conversion that requires dense motion fields (MF) and accurate motion trajectories. To obtain accurate motion trajectories, the circuit uses two frames as references and calculates the block matching errors for both the luminance and chrominance components of the images. In addition, the sum of squared pixel differences, instead of the sum of the absolute pixel differences, is used as the metric of the block matching errors in order to further improve the accuracy of the estimated motion trajectories. To achieve low computation complexity, the circuit has been designed based on a hierarchical structure and a pre-computed lookup table is used to provide the squared pixel differences. The implementation result shows that the circuit is able to support the frame rate up-conversion of high definition video (1080P format) from 30 to 60 frames per second at a clock frequency of 55 MHz.

Maximum Likelihood Estimation Sample Consensus with Validation of Individual Correspondences

This paper presents an extension of the maximum likelihood estimation sample consensus (MLESAC) by introducing an online validation of individual correspondences, which is based on the Law of Large Numbers (LLN). The outcomes of the samples, each considered a random event, are analyzed for useful information regarding the validities of individual correspondences. The information from the individual samples that have been processed is accumulated and then used to guide subsequent sampling and to score the estimate. To evaluate the performance of the proposed algorithm, the proposed method was applied to the problem of estimating the fundamental matrix. Experimental results with the Oxford image sequence, Corridor , showed that for a similar consensus the proposed algorithm reduced, on average, the Sampson error by about 13% and 12% in comparison to the RANSAC and the MLESAC estimator, while the associated number of samples decreased by about 14% and 15%, respectively.

Validation of correspondences in MLESAC robust estimation

This paper presents an extension to the maximum likelihood estimation sample consensus (MLESAC) algorithm by estimating the prior validity of correspondences using both the measured data and a model hypothesis. Validity is determined based on the data set associated with the model that is considered as the best one so far in the previous random trials. The proposed robust algorithm is applied to estimate the fundamental matrix using randomly generated synthetic test data. Experiment results show that at various outlier ratios the proposed algorithm reduces the Sampson error and is also faster (in terms of the number of trials) in comparison to other conventional algorithms.