Anthony Vetro

Use of two-dimensional deformable mesh structures for video coding. II. The analysis problem and a region-based coder employing an active mesh representation

For pt.I see ibid., vol.6, no.6, p.636-46 (1996). This paper explores the use of the deformable mesh structure for motion/shape analysis and synthesis in an image sequence. We present algorithms for the analysis problem, including scene-adaptive mesh generation and node tracking over successive frames. We also describe a region-based video coder that integrates the analysis and synthesis algorithms presented. The coder describes each region by an ensemble of connected quadrilateral elements embedded in a mesh structure. For each region, its shape and texture are described by the nodal positions and image functions of the elements in this region in an initial frame, while its motion (including shape deformation) is characterized by the nodal trajectories in the following frames, which are in turn specified by a few motion parameters. This coder has been applied to a typical common intermediate format (CIF) resolution, head-and-shoulder type sequence. The visual quality is significantly better than the H.263-TMN4 algorithm at about 50 kb/s (for the luminance component only, 30 Hz).

Bit allocation for MPEG-4 video coding with spatio-temporal tradeoffs

This paper describes rate-control algorithms that consider the tradeoff between coded quality and temporal rate. We target improved coding efficiency for both frame-based and object-based video coding. We propose models that estimate the rate-distortion characteristics for coded frames and objects, as well as skipped frames and objects. Based on the proposed models, we propose three types of rate-control algorithms. The first is for frame-based coding, in which the distortion of coded frames is balanced with the distortion incurred by frame skipping. The second algorithm applies to object-based coding, where the temporal rate of all objects is constrained to be the same, but the bit allocation is performed at the object level. The third algorithm also targets object-based coding, but in contrast to the second algorithm, the temporal rates of each object may vary. The algorithm also takes into account the composition problem, which may cause holes in the reconstructed frame when objects are encoded at different temporal rates. We propose a solution to this problem that is based on first detecting changes in the shape boundaries over time at the encoder, then employing a hole detection and recovery algorithm at the decoder. Overall, the proposed algorithms are able to achieve the target bit rate, effectively code frames and objects with different temporal rates, and maintain a stable buffer level.

An investigation of 3D dual-tree wavelet transform for video coding

This paper examines the properties of a recently introduced 3D dual-tree discrete wavelet transform (DDWT) for video coding. The 3D DDWT is an attractive video representation because it isolates motion along different directions in separate subbands. However, it is an overcomplete transform with 8:1 redundancy. We examine the effectiveness of the iterative projection-based noise shaping scheme proposed by Kingsbury et al., (2002) on reducing the number of coefficients. We also investigate the correlation between subbands at the same spatial/temporal location, both in the significance map and in actual coefficient values.

Frequency domain down-conversion of HDTV using an optimal motion compensation scheme

In the MPEG‐2 Test Model 5, the down‐conversion of an interlaced sequence is obtained by prefiltering and subsampling each field of the image sequence after it has been fully decoded. Although the quality is very good, the cost of such a system is quite high owing to large memory requirements. As a result, low‐resolution decoders have been proposed to reduce some of the costs incurred by this scheme. Here, incoming Discrete Cosine Transform (DCT) blocks are subject to a down‐conversion process within the decoding loop; hence, the motion compensation is performed using the down‐converted images. In past work, it has been proven that the optimal filters for performing this motion compensation are intimately related to the method of down‐conversion. Therefore, the choice of down‐conversion filter is viewed as the primary variable affecting the quality of the down‐converted sequence when such an optimal motion compensation scheme is considered. In the conventional method of frequency domain down‐conversion, the 4 × 4 low‐frequency coefficients are extracted from each 8 × 8 block. Two problems arise from this method: First, the discarding of high frequency data will introduce a disturbing amount of drift; and second, severe blocking artifacts will result in areas of large motion. To remedy the drift problem, a new method of down‐conversion which better preserves high‐frequency data is presented. This method is referred to as frame‐based frequency synthesis. Then, to overcome the blocking artifacts, the previous method is extended to a field‐based frequency synthesis method. Our simulation results clearly indicate that the amount of drift can be significantly reduced by retaining high‐frequency data and that severe blocking artifacts are eliminated by using the field‐based method. In addition, the quality achieved by the proposed field‐based frequency synthesis is much closer to the high‐quality results produced after full‐resolution decoding.

On the motion compensation within a down-conversion decoder

The most straightforward approach in obtaining a down-converted image sequence is to decimate each frame after it has been fully decoded. To reduce memory requirements and other costs incurred by this approach, a down-conversion decoder would perform a decimation within the decoding loop. In this way, predictions are made from a low-resolution reference which has experienced considerable loss of information. Additionally, the predictions must be made from a set of motion vectors which correspond to the full-resolution image sequence. Given these conditions, it is desirable to optimize the performance of the motion compensation process. In this paper we show that the optimal set of filters for performing the low-resolution motion compensation is dependent on the choice of down-conversion filter. The motion compensation filters are determined as the optimal solution of a least squares problem. This problem is formulated in the context of two general classes of down-conversion techniques: one which is dependent on a single block, and another which is dependent on multiple blocks. General solutions for each class of down-conversion are provided. To demonstrate the usefulness of these results, a sample set of motion compensation filters for each class of down-conversion is calculated, and the results are incorporated into a low-resolution decoder. In comparison to a sub-optimal motion compensation scheme, the optimal motion compensation filters realize a drastic reduction in the amount of drift. Simulation results also reveal that the filters which were based on multiple block down-conversion can reduce the amount of prediction drift found in the single block down-conversion by as much as 35%.

Video coding using 3D dual-tree discrete wavelet transforms

The paper explores the use of a recently introduced 3D dual-tree discrete wavelet transform (DDWT) for video coding. The 3D DDWT is an attractive video representation because it isolates motion along different directions in separate subbands. However, it is an overcomplete transform with 8:1 or 4:1 redundancy. Based on the effectiveness of the iterative projection-based noise shaping scheme proposed by Kingsbury in reducing the number of coefficients, and our prior investigation about the correlation between subbands at the same spatial/temporal location, both in the significance map and in actual coefficient values, a new video coding scheme using 3D DDWT is proposed. The proposed video codec does not require motion compensation and provides better performance than the 3D SPIHT codec, both objectively and subjectively, despite the fact that the raw number of coefficients resulting from the 3D DDWT is much more than that of the conventional 3D DWT. The proposed coder allows full scalability in spatial, temporal and quality dimensions.

Generalized motion compensation for drift reduction

The most straight-forward approach in obtaining a down- converted image sequence is to decimate each frame after it has been fully encoded. To reduce memory requirements and other costs incurred by this approach, a down-conversion decoder would perform a decimation within the decoding loop. In this way, predictions are made from a low-resolution reference which has experienced a considerable loss of information. Additionally the prediction s must be made from a set of motion vectors which correspond to the full- resolution image sequence. Given these conditions, it is desirable to optimize the performance of the motion compensation process. In this paper we show that the optimal set of filters for performing the low-resolution motion compensation are dependent on the choice of down-conversion are provided. To demonstrate the usefulness of these results, a sample set of motion compensation filters for each class of down-conversion are calculated. The results are incorporated into a low-resolution decoder and comparisons of each down-conversion class are made. Simulation results reveal that the filters which were based on multiple block down-conversion can reduce the amount of prediction drift found in the single block down-conversion by as much as 35 percent.