Reji Mathew

Quad-Tree Motion Modeling With Leaf Merging

In this paper, we are concerned with the modeling of motion between frames of a video sequence. Typically, it is not possible to represent the motion between frames by a single model and therefore a quad-tree structure is often employed where smaller, variable size regions or blocks are allowed to take on separate motion models. Previous work into quad-tree representations has demonstrated the sub-optimal performance of quad-trees where the dependency between neighboring leaf nodes with different parents is not exploited. Leaf merging has been proposed to rectify this performance loss as it allows joint coding and optimization of related nodes. In this paper, we describe how the merging step can be incorporated into quad-tree motion representations for a range of motion modeling contexts. In particular, we study the impact of rate-distortion optimized merging for two motion coding schemes, these being spatially predictive coding, as used in H.264, and hierarchical coding. We present experimental results which demonstrate that node merging can provide significant gains for both the hierarchical and spatial prediction schemes. Interestingly, experimental results also show that in the presence of merging, the rate-distortion performance of hierarchical coding is comparable to that of spatial prediction. We pursue the case of hierarchical coding further in this paper, introducing polynomial motion models to the quad-tree representation and exploring resolution scalability of the merged quad-tree structure. We also present a theoretical study of the impact of leaf merging in modeling motion, identifying the inherent advantages of merging which give rise to a more efficient description of frame motion.

Scalable Coding of Depth Maps With R-D Optimized Embedding

Recent work on depth map compression has revealed the importance of incorporating a description of discontinuity boundary geometry into the compression scheme. We propose a novel compression strategy for depth maps that incorporates geometry information while achieving the goals of scalability and embedded representation. Our scheme involves two separate image pyramid structures, one for breakpoints and the other for sub-band samples produced by a breakpoint-adaptive transform. Breakpoints capture geometric attributes, and are amenable to scalable coding. We develop a rate-distortion optimization framework for determining the presence and precision of breakpoints in the pyramid representation. We employ a variation of the EBCOT scheme to produce embedded bit-streams for both the breakpoint and sub-band data. Compared to JPEG 2000, our proposed scheme enables the same the scalability features while achieving substantially improved rate-distortion performance at the higher bit-rate range and comparable performance at the lower rates.

Layered coding using bitstream decomposition with drift correction

It is well known that layered video coding is useful both for service interworking and as an aid to error resilience. A major drawback of layered coding is that it invariably results in a reduction in overall coding efficiency for the high quality service. An attractive approach would be one in which the high quality service is coded, and then the resulting bitstream is decomposed in such a way that the lower resolution services can be reconstructed using only a subset of the total generated data. This would mean that there would be no impact on the coding efficiency of the highest quality service. In this paper we demonstrate that the quality of the lower layer when this approach is used is fundamentally limited by drift. It is shown that even a coarsely quantized, low rate correction signal provides a major benefit to the quality of this lower layer service while still not impacting significantly on the coding efficiency of the high quality service. Of course, some overhead is introduced in the coding of the lower quality service but the total overhead is still significantly better than simulcast.

Efficient layered video coding using data partitioning

Data partitioning has been incorporated within the MPEG 2 video coding standard as one means of achieving error resilience in transmission systems where more than one transmission priority level is available. However, data partitioning also includes many ideas from the study of frequency scalability. In particular, the base partition information could be used to form a lower spatial resolution reconstruction of the original service thus permitting service interworking. In this paper, we investigate service interworking using data partitioning and demonstrate that the quality of the lower resolution service when this approach is used is fundamentally limited by drift. It is also shown that a relatively low data rate correction signal can provide a significant increase in the quality of this lower resolution service.

Highly Scalable Coding of Depth Maps with Arc Breakpoints

Recent work highlights the importance of incorporating geometry information into the compression of depth maps. For many applications, features such as resolution scalability and embedded coding are also highly desirable. JPEG 2000 offers these scalability features but suffers from poor compression performance in the vicinity of strong discontinuities. We propose a novel compression strategy for depth maps that incorporates geometry information while retaining the highly scalable coding properties of JPEG 2000. Our scheme involves two separate image pyramid structures, one for arc breakpoints and other for sub-band samples produced by a breakpoint-adaptive transform. Breakpoints capture geometric attributes and are also amenable to scalable coding. We develop an R-D optimization framework for the breakpoint data. We also use a variation of the EBCOT scheme to produce embedded bit-streams for both the breakpoint and sub-band data, allowing them to be independently and incrementally sequenced based on R-D considerations.

Hierarchical and Polynomial Motion Modeling with Quad-Tree Leaf Merging

Recent work into quad-tree representations has commented on the sub-optimal performance of quad-trees due to not exploiting the dependency between neighboring leaf nodes with different parents. Leaf merging therefore has been proposed to rectify this performance loss. In the context of quad-tree motion models, the performance of leaf merging, was recently demonstrated in which an optimally pruned H.264 motion model was taken as the starting point for subsequent merging steps. In this paper we explore the performance of leaf merging over a wider range of conditions by starting with a general tree structure where each node is allowed to represent motion using either polynomial models or a single translational vector. We also explore two cases of motion prediction methods, these being spatial prediction and hierarchical prediction. Experimental results show the benefit of merging across these broad conditions. In comparison to the merged H.264 model reported in the previous work, substantial gains are evident. Furthermore we explore applying the merging principles to branch nodes of a quad-tree to achieve efficient hierarchical motion representation.

Scalable Modeling of Motion and Boundary Geometry With Quad-Tree Node Merging

Quad-tree structures are often used to model motion between frames of a video sequence. However, a fundamental limitation of the quad-tree structure is that it can only capture horizontal and vertical edge discontinuities at dyadically related locations. To address this limitation, recent work has focused on the introduction of geometry information to nodes of tree structured motion representations. In this paper we explore modeling boundary geometry and motion with separate quad-tree structures; thereby enabling each attribute to be refined separately. Recent work into quad-tree representations has also highlighted the benefits of leaf merging. We extend the leaf merging paradigm to incorporate both geometry and motion attributes. We also explore resolution scalability of the merged dual tree representation and present experimental results which demonstrate both rate-distortion and scabalility performance. Theoretical investigations conducted in this paper reveal that to achieve optimal rate-distortion behavior, quad-tree motion models need to incorporate both geometry information and node merging.

A Novel Motion Field Anchoring Paradigm for Highly Scalable Wavelet-Based Video Coding

Existing video coders anchor motion fields at frames that are to be predicted. In this paper, we demonstrate how changing the anchoring of motion fields to reference frames has some important advantages over conventional anchoring. We work with piecewise-smooth motion fields, and use breakpoints to signal discontinuities at moving object boundaries. We show how discontinuity information can be used to resolve double mappings arising when motion is warped from reference to target frames. We present an analytical model that allows to determine weights for texture, motion, and breakpoints to guide the rate-allocation for scalable encoding. Compared with the conventional way of anchoring motion fields, the proposed scheme requires fewer bits for the coding of motion; furthermore, the reconstructed video frames contain fewer ghosting artefacts. The experimental results show the superior performance compared with the traditional anchoring, and demonstrate the high scalability attributes of the proposed method.

Joint estimation of motion and arc breakpoints for scalable compression

An alternative, graphical model for motion estimation (ME) is proposed for video compression using a dense motion field. Unlike other types of optical flow-based motion analysis techniques, the chief goal of our work is jointly to discover a piece-wise smooth motion field with compression-motivated regularization, where the underlying motion compression scheme has resolution and quality scalability attributes. The graphical model formulation is presented and obtained results are discussed.

Motion Modeling with Geometry and Quad-tree Leaf Merging

Quad-tree structures are often used to model motion between frames of a video sequence. However, a fundamental limitation of the quadtree structure is that it can only capture horizontal and vertical edge discontinuities at dyadically related locations. In this paper we seek to address this limitation by introducing geometry information into the nodes of a pruned quad-tree. We start with a typical optimally pruned quad-tree where each node is allowed to model motion. Then for each node in the tree, we consider augmenting the nodes motion model with a linear geometry model. Experimental results show that the introduction of geometry information improves the performance of quad-trees in representing motion. Recent work into quad-tree representations have highlighted the benefits of leaf merging. In this paper we extend the leaf merging paradigm to incorporate both geometry and motion information, allowing the creation of regions that have both motion and geometry attributes.