Camilo C. Dorea

Super Resolution for Multiview Images Using Depth Information

Mixed resolution formats have been employed in video encoding complexity reduction as well as data compression of stereoscopic video. High resolution frames within such formats may also be used as a means of enhancing lower resolution images. In this paper we present a super-resolution method for use in a mixed resolution, multiview video plus depth setup. High resolution views are initially projected onto the view point of low resolution images with the aid of available depth maps. The method introduces the use of transform-domain techniques for up-sampling the low resolution images and for appending high frequency content from projected views. The DCT is used for the necessary frequency decompositions. Results show gains over previous work for several test sequences and affirm the aptitude of transform-domain approaches for super-resolution within mixed resolution formats.

Loss-Resilient Coding of Texture and Depth for Free-Viewpoint Video Conferencing

Free-viewpoint video conferencing allows a participant to observe the remote 3D scene from any freely chosen viewpoint. An intermediate virtual viewpoint image is typically synthesized using two pairs of transmitted texture and depth maps from two neighboring captured viewpoints via depth-image-based rendering (DIBR). To maintain high quality of synthesized images, it is imperative to contain the adverse effects of network packet losses that may arise during texture and depth video transmission. Towards this goal, we develop an integrated approach that exploits the representation redundancy inherent in the multiple streamed videos-a voxel in the 3D scene visible to two captured views is sampled and coded twice in the two views. In particular, at the receiver we first develop an error concealment strategy that adaptively blends corresponding pixels in the two captured views during DIBR, so that pixels from the more reliable transmitted view are weighted more heavily. We then couple it with a sender-side optimization of reference picture selection (RPS) during real-time video coding, so that blocks containing pixel samples of voxels that are visible in both views are more error-resiliently coded in one view only, given adaptive blending will mitigate errors in the other view. Further, synthesized view distortion sensitivities to texture versus depth errors are analyzed, so that relative importance of texture and depth code blocks can be computed for system-wide RPS optimization. Finally, quantization parameter (QP) is adaptively selected per frame, optimally trading off source distortion due to compression with channel distortion due to potential packet losses. Experimental results show that the proposed scheme can outperform previous work by up to 2.9 dB at 5% packet loss rate.

Super-resolution for multiview images using depth information

In stereoscopic and multiview video, binocular suppression theory states that the visual subjective quality of 3-D experience is not much affected by asymmetrical blurring of the individual views. Based on these studies, mixed-resolution frameworks applied for multiview systems offer great data-size reduction without incurring in significant quality degradation in 3-D video applications. However, it is interesting to recover high-frequency content of the blurred views, to reduce visual strain due to long-term exposure and to make the system suitable for free-viewpoint television. In this paper, we present a novel super-resolution technique, in which low-resolution views are enhanced with the aid of high-frequency content from neighboring full-resolution views, and the corresponding depth information for all views. Occlusions are handled by checking the consistency between views. Tests for synthetic and real image data in stereo and multiview cases are presented, and results show that significant objective quality gains can be achieved without any extra side information.

Reference frame selection for loss-resilient texture & depth map coding in multiview video conferencing

In a free-viewpoint video conferencing system, the viewer can choose any desired viewpoint of the 3D scene for observation. Rendering of images for arbitrarily chosen viewpoint can be achieved through depth-image-based rendering (DIBR), which typically employs “texture-plus-depth” video format for 3D data exchange. Robust and timely transmission of multiple texture and depth maps over bandwidth-constrained and loss-prone networks is a challenging problem. In this paper, we optimize transmission of multiview video in texture-plus-depth format over a lossy channel for free viewpoint synthesis at decoder. In particular, we construct a recursive model to estimate the distortion in synthesized view due to errors in both texture and depth maps, and formulate a rate-distortion optimization problem to select reference pictures for macroblock encoding in H.264 in a computation-efficient way, in order to provide unequal protection to different macroblocks. Results show that the proposed scheme can outperform random insertion of intra refresh blocks by up to 0.73 dB at 5% loss.

Trajectory Tree as an Object-Oriented Hierarchical Representation for Video

This paper presents the trajectory tree as a hierarchical region-based representation for video sequences. Motion, as well as spatial features from multiple frames are used to generate a set of temporal regions structured within a hierarchy of scale and motion coherency. The resulting representations offer a global description of the entire video sequence and enhance semantic analysis potential. A multiscale segmentation strategy is proposed whereby region-merging criteria of progressively greater complexity are used to define partition layers of increasing aptitude for object detection. A novel data structure, called the trajectory adjacency graph, is defined for the long-term analysis of partition sequences. Furthermore, mechanisms for assessing connectivity, verifying temporal continuity, and proposing merging operations based on color, affine, and translational motion homogeneity characteristics over the entire sequence are also introduced. Finally, as demonstrated through experimental results, the trajectory tree offers a concise yet versatile support for video object segmentation, description and retrieval tasks.

Depth map reconstruction using color-based region merging

This paper presents a novel depth map enhancement method which takes as inputs a single view and an initial depth estimate. A region-based framework is introduced wherein a color-based partition of the image is created and depth uncertainty areas are identified according to the alignment of detected depth discontinuities and region borders. A color-based homogeneity criterion is used to guide a constrained region merging process and reconstruct depth estimates within the uncertainty areas. Experimental results on publicly available test sequences illustrate the potential of the algorithm in significantly improving low quality depth estimates.

A motion-based binary partition tree approach to video object segmentation

This paper describes an approach for generating binary partition tree [P. Salembier and L. Garrido, 2000] representations and video object segmentations using a novel region merging strategy based on motion similarity measures of multiple frames of an image sequence. The system operates over color-homogeneous regions, tracked across frames of a shot, representing an over-segmentation of the objects. A long-term motion similarity measure is introduced for region merging, offering accurate segmentation of objects and extending temporal consistency between the tracked partitions to hierarchical representations of every frame within the shot. Experimental results are presented, illustrating the usefulness of the approach.

Generation of Long-Term Color and Motion Coherent Partitions

This paper describes a technique for generating partition sequences of regions presenting long-term homogeneity in color and motion coherency in terms of affine models. The technique is based on region merging schemes compatible with hierarchical representation frameworks and can be divided into two stages: partition tracking and partition sequence analysis. Partition tracking is a recursive algorithm whereby regions are constructed according to short-term spatio-temporal features, namely color and motion. Partition sequence analysis proposes the trajectory adjacency graph (TAG) to exploit the long-term connectivity relations of tracked regions. A novel trajectory merging strategy using color homogeneity criteria over multiple frames is introduced. Algorithm performance is assessed and comparisons to other proposals are drawn by means of established evaluation metrics.

Attention-Weighted Texture and Depth Bit-Allocation in General-Geometry Free-Viewpoint Television

In a free-viewpoint television network, each viewer chooses its point of view from which to watch a scene. We use the concept of total observed distortion, wherein we aim to minimize the distortion of the view observed by the viewers as opposed to the distortion of each camera, to develop an optimized bit-rate allocation for each camera. Our attention-weighted approach effectively gives more bits to the cameras that are more watched. The more concentrated the viewer distribution, the larger the bit-rate savings, for a given total observed distortion, compared with the uniform rate allocation. We analyze and model the distortion of a synthesized view as a function of the distortions (both in texture and/or depth) of the nearby cameras. Based on such models, we develop optimal rate-allocation methods for texture images, considering a uniform bit allocation for depth, and for both texture and depth simultaneously. Simulation results are shown, demonstrating not only the correctness of the optimized solution, but also measuring its improvement against uniform rate allocation for a few viewer distributions.

General rate-allocation in free-viewpoint television

We propose a framework for optimal rate-allocation in free-viewpoint television (FVTV) for a general camera arrangement based on the attention the viewers are paying to each camera. In a recent letter [1], the authors proposed a FVTV broadcast architecture and an optimal bit-allocation approach, assuming a uniformly-spaced one-dimensional arrangement of cameras. Quality (or bit-rate) at each camera was determined by viewer attention in order to minimize total observed distortion. Here, we extend the optimized bit-allocation scheme to allow for a more generic camera arrangement in FVTV. We present results on data sets from 1D and 2D array camera setups which show significant overall PSNR gains and bit-rate savings with respect to equally-balanced rate-allocation across cameras.