Giovanni Petrazzuoli

High order motion interpolation for side information improvement in DVC

A key step in distributed video coding is the generation of the side information (SI) i.e. the estimation of the Wyner-Ziv frame (WZF). This step is also frequently called image interpolation. State-of-the-art techniques perform a motion estimation between adjacent key frames (KFs) and linear interpolation in order to assess object positions in the WZF, and then the SI is produced by motion compensating the KFs. However the uniform motion model underlying this approach is not always able to produce a satisfying estimation of the motion, which can result in a low SI quality. In this paper we propose a new method for the generation of SI, based on higher order motion interpolation. We use more than two KFs to estimate the position of the current WZF block, which allows us to correctly estimate more complex motion (such as, for example, uniform accelerated motion). We performed a number of tests for the fine tuning of the parameters of the method. Our experiments show that the new interpolation technique has a small computational cost increase with respect to state of the art, but provides remarkably better performance with up to 0.5 dB of PSNR improvement in SI quality. Moreover the proposed method performs consistently well for several GOP sizes.

Using distributed source coding and depth image based rendering to improve interactive multiview video access

Multiple-views video is commonly believed to be the next significant achievement in video communications, since it enables new exciting interactive services such as free viewpoint television and immersive teleconferencing. However the interactivity requirement (i.e. allowing the user to change the viewpoint during video streaming) involves a trade-off between storage and bandwidth costs. Several solutions have been proposed in the literature, using redundant predictive frames, Wyner-Ziv frames, or a combination of them. In this paper, we adopt distributed video coding for interactive multiview video plus depth (MVD), taking advantage of depth image based rendering (DIBR) and depth-aided inpainting to fill the occlusion areas. To the authors best knowledge, very few works in interactive MVD consider the problem of continuity of the playback during the switching among streams. Therefore we survey the existing solutions, we propose a set of techniques for MVD coding and we compare them. As main results, we observe that DIBR can help in rate reduction (up to 13.36% for the texture video and up to 8.67% for the depth map, wrt the case where DIBR is not used), and we also note that the optimal strategy to combine DIBR and distributed video coding depends on the position of the switching time into the group of pictures. Choosing the best technique on a frame-to-frame basis can further reduce the rate from 1% to 6%.

Depth-Based Multiview Distributed Video Coding

Multiview distributed video coding (DVC) has gained much attention in the last few years because of its potential in avoiding communication between cameras without decreasing the coding performance. However, the current results are not matching the expectations mainly due to the fact that some theoretical assumptions are not satisfied in the current implementations. For example, in distributed source coding the encoder must know the correlation between the sources, which cannot be achieved in the traditional DVC systems without having a communication between the cameras. In this work, we propose a novel multiview distributed video coding scheme in which the depth maps are used to estimate the way two views are correlated with no exchanges between the cameras. Only their relative positions are known. We design the complete scheme and further propose a rate allocation algorithm to efficiently share the bit budget between the different components of our scheme. Then, a rate allocation algorithm for depth maps is proposed in order to maximize the quality of synthesized virtual views. We show, through detailed experiments, that our scheme significantly outperforms the state-of-the-art DVC system.

Novel solutions for side information generation and fusion in multiview DVC

One of the key problems in distributed video coding is the generation of side information. This task consists of producing an estimate of an image with some neighboring ones, such as those taken by the same camera at different time instants, or, in the case of multiview setups, images taken at the same time instant by different cameras. If both estimates are available, a further problem arises, which is how to merge them in order to create a single side information. This problem is very relevant since a good estimate of the unknown image will require only a few bits to be corrected. Considering a multiview distributed video-coding setup, we propose a novel technique for inter-view interpolation based on occlusion prediction, a new fusion technique from multiple estimates, and finally an adaptive validation step for switching among the three possible side information images: temporal, inter-view, and fusion. We provide a comprehensive set of experimental results, which indicate bit rate reductions of more than 9% in average; moreover, we observe much more consistent results with respect to state-of-the-art techniques.

Wyner-ziv coding for depth maps in multiview video-plus-depth

Three dimensional digital video services are gathering a lot of attention in recent years, thanks to the introduction of new and efficient acquisition and rendering devices. In particular, 3D video is often represented by a single view and a so called depth map, which gives information about the distance between the point of view and the objects. This representation can be extended to multiple views, each with its own depth map. Efficient compression of this kind of data is of course a very important topic in sight of a massive deployment of services such as 3D-TV and FTV (free viewpoint TV). In this paper we consider the application of distributed coding techniques to the coding of depth maps, in order to reduce the complexity of single view or multi view encoders and to enhance interactive multiview video streaming. We start from state-of-the-art distributed video coding techniques and we improve them by using high order motion interpolation and by exploiting texture motion information to encode the depth maps. The experiments reported here show that the proposed method achieves a rate reduction up to 11.06% compared to state-of-the-art distributed video coding technique.

Side information refinement for long duration GOPs in DVC

Side information generation is a critical step in distributed video coding systems. This is performed by using motion compensated temporal interpolation between two or more key frames (KFs). However, when the temporal distance between key frames increases (i.e. when the GOP size becomes large), the linear interpolation becomes less effective. In a previous work we showed that this problem can be mitigated by using high order interpolation. Now, in the case of long duration GOP, state-of-the-art algorithms propose a hierarchical algorithm for side information generation. By using this procedure, the quality of the central interpolated image in a GOP is consistently worse than images closer to the KFs. In this paper we propose a refinement of the central WZFs by higher order interpolation of the already decoded WZFs, that are closer to the WZF to be estimated. So we reduce the fluctuation of side information quality, with a beneficial impact on final rate-distortion characteristics of the system. The experimental results show an improvement on the SI up to 2.71 dB with respect the state-of-the-art and a global improvement of the PSNR on the decoded frames up to 0.71 dB and a bit rate reduction up to 15 %.

An MDC-based video streaming architecture for mobile networks

Multiple description coding (MDC) is a framework designed to improve the robustness of video content transmission in lossy environments. In this work, we propose an MDC technique using a legacy coder to produce two descriptions, based on separation of even and odd frames. If only one description is received, the missing frames are reconstructed using temporal high-order motion interpolation (HOMI), a technique originally proposed for distributed video coding. If both descriptions are received, the frames are reconstructed as a block-wise linear combination of the two descriptions, with the coefficient computed at the encoder in a RD-optimised fashion, encoded with a context-adaptive arithmetic coder, and sent as side information. We integrated the proposed technique in a mobile ad-hoc streaming protocol, and tested it using a group mobility model. The results show a non-negligible gain for the expected video quality, with respect to the reference technique.

Reference View Selection in DIBR-Based Multiview Coding

Augmented reality, interactive navigation in 3D scenes, multiview video, and other emerging multimedia applications require large sets of images, hence larger data volumes and increased resources compared with traditional video services. The significant increase in the number of images in multiview systems leads to new challenging problems in data representation and data transmission to provide high quality of experience on resource-constrained environments. In order to reduce the size of the data, different multiview video compression strategies have been proposed recently. Most of them use the concept of reference or key views that are used to estimate other images when there is high correlation in the data set. In such coding schemes, the two following questions become fundamental: 1) how many reference views have to be chosen for keeping a good reconstruction quality under coding cost constraints? And 2) where to place these key views in the multiview data set? As these questions are largely overlooked in the literature, we study the reference view selection problem and propose an algorithm for the optimal selection of reference views in multiview coding systems. Based on a novel metric that measures the similarity between the views, we formulate an optimization problem for the positioning of the reference views, such that both the distortion of the view reconstruction and the coding rate cost are minimized. We solve this new problem with a shortest path algorithm that determines both the optimal number of reference views and their positions in the image set. We experimentally validate our solution in a practical multiview distributed coding system and in the standardized 3D-HEVC multiview coding scheme. We show that considering the 3D scene geometry in the reference view, positioning problem brings significant rate-distortion improvements and outperforms the traditional coding strategy that simply selects key frames based on the distance between cameras.

Side information improvement in transform-domain distributed video coding

Side Information (SI) has a strong impact on the rate-distortion performance in distributed video coding. The quality of the SI can be impaired when the temporal distance between the neighboring reference frames increases. In this paper, we introduce two novel methods that allow improving the quality of the SI. In the first approach, we propose a new estimation method for the initial SI using backward and forward motion estimation. The second one consists in re-estimating the SI after decoding all WZFs within the current Group of Pictures (GOP). For this purpose, the SI is first successively refined after each decoded DCT band. Then, after decoding all WZFs within the GOP, we adapt the search area to the motion content. Finally, each already decoded WZF is used, along with the neighboring ones, to estimate a new SI closer to the original WZF. This new SI is then used to reconstruct again the WZF with better quality. The experimental results show that, compared to the DISCOVER codec, the proposed method reaches an improvement of up to 3.53 dB in rate-distortion performance (measured with the Bjontegaard metric) for a GOP size of 8.

Key view selection in distributed multiview coding

Multiview image and video systems with large number of views lead to new problems in data representation, transmission and user interaction. In order to reduce the data volumes, most distributed multiview coding schemes exploit the inter-view redundancies at the decoder side, using view synthesis from key views. In the situation where many views are considered, the two following questions become fundamental: i) how many key views have to be chosen for keeping a good reconstruction quality with reasonable coding cost? ii) where to place them optimally in the multiview sequences? We propose in this paper an algorithm for selecting the key views in a distributed multiview coding scheme. Based on a novel metric for the correlation between the views, we formulate an optimization problem for the positioning of the key views such that both the distortion of the reconstruction and the coding rate cost are effectively minimized. We then propose a new optimization strategy based on shortest path algorithm that permits to determine both the optimal number of key views and their positions in the image set. We experimentally validate our solution in a practical distributed multiview coding system and we show that considering the 3D scene geometry in the key view positioning brings significant rate-distortion improvements compared to distance-based key view selection as it is commonly done in the literature.