Masaki Kitahara

Multiview Video Coding Using View Interpolation and Color Correction

Neighboring views must be highly correlated in multiview video systems. We should therefore use various neighboring views to efficiently compress videos. There are many approaches to doing this. However, most of these treat pictures of other views in the same way as they treat pictures of the current view, i.e., pictures of other views are used as reference pictures (inter-view prediction). We introduce two approaches to improving compression efficiency in this paper. The first is by synthesizing pictures at a given time and a given position by using view interpolation and using them as reference pictures (view-interpolation prediction). In other words, we tried to compensate for geometry to obtain precise predictions. The second approach is to correct the luminance and chrominance of other views by using lookup tables to compensate for photoelectric variations in individual cameras. We implemented these ideas in H.264/AVC with inter-view prediction and confirmed that they worked well. The experimental results revealed that these ideas can reduce the number of generated bits by approximately 15% without loss of PSNR.

View Scalable Multiview Video Coding Using 3-D Warping With Depth Map

Multiview video coding demands high compression rates as well as view scalability, which enables the video to be displayed on a multitude of different terminals. In order to achieve view scalability, it is necessary to limit the inter-view prediction structure. In this paper, we propose a new multiview video coding scheme that can improve the compression efficiency under such a limited inter-view prediction structure. All views are divided into two groups in the proposed scheme: base view and enhancement views. The proposed scheme first estimates a view-dependent geometry of the base view. It then uses a video encoder to encode the video of base view. The view-dependent geometry is also encoded by the video encoder. The scheme then generates prediction images of enhancement views from the decoded video and the view-dependent geometry by using image-based rendering techniques, and it makes residual signals for each enhancement view. Finally, it encodes residual signals by the conventional video encoder as if they were regular video signals. We implement one encoder that employs this scheme by using a depth map as the view-dependent geometry and 3-D warping as the view generation method. In order to increase the coding efficiency, we adopt the following three modifications: (1) object-based interpolation on 3-D warping; (2) depth estimation with consideration of rate-distortion costs; and (3) quarter-pel accuracy depth representation. Experiments show that the proposed scheme offers about 30% higher compression efficiency than the conventional scheme, even though one depth map video is added to the original multiview video.

System design of free viewpoint video communication

We propose a free viewpoint video communication method that allows the user to change his/her viewing point and viewing direction freely. This application will be a next generation visual application in the near future. We propose suitable multiple view video coding method and communication protocol for this communication application, and demonstrate the coding efficiency of the proposed method. We also demonstrate that the developed viewer can generate the view of arbitrary viewpoint and view direction. This viewer uses the Ray-Space interpolation and extrapolation methods and generates a natural view.

Multi-View Video Coding using View Interpolation and Reference Picture Selection

We propose a new multi-view video coding method using adaptive selection of motion/disparity compensation based on H.264/AVC. One of the key points of the proposed method is the use of view interpolation as a tool for disparity compensation by assigning reference picture indices to interpolated images. Experimental results show that significant gains can be obtained compared to the conventional approach that was often used

Hierarchical reference picture selection method for temporal scalability beyond H.264

Temporal scalability is effective to adapt the bitstream adaptation for various capabilities of the video terminals and the delivery network, e.g. the processing speed of video terminals and the transmission rate, respectively. This technique uses the reference picture selection method on both the current layer and lower layers. The conventional scalability selects only from the last previous picture of the current layer and that of the first lower layer. This paper proposes a novel prediction scheme, the HRPS (hierarchical reference picture selection) method in which the reference picture is selected from more previous pictures in more layers, in order to improve coding efficiency, keeping the temporal scalability functionality. The proposed method is developed with a modification of the H.264. This paper demonstrates the effectiveness of the HRPS compared with the conventional temporal scalable methods.

3D motion vector coding with block base adaptive interpolation filter on H.264

Fractional pel motion compensation generally improves coding efficiency due to more precise motion accuracy and low path filtering effect in generating an image at fractional pel positions. In H.264, quarter pel motion compensation is applied, where the image at half pel position is generated by a 6 tap Wiener filter. And the adaptive interpolation filter technique, which adaptively changes filter characteristics for half pel positions has been proposed. That technique also changes the image at quarter pel positions, so it can be exploited to extend motion accuracy to be more precise. In this paper, a 3D motion vector coding (3DMVC) technique with block base adaptive interpolation filter (BAIF) is proposed. This paper also demonstrates the proposed method ensures filter data is successfully integrated into motion vector coding and outperforms the normal H.264.

Hardware accelerated image-based rendering with compressed surface light fields and multiresolution geometry

Surface Light Fields(SLF) are light fields parameterized to the surface of a geometric model. Past research in this area has explored the use of parameterization of aquired geometric model to the base mesh, in the context of rendering from compressed representation of the SLF. We further extend this aproach to enable hardware-accelerated SLF rendering. We show that our aproach enables hardware-accelerated, scalable rendering with commodity graphics hardware.

Recursively weighting pixel domain intra prediction on H.264

Intra coding for lossy block base transform video coding and still picture coding has been studied. In H.264, pixel domain prediction is applied, where all pixel values in a block are predicted from decoded images in surrounding blocks. There are some advantages in pixel domain prediction comparing with DCT domain prediction. One thing is that in pixel domain prediction, residual data at block boundaries becomes smaller. On the other hand, in pixel base prediction scheme for lossless coding, each pixel value is predicted from surrounding pixels generally. In Multiplicative Autoregressive Models (MAR) or JPEG-LS, each pixel is predicted from some neighboring pixels. This pixel base prediction scheme is more effective to reduce prediction error than block base prediction. In this paper, the new intra prediction method, Recursively Weighting pixel domain Intra Prediction (RWIP) method for block base transform coding is proposed. The RWIP applies similar approach to pixel base prediction scheme in order to reduce prediction error more than the conventional block base prediction scheme, especially for blur or complicated directional edge images. This paper also demonstrates the efficiency of the RWIP over the normal intra prediction of H.264.

Progressive Coding of Surface Light Fields for Efficient Image Based Rendering

Surface light field is a light field parameterized to the surface of a geometric model. Although this multiview object representation enables rendering of objects from arbitrary views, the data size of a surface light field is huge. In this paper, a compression scheme is proposed. Together with compression efficiency, scalability and fast, memory efficient rendering is considered. To enable such functionalities, we base our compression scheme on a factorized representation of surface light field, and utilize a parameterization of the geometric model to a base mesh for application of wavelet transforms for progressive compression and scalable rendering. Experiments were conducted on a real world and synthetic data to study the compression efficiency and the characteristics of our proposed scheme.

Bit position quantization in scalable video coding for representing detail of image

We have studied a coding method to provide a different type of distortion from in the images of the base layer using a scalable coding scheme. Conventionally for the enhancement layer in the scalable video coding scheme, quantization in transformed domain on the residual coding reduces much high frequency information of images. The decoded images are basically blurred, i.e. it lacks detail information. This paper proposes a novel residual coding method to add detail information upon the decoded images of the base layer. The proposed method uses the BPQ, which is one of non-linear scalar quantization methods and which represents more efficiently the feature of detail information than a quantization method in DCT domain. And the proposed method uses the appropriate entropy coding method for the q uantized values. The proposed entropy coding uses the lossless coding method extended f rom the H.264/AVC.