Norishige Fukushima

View generation with 3D warping using depth information for FTV

Free viewpoint images can be generated from multi-view images using Ray-Space method. Ray-Space data requires ray interpolation so as to satisfy the plenoptic function. Ray interpolation is realized by estimating view-dependent depth. Depth estimation is usually costly process, thus it is desirable that this process is skipped from rendering process to achieve real-time rendering. This paper proposes a method to render a novel view image using multi-view images and depth maps which are computed in advance. Virtual viewpoint image is generated by 3D warping, which causes some problems that have not occurred in the method with view dependent depth estimation. We handled these problems by projecting depth map to virtual image plane first and perform post-filtering on the projected depth map. We succeeded in obtaining high quality arbitrary viewpoint images from relatively small number of cameras.

View Generation with 3D Warping Using Depth Information for FTV

Free viewpoint images can be generated from multi-view images using Ray-Space method. Ray-Space data requires ray interpolation so as to satisfy the plenoptic function. Ray interpolation is realized by estimating view-dependent depth. Depth estimation is usually costly process, thus it is desirable that this process is skipped from rendering process to achieve real-time rendering. This paper proposes a method to render a novel view image using multi-view images and depth maps which are computed in advance. Virtual viewpoint image is generated by 3D warping, which causes some problems that have not occurred in the method with view dependent depth estimation. We handled these problems by projecting depth map to virtual image plane first and perform post-filtering on the projected depth map. We succeeded in obtaining high quality arbitrary viewpoint images from relatively small number of cameras.

Free viewpoint image generation using multi-pass dynamic programming

Ray-Space is categorized by Image-Based Rendering (IBR), thus generated views have photo-realistic quality. While this method has the performance of high quality imaging, this needs a lot of images or cameras. The reason why that is Ray-Space requires various directions and positions views instead of 3D depth information. In this paper, we reduce that flood of information using view-centered ray interpolation. View-centered interpolation means estimating view dependent depth value (or disparity map) at generating view-point and interpolating that of pixel values using multi-view images and depth information. The combination of depth estimation and interpolation realizes the rendering photo-realistic images effectively. Unfortunately, however, if depth estimation is week or mistake, a lot of artifacts appear in creating images. Thus powerful depth estimation method is required. When we render the free viewpoint images video, we perform the depth estimation at every frame. Thus we want to keep a lid on computing cost. Our depth estimation method is based on dynamic programming (DP). This method optimizes and solves depth images at the weak matching area with high-speed performance. But scan-line noises become appeared because of the limit of DP. So, we perform the DP multi-direction pass and sum-up the result of multi-passed DPs. Our method fulfills the low computation cost and high depth estimation performance.

QoE assessment in olfactory and haptic media transmission: Influence of inter-stream synchronization error

In this paper, we deal with harvesting fruit in a 3-D virtual space for an olfactory and haptic media display system. Making use of the system, a user can pick fruit from a tree in the 3-D virtual space and perceive the smell of picked fruit by using an olfactory display (SyP@D2) and the reaction force when picking fruit by using a haptic interface device (PHANToM Omni). In the case where there exists inter-stream synchronization error between olfactory and haptic media, we assess the inter-stream synchronization quality as the quality of experience (QoE).

A semi-automatic multi-view depth estimation method

In this paper, we propose a semi-automatic depth estimation algorithm whereby the user defines object depth boundaries and disparity initialization. Automatic depth estimation methods generally have difficulty to obtain good depth results around object edges and in areas with low texture. The goal of our method is to improve the depth in these areas and reduce view synthesis artifacts in Depth Image Based Rendering. Good view synthesis quality is very important in applications such as 3DTV and Free-viewpoint Television (FTV). In our proposed method, initial disparity values for smooth areas can be input through a so-called manual disparity map, and depth boundaries are defined by a manually created edge map which can be supplied for one or multiple frames. For evaluation we used MPEG multi-view videos and we demonstrate our algorithm can significantly improve the depth maps and reduce view synthesis artifacts.

QoE assessment in haptic media, sound, and video transmission: Effect of playout buffering control

In this article we investigate the effect of playout buffering control in haptic media, sound, and video transmission via QoE (Quality of Experience) assessment. We assess the operability of the haptic interface device, sound output quality, video output quality, inter-stream synchronization quality, and comprehensive quality. We also evaluate the application-level QoS (Quality of Service), and demonstrate that, with high accuracy, it is possible to estimate QoE parameters from QoS parameters.

QoE assessment in haptic media, sound and video transmission: Influences of network latency

In this paper, we investigate the influences of the difference in network latency among haptic media, sound and video on the quality of experience (QoE) for a haptic media, sound and video transfer system. In the case where network delay jitter exists, we subjectively assess the operability of haptic interface device, sound output quality, video output quality, inter-stream synchronization quality, and comprehensive quality as QoE. We also evaluate the application-level quality of service (QoS), and we demonstrate that it is possible to estimate QoE parameters from QoS parameters with high accuracy.

QoE assessment in networked air hockey game with haptic media

This paper assesses QoE (Quality of Experience) of output quality and interactivity in a networked air hockey game with haptic media. For consistency and causality, we employ the adaptive Δ-causality control scheme with adaptive dead-reckoning. We clarify the influence of network delay on QoE.

Removing depth map coding distortion by using post filter set

We propose a fast post filter set for removing coding distortions of depth maps. Lossy coding generates various distortions, such as mosquito and block noises, edge blurs, and over quantization. These distortions seriously deteriorate image quality of synthesized views in free viewpoint image rendering. Thus, we propose the post filter set which includes median filter, Gaussian filter, min-max blur remove filer, and binary weighted range filter to remove these noises. In experiments, we use various codecs, which are JPEG, JPEG-LS, JPEG2000, and H.264/AVC, for depth map coding, and synthesize views with the coded depth maps. Experimental results show that our post filter set can improves every codecs performance. Improvement of PSNR is larger than the conventional post filter, and especially JPEG is large. The computational time of the filter set, which is implemented by C++ with SIMD optimization, is within 5.2 ms at high bit rate cases, and within 15.3 ms at low bit rate cases.

QoE assessment of olfactory media in remote ikebana with haptic media

In this paper, we handle a remote ikebana (i.e., flower arrangement) system with olfactory and haptic media. In the system, a teacher or a student can hold a flower, adjust the length of the held flowers stem with a pair of scissors, and impale the flower on a flower pinholder in a 3-D virtual space. We make QoE (Quality of Experience) assessment of olfactory media to investigate the influence of the output timing of olfactory media by changing the size of smell space (defined as a sphere in which we can perceive the smell of flower). In the assessment, each subject is asked to move a rose toward or away from his/her viewpoint, and he/she judges how good the output timing of olfactory media is. We illustrate that there exists the optimum value of the smell space size, and the optimum value depends on the average movement speed and the direction of movement.