Takafumi Koike

TransCAIP: A Live 3D TV System Using a Camera Array and an Integral Photography Display with Interactive Control of Viewing Parameters

The system described in this paper provides a real-time 3D visual experience by using an array of 64 video cameras and an integral photography display with 60 viewing directions. The live 3D scene in front of the camera array is reproduced by the full-color, full-parallax autostereoscopic display with interactive control of viewing parameters. The main technical challenge is fast and flexible conversion of the data from the 64 multicamera images to the integral photography format. Based on image-based rendering techniques, our conversion method first renders 60 novel images corresponding to the viewing directions of the display, and then arranges the rendered pixels to produce an integral photography image. For real-time processing on a single PC, all the conversion processes are implemented on a GPU with GPGPU techniques. The conversion method also allows a user to interactively control viewing parameters of the displayed image for reproducing the dynamic 3D scene with desirable parameters. This control is performed as a software process, without reconfiguring the hardware system, by changing the rendering parameters such as the convergence point of the rendering cameras and the interval between the viewpoints of the rendering cameras.

Seam carving for stereo images

We propose a content-preserving stereo image editing technique by using stereo extension of seam carving. Seam carving is the process of deleting or duplicating connected paths, or seams, that consist of less important pixels in order to resize the image while preserving the image content as much as possible. The problem targeted in this paper is how to apply the seam carving method to a pair of stereo images, where the consistency between the left and right images should also be preserved through the seam carving process. For this consistency, we introduce new energy terms. Based on these energy terms, seam pairs between the input images are classified into two types, corresponding seams in order to maintain consistency or occluded seams to change the consistency intentionally. The novelty of this paper is that stereo matching results are fused into the framework in the seam carving. We demonstrate that by selecting seams in an appropriate way we can virtually manipulate the depths of objects in the scene.

41.2: Autostereoscopic Display Based on Enhanced Integral Photography Using Overlaid Multiple Projectors

We constructed an autostereoscopic display based on enhanced IP using overlaid multiple projectors. The display provides autostereoscopic images whose resolution is greater than SVGA in an 800 by 400 mm area. Tilting the display surface increases the depth of the autostereoscopic images. We also constructed equipment for calibrating the display.

Full-parallax autostereoscopic display with scalable lateral resolution using overlaid multiple projection

— A method to increase the viewing resolution of an autostereoscopic display without increasing the density of microlenses is proposed. Multiple projectors are used for the projection images to be focused and overlaid on a common plane in the air behind the microlens array. The multiple overlaid projection images yield multiple light spots inside the region of each elemental lenslet of the microlens array. This feature provides scalable high-resolution images by increasing the number of projectors. Based on the proposed method, a prototype display that includes 15 projectors was designed and built. 3-D images were successfully reproduced on the prototype display with full parallax and a wide viewing angle of 70°.

BRDF display: interactive view dependent texture display using integral photography

We developed a bidirectional reflectance distribution function (BRDF) display that reproduces the texture of objects and rays of a light field. Our BRDF display is based on the technology used in integral photography (IP), which was originally investigated for use in 3D displays. An IP display is an ideal method for reproducing a light field in real space. The proposed BRDF display consists of a high-definition panel and a micro lens array, which is the same as in an IP display. The main difference to an IP display is that several optical parameters are optimized for BRDF use. We also describe content creation methods and show some content created using several of these methods.

A video-based virtual reality system

We introduce a new environment to make and play interactive contents with more than video game quality. The system consists of a projector array, a viewer and an editor for the special contents. The projector array projects multiple digital images seamlessly both in time and space, thus a very high quality video projection system. The viewer features a function to composite a passive video and interactive CG in real time. The editor is a high-end non-linear editing system combined with some new plug-in software to pre-compute the information necessary for real-time compositing. A new method of digital image recognition assisted by human operators is used. Unlike general purpose computer vision algorithms, it minimizes the error of 3D estimation at the compositing position. We call this approach V2R or the video-based virtual reality. It allows the operator to experience interactive communications with objects in a very high quality video.

Influence of 3‐D cross‐talk on qualified viewing spaces in two‐ and multi‐view autostereoscopic displays

— To estimate the qualified viewing spaces for two- and multi-view autostereoscopic displays, the relationship between image quality (image comfort, annoying ghost image, depth perception) and various pairings between 3-D cross-talk in the left and right views are studied subjectively using a two-view autostereoscopic display and test charts for the left and right views with ghost images due to artificial 3-D cross-talk. The artificial 3-D cross-talk was tuned to simulate the view in the intermediate zone of the viewing spaces. It was shown that the stereoscopic images on a two-view autostereoscopic display cause discomfort when they are observed by the eye in the intermediate zone between the viewing spaces. This is because the ghost image due to large 3-D cross-talk in the intermediate zone elicits different depth perception from the depth induced by the original images for the left and right views, so the observers depth perception is confused. Image comfort is also shown to be better for multi-views, especially the width of the viewing space, which is narrower than the interpupillary distance, where the parallax of the cross-talking image is small.

Proportional constraint for seam carving

Seam carving is an image processing operator for content-aware image resizing [Avidan and Shamir 2007]. It generates an energy map from gradient intensity of pixels and searches for seams, which are vertical or horizontal continuous paths of pixels that run through local minimum energy areas. Removing or inserting pixels along a seam enables users to shrink or enlarge pictures by a wide range, while still retaining all details of the image.

TransCAIP: live transmission of light field from a camera array to an integral photography display

TransCAIP provides a real-time 3D visual experience by using an array of 64 cameras and an integral photography display with 60 viewing directions. The live 3D scene in front of the camera array is reproduced by the full-colour, full-parallax auto-stereoscopic display with interactive control of viewing parameters.

High-density light field reproduction using overlaid multiple projection images

We propose a method to increase the viewing resolution of an autostereoscopic display without increasing the density of microlenses. Multiple projectors are used for the projection images to be focused and overlaid on a common plane in the air behind the microlens array. The multiple overlaid projection images yield multiple light spots inside the region of each elemental lenslet of the microlens array. This feature provides scalable high resolution images by increasing the number of projectors.