Shohei Nobuhara

Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video

3D video [IEEE Multimedia (1997) 18] is the ultimate image media recording dynamic visual events in the real world as is; it records time varying 3D object shape with high fidelity surface properties (i.e., color and texture), Its applications cover wide varieties of personal and social human activities: entertainment (e.g., 3D game and 3D TV), education (e.g., 3D animal picture books), sports (e.g., sport performance analysis), medicine (e.g., 3D surgery monitoring), culture (e.g., 3D archive of traditional dances), and so on. In this paper, we propose: (1) a PC cluster system for real-time reconstruction of dynamic 3D object action from multiview video images, (2) a deformable 3D mesh model for reconstructing the accurate dynamic 3D object shape, and (3) an algorithm of rendering natural-looking texture on the 3D object surface from the multi-view video images. Experimental results with quantitative performance evaluations demonstrate the effectiveness of these methods in generating high fidelity 3D video from multi-view video images.

The Multiple-Camera 3-D Production Studio

Multiple-camera systems are currently widely used in research and development as a means of capturing and synthesizing realistic 3-D video content. Studio systems for 3-D production of human performance are reviewed from the literature, and the practical experience gained in developing prototype studios is reported across two research laboratories. System design should consider the studio backdrop for foreground matting, lighting for ambient illumination, camera acquisition hardware, the camera configuration for scene capture, and accurate geometric and photometric camera calibration. A ground-truth evaluation is performed to quantify the effect of different constraints on the multiple-camera system in terms of geometric accuracy and the requirement for high-quality view synthesis. As changing camera height has only a limited influence on surface visibility, multiple-camera sets or an active vision system may be required for wide area capture, and accurate reconstruction requires a camera baseline of 25deg, and the achievable accuracy is 5-10-mm at current camera resolutions. Accuracy is inherently limited, and view-dependent rendering is required for view synthesis with sub-pixel accuracy where display resolutions match camera resolutions. The two prototype studios are contrasted and state-of-the-art techniques for 3-D content production demonstrated.

Simultaneous super-resolution and 3D video using graph-cuts

This paper presents a new method to increase the quality of 3D video, a new media developed to represent 3D objects in motion. This representation is obtained from multi-view reconstruction techniques that require images recorded simultaneously by several video cameras. All cameras are calibrated and placed around a dedicated studio to fully surround the models. The limited quality and quantity of cameras may produce inaccurate 3D model reconstruction with low quality texture. To overcome this issue, first we propose super-resolution (SR) techniques for 3D video: SR on multi-view images and SR on single-view video frames. Second, we propose to combine both super-resolution and dynamic 3D shape reconstruction problems into a unique Markov random field (MRF) energy formulation. The MRF minimization is performed using graph-cuts. Thus, we jointly compute the optimal solution for super-resolved texture and 3D shape model reconstruction. Moreover, we propose a coarse-to-fine strategy to iteratively produce 3D video with increasing quality. Our experiments show the accuracy and robustness of the proposed technique on challenging 3D video sequences.

A new mirror-based extrinsic camera calibration using an orthogonality constraint

This paper is aimed at calibrating the relative posture and position, i.e. extrinsic parameters, of a stationary camera against a 3D reference object which is not directly visible from the camera. We capture the reference object via a mirror under three different unknown poses, and then calibrate the extrinsic parameters from 2D appearances of reflections of the reference object in the mirrors. The key contribution of this paper is to present a new algorithm which returns a unique solution of three P3P problems from three mirrored images. While each P3P problem has up to four solutions and therefore a set of three P3P problems has up to 64 solutions, our method can select a solution based on an orthogonality constraint which should be satisfied by all families of reflections of a single reference object. In addition we propose a new scheme to compute the extrinsic parameters by solving a large system of linear equations. These two points enable us to provide a unique and robust solution. We demonstrate the advantages of the proposed method against a state-of-the-art by qualitative and quantitative evaluations using synthesized and real data.

3D Video and Its Applications

This book presents a broad review of state-of-the-art 3D video production technologies and applications. The text opens with a concise introduction to the field, before examining the design and calibration methods for multi-view camera systems, including practical implementation technologies. A range of algorithms are then described for producing 3D video from video data. A selection of 3D video applications are also demonstrated. Features: describes real-time synchronized multi-view video capture, and object tracking with a group of active cameras; discusses geometric and photometric camera calibration, and 3D video studio design with active cameras; examines 3D shape and motion reconstruction, texture mapping and image rendering, and lighting environment estimation; demonstrates attractive 3D visualization, visual contents analysis and editing, 3D body action analysis, and data compression; highlights the remaining challenges and the exciting avenues for future research in 3D video technology.

Complete multi-view reconstruction of dynamic scenes from probabilistic fusion of narrow and wide baseline stereo

This paper presents a novel approach to achieve accurate and complete multi-view reconstruction of dynamic scenes (or 3D videos). 3D videos consist in sequences of 3D models in motion captured by a surrounding set of video cameras. To date 3D videos are reconstructed using multiview wide baseline stereo (MVS) reconstruction techniques. However it is still tedious to solve stereo correspondence problems: reconstruction accuracy falls when stereo photo-consistency is weak, and completeness is limited by self-occlusions. Most MVS techniques were indeed designed to deal with static objects in a controlled environment and therefore cannot solve these issues. Hence we propose to take advantage of the image content stability provided by each single-view video to recover any surface regions visible by at least one camera. In particular we present an original probabilistic framework to derive and predict the true surface of models. We propose to fuse multi-view structure-from-motion with robust 3D features obtained by MVS in order to significantly improve reconstruction completeness and accuracy. A min-cut problem where all exact features serve as priors is solved in a final step to reconstruct the 3D models. In addition, experimental results were conducted on synthetic and challenging real world datasets to illustrate the robustness and accuracy of our method.

Panoptic Studio: A Massively Multiview System for Social Motion Capture

We present an approach to capture the 3D structure and motion of a group of people engaged in a social interaction. The core challenges in capturing social interactions are: (1) occlusion is functional and frequent, (2) subtle motion needs to be measured over a space large enough to host a social group, and (3) human appearance and configuration variation is immense. The Panoptic Studio is a system organized around the thesis that social interactions should be measured through the perceptual integration of a large variety of view points. We present a modularized system designed around this principle, consisting of integrated structural, hardware, and software innovations. The system takes, as input, 480 synchronized video streams of multiple people engaged in social activities, and produces, as output, the labeled time-varying 3D structure of anatomical landmarks on individuals in the space. The algorithmic contributions include a hierarchical approach for generating skeletal trajectory proposals, and an optimization framework for skeletal reconstruction with trajectory re-association.

Dynamic 3D shape from multi-viewpoint images using deformable mesh model

This paper presents a framework for dynamic 3D shape reconstruction from multi-viewpoint images using a deformable mesh model. With our method, we can obtain 3D shape and 3D motion of the object simultaneously. We represent the shape by a surface mesh model and the motion by translations of its vertices, i.e., deformation. Thus global and local topological structure of the mesh are preserved from frame to frame. This helps us to analyse the motion of the object, to compress the 3D data, and so on. Our model deforms its shape so as to satisfy several constraints. This constraint-based deformation provides a computational framework to integrate several reconstruction cues such as surface texture, silhouette, and motion flow observed in multi-viewpoint images.

Heterogeneous deformation model for 3D shape and motion recovery from multi-viewpoint images

This work presents a framework for dynamic 3D shape and motion reconstruction from multiviewpoint images using a deformable mesh model. By deforming a mesh at a frame to that at the next frame, we can obtain both 3D shape and motion of the object simultaneously. The deformation process of our mesh model is heterogeneous. Each vertex changes its deformation process according to its 1) photometric property (i.e., if it has prominent texture or not), and 2) physical property (i.e., if it is an element of rigid part of the object or not). This heterogeneous deformation model enables us to reconstruct the object which consists of different kinds of materials or parts with different motion models, e.g., rigidly acting body parts and deforming soft clothes or its skins, by a single and unified computational framework.

A Pixel-Wise Varifocal Camera Model for Efficient Forward Projection and Linear Extrinsic Calibration of Underwater Cameras with Flat Housings

This paper is aimed at presenting a new virtual camera model which can efficiently model refraction through flat housings in underwater photography. The key idea is to employ a pixel-wise virtual focal length concept to encode the refractive projection inside the flat housing. The radially-symmetric structure of the varifocal length around the normal of the housing surface allows us to encode the refractive projection with a compact representation. We show that this model realizes an efficient forward projection computation and a linear extrinsic calibration in water. Evaluations using synthesized and real data demonstrate the performance quantitatively and qualitatively.