Hansung Kim

The i3DPost Multi-View and 3D Human Action/Interaction Database

In this paper a new multi-view/3D human action/interaction database is presented. The database has been created using a convergent eight camera setup to produce high definition multi-view videos, where each video depicts one of eight persons performing one of twelve different human motions. Various types of motions have been recorded, i.e., scenes where one person performs a specific movement, scenes where a person executes different movements in a succession and scenes where two persons interact with each other. Moreover, the subjects have different body sizes, clothing and are of different sex, nationalities, etc.. The multi-view videos have been further processed to produce a 3D mesh at each frame describing the respective 3D human body surface. To increase the applicability of the database, for each person a multi-view video depicting the person performing sequentially the six basic facial expressions separated by the neutral expression has also been recorded. The database is freely available for research purposes.

3D Scene Reconstruction from Multiple Spherical Stereo Pairs

We propose a 3D environment modelling method using multiple pairs of high-resolution spherical images. Spherical images of a scene are captured using a rotating line scan camera. Reconstruction is based on stereo image pairs with a vertical displacement between camera views. A 3D mesh model for each pair of spherical images is reconstructed by stereo matching. For accurate surface reconstruction, we propose a PDE-based disparity estimation method which produces continuous depth fields with sharp depth discontinuities even in occluded and highly textured regions. A full environment model is constructed by fusion of partial reconstruction from spherical stereo pairs at multiple widely spaced locations. To avoid camera calibration steps for all camera locations, we calculate 3D rigid transforms between capture points using feature matching and register all meshes into a unified coordinate system. Finally a complete 3D model of the environment is generated by selecting the most reliable observations among overlapped surface measurements considering surface visibility, orientation and distance from the camera. We analyse the characteristics and behaviour of errors for spherical stereo imaging. Performance of the proposed algorithm is evaluated against ground-truth from the Middlebury stereo test bed and LIDAR scans. Results are also compared with conventional structure-from-motion algorithms. The final composite model is rendered from a wide range of viewpoints with high quality textures.

Influence of Colour and Feature Geometry on Multi-modal 3D Point Clouds Data Registration

With the current transition of various digital contents from 2D to 3D, the problem of 3D data matching and registration is increasingly important. Registration of multi-modal 3D data acquired from different sensors remains a challenging problem due to the difference in types and characteristics of the data. In this paper, we evaluate the registration performance of 3D feature descriptors with different domains on datasets from various environments and modalities. Datasets are acquired in indoor and outdoor environments with 2D and 3D sensing devices including LIDAR, spherical imaging, digital camera and RGBD camera. FPFH, PFH and SHOT feature descriptors are applied to the 3D point clouds generated from the multi-modal datasets. Local neighbouring point distribution, key points distribution, colour information and their combinations are used for feature description. Finally we analyse their influences on the multi-modal 3D point clouds data registration.

Robust foreground extraction technique using Gaussian family model and multiple thresholds

We propose a robust method to extract silhouettes of foreground objects from color video sequences. To cope with various changes in the background, the background is modeled as generalized Gaussian Family of distributions and updated by the selective running average and static pixel observation. All pixels in the input video image are classified into four initial regions using background subtraction with multiple thresholds, after which shadow regions are eliminated using color components. The final foreground silhouette is extracted by refining the initial region using morphological processes. We have verified that the proposed algorithm works very well in various background and foreground situations through experiments.

General Dynamic Scene Reconstruction from Multiple View Video

This paper introduces a general approach to dynamic scene reconstruction from multiple moving cameras without prior knowledge or limiting constraints on the scene structure, appearance, or illumination. Existing techniques or dynamic scene reconstruction from multiple wide-baseline camera views primarily focus on accurate reconstruction in controlled environments, where the cameras are fixed and calibrated and background is known. These approaches are not robust for general dynamic scenes captured with sparse moving cameras. Previous approaches for outdoor dynamic scene reconstruction assume prior knowledge of the static background appearance and structure. The primary contributions of this paper are twofold: an automatic method for initial coarse dynamic scene segmentation and reconstruction without prior knowledge of background appearance or structure, and a general robust approach for joint segmentation refinement and dense reconstruction of dynamic scenes from multiple wide-baseline static or moving cameras. Evaluation is performed on a variety of indoor and outdoor scenes with cluttered backgrounds and multiple dynamic non-rigid objects such as people. Comparison with state-of-the-art approaches demonstrates improved accuracy in both multiple view segmentation and dense reconstruction. The proposed approach also eliminates the requirement for prior knowledge of scene structure and appearance.

Evaluation of 3D Feature Descriptors for Multi-modal Data Registration

We propose a framework for 2D/3D multi-modal data registration and evaluate 3D feature descriptors for registration of 3D datasets from different sources. 3D datasets of outdoor environments can be acquired using a variety of active and passive sensor technologies. Registration of these datasets into a common coordinate frame is required for subsequent modelling and visualisation. 2D images are converted into 3D structure by stereo or multiview reconstruction techniques and registered to a unified 3D domain with other datasets in a 3D world. Multi-modal datasets have different density, noise, and types of errors in geometry. This paper provides a performance benchmark for existing 3D feature descriptors across multi-modal datasets. This analysis highlights the limitations of existing 3D feature detectors and descriptors which need to be addressed for robust multi-modal data registration. We analyse and discuss the performance of existing methods in registering various types of datasets then identify future directions required to achieve robust multi-modal data registration.

Robust foreground extraction technique using background subtraction with multiple thresholds

We propose a robust method to extract silhouettes of foreground objects from color-video sequences. To cope with various changes in the background, we model the background as a Laplace distribution and update it with a selective running average and static pixel observation. All pixels in the input video image are classified into four initial regions using background subtraction with multiple thresholds. Shadow regions are eliminated using color components, and the final foreground silhouette is extracted by smoothing the boundaries of the foreground and eliminating errors inside and outside of the regions. Experimental results show that the proposed algorithm works very well in various background and foreground situations.

Outdoor Dynamic 3-D Scene Reconstruction

Existing systems for 3-D reconstruction from multiple view video use controlled indoor environments with uniform illumination and backgrounds to allow accurate segmentation of dynamic foreground objects. In this paper, we present a portable system for 3-D reconstruction of dynamic outdoor scenes that require relatively large capture volumes with complex backgrounds and nonuniform illumination. This is motivated by the demand for 3-D reconstruction of natural outdoor scenes to support film and broadcast production. Limitations of existing multiple view 3-D reconstruction techniques for use in outdoor scenes are identified. Outdoor 3-D scene reconstruction is performed in three stages: 1) 3-D background scene modeling using spherical stereo image capture; 2) multiple view segmentation of dynamic foreground objects by simultaneous video matting across multiple views; and 3) robust 3-D foreground reconstruction and multiple view segmentation refinement in the presence of segmentation and calibration errors. Evaluation is performed on several outdoor productions with complex dynamic scenes including people and animals. Results demonstrate that the proposed approach overcomes limitations of previous indoor multiple view reconstruction approaches enabling high-quality free-viewpoint rendering and 3-D reference models for production.

Temporally Coherent 4D Reconstruction of Complex Dynamic Scenes

This paper presents an approach for reconstruction of 4D temporally coherent models of complex dynamic scenes. No prior knowledge is required of scene structure or camera calibration allowing reconstruction from multiple moving cameras. Sparse-to-dense temporal correspondence is integrated with joint multi-view segmentation and reconstruction to obtain a complete 4D representation of static and dynamic objects. Temporal coherence is exploited to overcome visual ambiguities resulting in improved reconstruction of complex scenes. Robust joint segmentation and reconstruction of dynamic objects is achieved by introducing a geodesic star convexity constraint. Comparative evaluation is performed on a variety of unstructured indoor and outdoor dynamic scenes with hand-held cameras and multiple people. This demonstrates reconstruction of complete temporally coherent 4D scene models with improved nonrigid object segmentation and shape reconstruction.

Compensated Visual Hull for Defective Segmentation and Occlusion

We propose an advanced visual hull technique to compensate for outliers using the reliabilities of the silhouettes. The proposed method consists of a foreground extraction technique based on the generalized Gaussian family model and a compensated shape-from-silhouette algorithm. They are connected by the intra-/inter-silhouette reliabilities to compensate for carving errors from defective segmentation or partial occlusion which may occur in a real environment. The 3D reconstruction process is implemented on a graphics processing unit (GPU) to accelerate the processing speed by using the huge computational power of modern graphics hardware. Experimental results show that the proposed method provides reliable silhouette information and an accurate visual hull in real environments at a very high speed on a common PC.