Dimitrios Zarpalas

Real-Time, Full 3-D Reconstruction of Moving Foreground Objects From Multiple Consumer Depth Cameras

The problem of robust, realistic and especially fast 3-D reconstruction of objects, although extensively studied, is still a challenging research task. Most of the state-of-the-art approaches that target real-time applications, such as immersive reality, address mainly the problem of synthesizing intermediate views for given view-points, rather than generating a single complete 3-D surface. In this paper, we present a multiple-Kinect capturing system and a novel methodology for the creation of accurate, realistic, full 3-D reconstructions of moving foreground objects, e.g., humans, to be exploited in real-time applications. The proposed method generates multiple textured meshes from multiple RGB-Depth streams, applies a coarse-to-fine registration algorithm and finally merges the separate meshes into a single 3-D surface. Although the Kinect sensor has attracted the attention of many researchers and home enthusiasts and has already appeared in many applications over the Internet, none of the already presented works can produce full 3-D models of moving objects from multiple Kinect streams in real-time. We present the capturing setup, the methodology for its calibration and the details of the proposed algorithm for real-time fusion of multiple meshes. The presented experimental results verify the effectiveness of the approach with respect to the 3-D reconstruction quality, as well as the achieved frame rates.

3D model search and retrieval using the spherical trace transform

This paper presents a novel methodology for content-based search and retrieval of 3D objects. After proper positioning of the 3D objects using translation and scaling, a set of functionals is applied to the 3D model producing a new domain of concentric spheres. In this new domain, a new set of functionals is applied, resulting in a descriptor vector which is completely rotation invariant and thus suitable for 3D model matching. Further, weights are assigned to each descriptor, so as to significantly improve the retrieval results. Experiments on two different databases of 3D objects are performed so as to evaluate the proposed method in comparison with those most commonly cited in the literature. The experimental results show that the proposed method is superior in terms of precision versus recall and can be used for 3D model search and retrieval in a highly efficient manner.

Efficient 3-D model search and retrieval using generalized 3-D radon transforms

Measuring the similarity between three-dimensional (3-D) objects is a challenging problem, with applications in computer vision, molecular biology, computer graphics, and many other areas. This paper describes a novel method for 3-D model content-based search based on the 3-D Generalized Radon Transform and a querying by-3-D-model approach. A set of descriptor vectors is extracted using the Radial Integration Transform (RIT) and the Spherical Integration Transform (SIT), which represent significant shape characteristics. After the proper alignment of the models, descriptor vectors are produced which are invariant in terms of translation, scaling and rotation. Experiments were performed using three different databases and comparing the proposed method with those most commonly cited in the literature. Experimental results show that the proposed method is adequately satisfactory in terms of both precision versus recall and time needed for retrieval, and that it can be used for 3-D model search and retrieval in a highly efficient manner.

Three-Dimensional Shape-Structure Comparison Method for Protein Classification

In this paper, a 3D shape-based approach is presented for the efficient search, retrieval, and classification of protein molecules. The method relies primarily on the geometric 3D structure of the proteins, which is produced from the corresponding PDB files and secondarily on their primary and secondary structure. After proper positioning of the 3D structures, in terms of translation and scaling, the spherical trace transform is applied to them so as to produce geometry-based descriptor vectors, which are completely rotation invariant and perfectly describe their 3D shape. Additionally, characteristic attributes of the primary and secondary structure of the protein molecules are extracted, forming attribute-based descriptor vectors. The descriptor vectors are weighted and an integrated descriptor vector is produced. Three classification methods are tested. A part of the FSSP/DALI database, which provides a structural classification of the proteins, is used as the ground truth in order to evaluate the classification accuracy of the proposed method. The experimental results show that the proposed method achieves more than 99 percent classification accuracy while remaining much simpler and faster than the DALI method

Immersive 3D Holoscopic Video System

We demonstrated a 3D holoscopic video system for 3DTV application. We showed that using a field lens and a square aperture significantly reduces the vignetting problem associated with a relay system and achieves over 95 percent fill factor. The main problem for such a relay system is the nonlinear distortion during the 3D image capturing, which can seriously affect the reconstruction process for a 3D display. The nonlinear distortion mainly includes lens radial distortion (intrinsic) and microlens array perspective distortion (extrinsic). This is the task of future work. Our results also show that the SS coding approach performs better than the standard HEVC scheme. Furthermore, we show that search and retrieval performance relies on the depth maps quality and that the multimodal fusion boosts the retrieval performance.

Shape matching using the 3D Radon transform

A method for 3D model content-based search and retrieval based on the 3D Radon transform and a querying-by-3D-model approach, is presented. Descriptors are extracted using the 3D Radon transform and applying a set of functionals on the transform coefficients. Similarity measures are then created for the extracted descriptors and introduced into a 3D model-matching algorithm. This results to a very fast and accurate matching method. Experiments were performed using two different databases and comparing the proposed method with others. Experimental results show that the proposed method can be used for 3D model search and retrieval in a highly efficient manner.

Toward Real-Time and Efficient Compression of Human Time-Varying Meshes

In this paper, a novel skeleton-based approach to human time-varying mesh (H-TVM) compression is presented. The topic of TVM compression is new and has many challenges, such as handling the lack of obvious mapping of vertices across frames and handling the variable connectivity across frames, while maintaining efficiency, which are the most important ones. Very few works exist in the literature, while not all of the challenges have been addressed yet. In addition, developing an efficient and real-time solution, handling the above, obviously is a difficult task. We attempt to address the H-TVM compression problem inspired from video coding using different types of frames and trying to efficiently remove inter-frame geometric redundancy utilizing the recent advances in human skeleton tracking. The overall approach focuses on compression efficiency, low distortion, and low computation time enabling for real-time transmission of H-TVMs. It efficiently compresses geometry and vertex attributes of TVMs. In addition, this paper is the first to provide an efficient method for connectivity coding of TVMs, by introducing a modification to the state-of-the-art MPEG-4 TFAN algorithm. Experiments are conducted in the MPEG-3DGC TVM database. The method outperforms the state-of-the-art standardized static mesh coder MPEG-4 TFAN at low bit-rates, while remaining competent at high bit-rates. It gives a practical proof of concept that in the combined problem of geometry, connectivity, and vertex attribute coding of TVMs, efficient inter-frame redundancy removal is possible, establishing ground for further improvements. Finally, this paper proposes a method for motion-based coding of H-TVMs that can further enhance the overall experience when H-TVM compression is used in a tele-immersion scenario.

Real-time, realistic full-body 3D reconstruction and texture mapping from multiple Kinects

Multi-party 3D Tele-Immersive (TI) environments, supporting realistic interaction among distant users, is the future of tele-conferencing. Real-time, full-body 3D reconstruction, an important task for TI applications, is addressed in this paper. A volumetric method for the reconstruction of watertight models of moving humans is presented, along with details for appropriate texture-mapping to enhance the visual quality. The reconstruction uses the input from multiple consumer depth cameras and specifically Kinect sensors. The presented results verify the effectiveness of the proposed methodologies, with respect to the visual quality and frame rates.

3D model search and retrieval based on the spherical trace transform

This paper presents a novel approach in 3D content-based search and retrieval. First, a set of functional are applied on a 3D models volume producing a new domain of concentric spheres. In this new domain a new set of functionals is applied, resulting to a completely rotation invariant descriptor vector, which is used for 3D model matching. Experiments were performed using a database and comparing the proposed method with the MPEG-7 3D shape spectrum descriptor. Experimental results show that the proposed method is superior in terms of precision versus recall and can be used for 3D model search and retrieval in a highly efficient manner.

An Integrated Platform for Live 3D Human Reconstruction and Motion Capturing

The latest developments in 3D capturing, processing, and rendering provide means to unlock novel 3D application pathways. The main elements of an integrated platform, which target tele-immersion and future 3D applications, are described in this paper, addressing the tasks of real-time capturing, robust 3D human shape/appearance reconstruction, and skeleton-based motion tracking. More specifically, initially, the details of a multiple RGB-depth (RGB-D) capturing system are given, along with a novel sensors’ calibration method. A robust, fast reconstruction method from multiple RGB-D streams is then proposed, based on an enhanced variation of the volumetric Fourier transform-based method, parallelized on the Graphics Processing Unit, and accompanied with an appropriate texture-mapping algorithm. On top of that, given the lack of relevant objective evaluation methods, a novel framework is proposed for the quantitative evaluation of real-time 3D reconstruction systems. Finally, a generic, multiple depth stream-based method for accurate real-time human skeleton tracking is proposed. Detailed experimental results with multi-Kinect2 data sets verify the validity of our arguments and the effectiveness of the proposed system and methodologies.