H Hani Javan Hemmat

Fast planar segmentation of depth images

One of the major challenges for applications dealing with the 3D concept is the real-time execution of the algorithms. Besides this, for the indoor environments, perceiving the geometry of surrounding structures plays a prominent role in terms of application performance. Since indoor structures mainly consist of planar surfaces, fast and accurate detection of such features has a crucial impact on quality and functionality of the 3D applications, e.g. decreasing model size (decimation), enhancing localization, mapping, and semantic reconstruction. The available planar-segmentation algorithms are mostly developed using surface normals and/or curvatures. Therefore, they are computationally expensive and challenging for real-time performance. In this paper, we introduce a fast planar-segmentation method for depth images avoiding surface normal calculations. Firstly, the proposed method searches for 3D edges in a depth image and finds the lines between identified edges. Secondly, it merges all the points on each pair of intersecting lines into a plane. Finally, various enhancements (e.g. filtering) are applied to improve the segmentation quality. The proposed algorithm is capable of handling VGA-resolution depth images at a 6 FPS frame-rate with a single-thread implementation. Furthermore, due to the multi-threaded design of the algorithm, we achieve a factor of 10 speedup by deploying a GPU implementation.

Real-time planar segmentation of depth images: from three-dimensional edges to segmented planes

Abstract. Real-time execution of processing algorithms for handling depth images in a three-dimensional (3-D) data framework is a major challenge. More specifically, considering depth images as point clouds and performing planar segmentation requires heavy computation, because available planar segmentation algorithms are mostly based on surface normals and/or curvatures, and, consequently, do not provide real-time performance. Aiming at the reconstruction of indoor environments, the spaces mainly consist of planar surfaces, so that a possible 3-D application would strongly benefit from a real-time algorithm. We introduce a real-time planar segmentation method for depth images avoiding any surface normal calculation. First, we detect 3-D edges in a depth image and generate line segments between the identified edges. Second, we fuse all the points on each pair of intersecting line segments into a plane candidate. Third and finally, we implement a validation phase to select planes from the candidates. Furthermore, various enhancements are applied to improve the segmentation quality. The GPU implementation of the proposed algorithm segments depth images into planes at the rate of 58 fps. Our pipeline-interleaving technique increases this rate up to 100 fps. With this throughput rate improvement, the application benefit of our algorithm may be further exploited in terms of quality and enhancing the localization.

Camera-to-model back-raycasting for extraction of RGB-D images from pointclouds

Conventional raycasting methods extract 2D-images from pointclouds in two main steps. The pointcloud is voxelized and then, rays are casted from a virtual-camera center towards the model. The value for each pixel in the resulting image is calculated based on the closest non-empty voxel intersected with the corresponding ray. Both voxelizing and such raycasting limit the quality (resolution) of the extracted image and impose high memory demands. In this paper, we propose an alternative backraycasting method, where rays are casted from the model towards the virtual-camera center and intersecting an image plane. This does not require any voxel grid to be generated. Moreover, this method allows to obtain images with any required resolution with all the points involved. Besides this, a neighbours-consistency technique is introduced to enhance the resulting image quality. The proposed method has been evaluated based on several criteria and for various resolutions. Evaluation results show that the proposed method compared to the conventional approach executes upto 49 times faster and improves PSNR and SSIM metrics for the resulting images by 26% and 12%; respectively. This improvement is beneficial for such domains as feature matching, edge detection, OCR and calibration. To enable researchers generating the same results and extend this work, the dataset and implementation codes are publicly available [1].

R^{3}P: Real-time RGB-D Registration Pipeline

Applications based on colored 3-D data sequences suffer from lack of efficient algorithms for transformation estimation and key points extraction to perform accurate registration and sensor localization either in the 2-D or 3-D domain. Therefore, we propose a real-time RGB-D registration pipeline, named R3P, presented in processing layers. In this paper, we present an evaluation of several algorithm combinations for each layer, to optimize the registration and sensor localization for specific applications. The resulting dynamic reconfigurability of R3P makes it suitable as a front-end system for any SLAM reconstruction algorithm. Evaluation results on several public datasets reveal that R3P delivers real-time registration with 59 fps and high accuracy with the relative pose error (for a time span of 40 frames) for rotation and translation of \(0.5^\circ \) and 8 mm, respectively. All the heterogeneous dataset and implementations are publicly available under an open-source license [21].

Solidarity Filter for Noise Reduction of 3D Edges in Depth Images

3D applications processing depth images significantly benefit from 3D-edge extraction techniques. Intrinsic sensor noise in depth images is largely inherited to the extracted 3D edges. Conventional denoising algorithms remove some of this noise, but also weaken narrow edges, amplify noisy pixels and introduce false edges. We therefore propose a novel solidarity filter for noise removal in 3D edge images without artefacts such as false edges. The proposed filter is defining neighbouring pixels with similar properties and connecting those into larger segments beyond the size of a conventional filter aperture. The experimental results show that the solidarity filter outperforms the median and morphological close filters with 42% and 69% higher PSNR, respectively. In terms of the mean SSIM metric, the solidarity filter provides results that are 11% and 21% closer to the ground truth than the corresponding results obtained by the median and close filters, respectively.