Lingni Ma

FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture

In this paper we address the problem of semantic labeling of indoor scenes on RGB-D data. With the availability of RGB-D cameras, it is expected that additional depth measurement will improve the accuracy. Here we investigate a solution how to incorporate complementary depth information into a semantic segmentation framework by making use of convolutional neural networks (CNNs). Recently encoder-decoder type fully convolutional CNN architectures have achieved a great success in the field of semantic segmentation. Motivated by this observation we propose an encoder-decoder type network, where the encoder part is composed of two branches of networks that simultaneously extract features from RGB and depth images and fuse depth features into the RGB feature maps as the network goes deeper. Comprehensive experimental evaluations demonstrate that the proposed fusion-based architecture achieves competitive results with the state-of-the-art methods on the challenging SUN RGB-D benchmark obtaining 76.27% global accuracy, 48.30% average class accuracy and 37.29% average intersection-over-union score.

CPA-SLAM: Consistent plane-model alignment for direct RGB-D SLAM

Planes are predominant features of man-made environments which have been exploited in many mapping approaches. In this paper, we propose a real-time capable RGB-D SLAM system that consistently integrates frame-to-keyframe and frame-to-plane alignment. Our method models the environment with a global plane model and - besides direct image alignment - it uses the planes for tracking and global graph optimization. This way, our method makes use of the dense image information available in keyframes for accurate short-term tracking. At the same time it uses a global model to reduce drift. Both components are integrated consistently in an expectation-maximization framework. In experiments, we demonstrate the benefits our approach and its state-of-the-art accuracy on challenging benchmarks.

Incremental and batch planar simplification of dense point cloud maps

Dense RGB-D SLAM techniques and high-fidelity LIDAR scanners are examples from an abundant set of systems capable of providing multi-million point datasets. These datasets quickly become difficult to process due to the sheer volume of data, typically containing significant redundant information, such as the representation of planar surfaces with millions of points. In order to exploit the richness of information provided by dense methods in real-time robotics, techniques are required to reduce the inherent redundancy of the data. In this paper we present a method for incremental planar segmentation of a gradually expanding point cloud map and a method for efficient triangulation and texturing of planar surface segments. Experimental results show that our incremental segmentation method is capable of running in real-time while producing a segmentation faithful to what would be achieved using a batch segmentation method. Our results also show that the proposed planar simplification and triangulation algorithm removes more than 90% of the input planar points, leading to a triangulation with only 10% of the original quantity of triangles per planar segment. Additionally, our texture generation algorithm preserves all colour information contained within planar segments, resulting in a visually appealing and geometrically accurate simplified representation. Online incremental planar segmentation algorithm capable of running in real-time.Point cloud plane segment triangulation that preserves principle geometric features.Texture generation algorithm that maintains visual appearance of planar surfaces.

De-noising, stabilizing and completing 3D reconstructions on-the-go using plane priors

Creating 3D maps on robots and other mobile devices has become a reality in recent years. Online 3D reconstruction enables many exciting applications in robotics and AR/VR gaming. However, the reconstructions are noisy and generally incomplete. Moreover, during online reconstruction, the surface changes with every newly integrated depth image which poses a significant challenge for physics engines and path planning algorithms. This paper presents a novel, fast and robust method for obtaining and using information about planar surfaces, such as walls, floors, and ceilings as a stage in 3D reconstruction based on Signed Distance Fields (SDFs). Our algorithm recovers clean and accurate surfaces, reduces the movement of individual mesh vertices caused by noise during online reconstruction and fills in the occluded and unobserved regions. We implemented and evaluated two different strategies to generate plane candidates and two strategies for merging them. Our implementation is optimized to run in real-time on mobile devices such as the Tango tablet. In an extensive set of experiments, we validated that our approach works well in a large number of natural environments despite the presence of significant amount of occlusion, clutter and noise, which occur frequently. We further show that plane fitting enables in many cases a meaningful semantic segmentation of real-world scenes.

Planar simplification and texturing of dense point cloud maps

Dense RGB-D based SLAM techniques and high-fidelity LIDAR scanners are examples from an abundant set of systems capable of providing multi-million point datasets. These large datasets quickly become difficult to process and work with due to the sheer volume of data, which typically contains significant redundant information, such as the representation of planar surfaces with hundreds of thousands of points. In order to exploit the richness of information provided by dense methods in real-time robotics, techniques are required to reduce the inherent redundancy of the data. In this paper we present a method for efficient triangulation and texturing of planar surfaces in large point clouds. Experimental results show that our algorithm removes more than 90% of the input planar points, leading to a triangulation with only 10% of the original amount of triangles per planar segment, improving upon an existing planar simplification algorithm. Despite the large reduction in vertex count, the principal geometric features of each segment are well preserved. In addition to this, our texture generation algorithm preserves all colour information contained within planar segments, resulting in a visually appealing and geometrically accurate simplified representation.

Plane segmentation and decimation of point clouds for 3D environment reconstruction

Three-dimensional (3D) models of environments are a promising technique for serious gaming and professional engineering applications. In this paper, we introduce a fast and memory-efficient system for the reconstruction of large-scale environments based on point clouds. Our main contribution is the emphasis on the data processing of large planes, for which two algorithms have been designed to improve the overall performance of the 3D reconstruction. First, a flatness-based segmentation algorithm is presented for plane detection in point clouds. Second, a quadtree-based algorithm is proposed for decimating the point cloud involved with the segmented plane and consequently improving the efficiency of triangulation. Our experimental results have shown that the proposed system and algorithms have a high efficiency in speed and memory for environment reconstruction. Depending on the amount of planes in the scene, the obtained efficiency gain varies between 20% and 50%.

On photo-realistic 3D reconstruction of large-scale and arbitrary-shaped environments

This paper presents a system architecture for reconstructing photorealistic and accurate 3D models of indoor environments. The system specifically targets large-scale and arbitrary-shaped environments and enables processing of data obtained with an arbitrary-chosen capturing path. The system extends the baseline Kinect Fusion algorithm with a buffering algorithm to remove scene-size capturing limitations. Beside this, the paper presents the complete chain of advanced algorithms for point cloud segmentation/decimation, camera pose correction and texture mapping with post-processing filters. The presented architecture features memory- and processor-efficient processing, such that it can be executed on a conventional PC with a mainstream GPU card at the consumer premises.

3D colored model generation based on multiview textures and triangular mesh

Three-dimensional colored models are of great interests to many fields. With the growing availability of inexpensive 3D sensing systems, it is easy to obtain triangular mesh and multiview textures. These range and vision data can be fused to provide such 3D colored models. However, low-cost sensing generates various noise components involving low-quality texture, errors in calibration and mesh modeling. Our primary objective is to establish high-quality 3D colored models on the basis of mesh and textures, while considering the noise types and characteristics. In this paper, we contribute in two ways. The first contribution is a point-based algorithm to color 3D models, where 3D surface points are used as primitives to process and store color information. The algorithm features three novel techniques: (a) accurate depth image estimation, (b) adaptive 3D surface point upsampling and (c) texture blending using those points. The algorithm provides colored models as dense colored point clouds, which can be rendered with various standard techniques for visualization. Our second contribution is an algorithm for textured model rendering, where blended textures are generated and mapped onto the mesh. The experimental results show that our algorithms efficiently provide high-quality colored models and enable visually appealing rendering, while being tolerant to errors from data acquisition. We also quantify the efficiency of our point upsampling algorithm with novel metrics assessing the influence of the 3D points.