Lixin Fan

Fast Localization in Large-Scale Environments Using Supervised Indexing of Binary Features

The essence of image-based localization lies in matching 2D key points in the query image and 3D points in the database. State-of-the-art methods mostly employ sophisticated key point detectors and feature descriptors, e.g., Difference of Gaussian (DoG) and Scale Invariant Feature Transform (SIFT), to ensure robust matching. While a high registration rate is attained, the registration speed is impeded by the expensive key point detection and the descriptor extraction. In this paper, we propose to use efficient key point detectors along with binary feature descriptors, since the extraction of such binary features is extremely fast. The naive usage of binary features, however, does not lend itself to significant speedup of localization, since existing indexing approaches, such as hierarchical clustering trees and locality sensitive hashing, are not efficient enough in indexing binary features and matching binary features turns out to be much slower than matching SIFT features. To overcome this, we propose a much more efficient indexing approach for approximate nearest neighbor search of binary features. This approach resorts to randomized trees that are constructed in a supervised training process by exploiting the label information derived from that multiple features correspond to a common 3D point. In the tree construction process, node tests are selected in a way such that trees have uniform leaf sizes and low error rates, which are two desired properties for efficient approximate nearest neighbor search. To further improve the search efficiency, a probabilistic priority search strategy is adopted. Apart from the label information, this strategy also uses non-binary pixel intensity differences available in descriptor extraction. By using the proposed indexing approach, matching binary features is no longer much slower but slightly faster than matching SIFT features. Consequently, the overall localization speed is significantly improved due to the much faster key point detection and descriptor extraction. It is empirically demonstrated that the localization speed is improved by an order of magnitude as compared with state-of-the-art methods, while comparable registration rate and localization accuracy are still maintained.The essence of image-based localization lies in matching 2D key points in the query image and 3D points in the database. State-of-the-art methods mostly employ sophisticated key point detectors and feature descriptors, e.g., Difference of Gaussian (DoG) and Scale Invariant Feature Transform (SIFT), to ensure robust matching. While a high registration rate is attained, the registration speed is impeded by the expensive key point detection and the descriptor extraction. In this paper, we propose to use efficient key point detectors along with binary feature descriptors, since the extraction of such binary features is extremely fast. The naive usage of binary features, however, does not lend itself to significant speedup of localization, since existing indexing approaches, such as hierarchical clustering trees and locality sensitive hashing, are not efficient enough in indexing binary features and matching binary features turns out to be much slower than matching SIFT features. To overcome this, we propose a much more efficient indexing approach for approximate nearest neighbor search of binary features. This approach resorts to randomized trees that are constructed in a supervised training process by exploiting the label information derived from that multiple features correspond to a common 3D point. In the tree construction process, node tests are selected in a way such that trees have uniform leaf sizes and low error rates, which are two desired properties for efficient approximate nearest neighbor search. To further improve the search efficiency, a probabilistic priority search strategy is adopted. Apart from the label information, this strategy also uses non-binary pixel intensity differences available in descriptor extraction. By using the proposed indexing approach, matching binary features is no longer much slower but slightly faster than matching SIFT features. Consequently, the overall localization speed is significantly improved due to the much faster key point detection and descriptor extraction. It is empirically demonstrated that the localization speed is improved by an order of magnitude as compared with state-of-the-art methods, while comparable registration rate and localization accuracy are still maintained.

A feature-based object tracking approach for realtime image processing on mobile devices

In this paper we present a robust object tracking approach which is suitable for real-time image processing on mobile devices. Challenging mobile environments render traditional color-based tracking methods useless. Many online learning tracking methods are too computationally complex to be used for real-time mobile applications, which only have access to limited computational resource and memory storage. The proposed method takes advantage of local feature to deal with rapid camera motion, and employs an online feature updating scheme to cope with variation in object appearances. The method is also computationally lightweight, being able to support real-time image processing on mobile devices.

On-line Object Reconstruction and Tracking for 3D Interaction

This paper presents a flexible and easy-to-use tracking method for 3D interaction. The method reconstructs points of a user-specified object from a video sequence, and recovers the 6 degrees of freedom (DOF) camera pose and position relative to the reconstructed points in each video frame. As opposed to most existing 3D object tracking methods, the proposed method does not need any off-line modeling or training process. Instead, it first segments the object from the background, then reconstructs and tracks the object using the Visual Simultaneous Localization And Mapping (VSLAM) techniques. To our knowledge, there are no existing works investigating this kind of on line reconstruction and tracking of moving objects. The proposed method employs the adapted pyramidal Lucas-Kanade tracker to increase the stability and the robustness of the tracking when dealing with a lightly textured or fast moving object. Experiments show that fast, accurate, stable and robust tracking can be achieved in everyday environment. Moreover, a simple stereo initialization approach is adopted to minimize user intervention. All these attributes conspire to make the method an adequate tool for some interaction applications. As a concrete example, an interactive 3D scene displaying system is demonstrated.

Efficient pose tracking on mobile phones with 3D points grouping

With the rapid growth of computational capability and popularity of mobile phones, Mobile Augmented Reality (MAR) in large scale 3D scenes becomes an emerging field in recent years. The core of MAR is to continuously compute a precise 6 Degree-of-Freedom (DOF) camera pose for each frame, i.e. localization. However, as a crucial part of localization, the 2D-3D points matching is usually inefficient due to the usage of traditional features, e.g. SIFT, and the large number of 3D points candidates. This paper aims to tackle this problem by designing an efficient 6DOF pose tracking system on mobile phone. In this system, binary features are used in both offline sparse reconstruction and online tracking, while a PCA-based 3D points partition method is proposed to reduce the searching space of 2D-3D points matching, making it capable to achieve a low computational cost. Experiments on a NOKIA N900 smartphone show that our system could efficiently and robustly estimate the 6DOF camera pose.

Street view cross-sourced point cloud matching and registration

Object registration has been widely discussed with the development of various range sensing technologies. In most work, however, the point clouds of reference and target are generated by the same technology, such as a Kinect range camera, LiDAR sensor, or Structure from Motion technique. Cases in which reference and target point clouds are generated by different technologies are rarely discussed. Due to the significant differences across various point cloud data in terms of point cloud density, sensing noise, scale, occlusion etc., object registration between such different point clouds becomes extremely difficult. In this study, we address for the first time an even more challenging case in which the differently-sourced point clouds are acquired from a real street view. One is generated on the basis of an image sequence through the SfM process, and the other is produced directly by the LiDAR system. We propose a two-stage matching and registration algorithm to achieve object registration between these two different point clouds. The experiments are based on real building object point cloud data and demonstrate the effectiveness and efficiency of the proposed solution. The newly proposed solution can be further developed to contribute to several related applications, such as Location Based Service.

Online Learning of Binary Feature Indexing for Real-Time SLAM Relocalization

In this paper, we propose an indexing method for approximate nearest neighbor search of binary features. Being different from the popular Locality Sensitive Hashing (LSH), the proposed method construct the hash keys by an online learning process instead of pure randomness. In the learning process, the hash keys are constructed with the aim of obtaining uniform hash buckets and high collision rates, which makes the method more efficient on approximate nearest neighbor search than LSH. By distributing the online learning into the simultaneous localization and mapping (SLAM) process, we successfully apply the method to SLAM relocalization. Experiments show that camera poses can be successfully recovered in real time even there are tens of thousands of landmarks in the map.

Real-time SLAM relocalization with online learning of binary feature indexing

A visual simultaneous localization and mapping (SLAM) system usually contains a relocalization module to recover the camera pose after tracking failure. The core of this module is to establish correspondences between map points and key points in the image, which is typically achieved by local image feature matching. Since recently emerged binary features have orders of magnitudes higher extraction speed than traditional features such as scale invariant feature transform, they can be applied to develop a real-time relocalization module once an efficient method of binary feature matching is provided. In this paper, we propose such a method by indexing binary features with hashing. Being different from the popular locality sensitive hashing, the proposed method constructs the hash keys by an online learning process instead of pure randomness. Specifically, the hash keys are trained with the aim of attaining uniform hash buckets and high collision rates of matched feature pairs, which makes the method more efficient on approximate nearest neighbor search. By distributing the online learning into the simultaneous localization and mapping process, we successfully apply the method to SLAM relocalization. Experiments show that camera poses can be recovered in real time even when there are tens of thousands of landmarks in the map.

Real Time Complete Dense Depth Reconstruction for a Monocular Camera

In this paper, we aim to solve the problem of estimating complete dense depth maps from a monocular moving camera. By complete, we mean depth information is estimated for every pixel and detailed reconstruction is achieved. Although this problem has previously been attempted, the accuracy of complete dense depth reconstruction is a remaining problem. We propose a novel system which produces accurate complete dense depth map. The new system consists of two subsystems running in separated threads, namely, dense mapping and sparse patch-based tracking. For dense mapping, a new projection error computation method is proposed to enhance the gradient component in estimated depth maps. For tracking, a new sparse patch-based tracking method estimates camera pose by minimizing a normalized error term. The experiments demonstrate that the proposed method obtains improved performance in terms of completeness and accuracy compared to three state-of the-art dense reconstruction methods VSFM+CMVC, LSDSLAM and REMODE.

A Systematic Approach for Cross-Source Point Cloud Registration by Preserving Macro and Micro Structures

We propose a systematic approach for registering cross-source point clouds that come from different kinds of sensors. This task is especially challenging due to the presence of significant missing data, large variations in point density, scale difference, large proportion of noise, and outliers. The robustness of the method is attributed to the extraction of macro and micro structures. Macro structure is the overall structure that maintains similar geometric layout in cross-source point clouds. Micro structure is the element (e.g., local segment) being used to build the macro structure. We use graph to organize these structures and convert the registration into graph matching. With a novel proposed descriptor, we conduct the graph matching in a discriminative feature space. The graph matching problem is solved by an improved graph matching solution, which considers global geometrical constraints. Robust cross source registration results are obtained by incorporating graph matching outcome with RANSAC and ICP refinements. Compared with eight state-of-the-art registration algorithms, the proposed method invariably outperforms on Pisa Cathedral and other challenging cases. In order to compare quantitatively, we propose two challenging cross-source data sets and conduct comparative experiments on more than 27 cases, and the results show we obtain much better performance than other methods. The proposed method also shows high accuracy in same-source data sets.

A Coarse-to-Fine Algorithm for Registration in 3D Street-View Cross-Source Point Clouds

With the development of numerous 3D sensing technologies, object registration on cross-source point cloud has aroused researchers interests. When the point clouds are captured from different kinds of sensors, there are large and different kinds of variations. In this study, we address an even more challenging case in which the differently-source point clouds are acquired from a real street view. One is produced directly by the LiDAR system and the other is generated by using VSFM software on image sequence captured from RGB cameras. When it confronts to large scale point clouds, previous methods mostly focus on point-to-point level registration, and the methods have many limitations.The reason is that the least mean error strategy shows poor ability in registering large variable cross-source point clouds. In this paper, different from previous ICP-based methods, and from a statistic view, we propose a effective coarse-to-fine algorithm to detect and register a small scale SFM point cloud in a large scale Lidar point cloud. Seen from the experimental results, the model can successfully run on LiDAR and SFM point clouds, hence it can make a contribution to many applications, such as robotics and smart city development.