Hainan Cui

Efficient Large-Scale Structure From Motion by Fusing Auxiliary Imaging Information

One of the potentially effective means for large-scale 3D scene reconstruction is to reconstruct the scene in a global manner, rather than incrementally, by fully exploiting available auxiliary information on the imaging condition, such as camera location by Global Positioning System (GPS), orientation by inertial measurement unit (or compass), focal length from EXIF, and so on. However, such auxiliary information, though informative and valuable, is usually too noisy to be directly usable. In this paper, we present an approach by taking advantage of such noisy auxiliary information to improve structure from motion solving. More specifically, we introduce two effective iterative global optimization algorithms initiated with such noisy auxiliary information. One is a robust rotation averaging algorithm to deal with contaminated epipolar graph, the other is a robust scene reconstruction algorithm to deal with noisy GPS data for camera centers initialization. We found that by exclusively focusing on the estimated inliers at the current iteration, the optimization process initialized by such noisy auxiliary information could converge well and efficiently. Our proposed method is evaluated on real images captured by unmanned aerial vehicle, StreetView car, and conventional digital cameras. Extensive experimental results show that our method performs similarly or better than many of the state-of-art reconstruction approaches, in terms of reconstruction accuracy and completeness, but is more efficient and scalable for large-scale image data sets.

HSfM: Hybrid Structure-from-Motion

Structure-from-Motion (SfM) methods can be broadly categorized as incremental or global according to their ways to estimate initial camera poses. While incremental system has advanced in robustness and accuracy, the efficiency remains its key challenge. To solve this problem, global reconstruction system simultaneously estimates all camera poses from the epipolar geometry graph, but it is usually sensitive to outliers. In this work, we propose a new hybrid SfM method to tackle the issues of efficiency, accuracy and robustness in a unified framework. More specifically, we propose an adaptive community-based rotation averaging method first to estimate camera rotations in a global manner. Then, based on these estimated camera rotations, camera centers are computed in an incremental way. Extensive experiments show that our hybrid method performs similarly or better than many of the state-of-the-art global SfM approaches, in terms of computational efficiency, while achieves similar reconstruction accuracy and robustness with two other state-of-the-art incremental SfM approaches.

Tracks selection for robust, efficient and scalable large-scale structure from motion

Currently global structure-from-motion (SfM) pipeline consists of four steps: estimating camera rotations first, then computing camera positions, triangulating tracks, and finally doing bundle adjustment. However, for large-scale SfM problems, the tracks are usually too noisy and redundant for the bundle adjustment. Thus in this work, we propose a novel fast tracks selection method to improve both efficiency and robustness of the bundle adjustment. Firstly, three selection criteria: Compactness, Accurateness, and Connectedness, are introduced, where the first two are to calculate a selection priority for each track and the third is to guarantee the completeness of scene structure. Then, to satisfy these criteria, a more informative subset of tracks is selected by covering multiple spanning trees of epipolar geometry graph. Since tracks selection acts only an intermediate step in the whole SfM pipeline, it can be in principle embedded into any global SfM pipelines. To validate the effectiveness of our tracks selection module, we insert it into a state-of-the-art global SfM system and compare it with three other selection methods. Extensive experiments show that by embedding our tracks selection module, the new SfM system performs similarly or better than the original one in terms of reconstruction completeness and accuracy, but is much more efficient and scalable for large-scale scene reconstructions. Finally, our tracks selection module is further embedded into two other global SfM systems to demonstrated its versatility

Fusion of Auxiliary Imaging Information for Robust, Scalable and Fast 3D Reconstruction

One of the potentially effective means for 3D reconstruction is to reconstruct the scene in a global manner, rather than incrementally, by fully exploiting available auxiliary information on imaging condition, such as camera location by GPS, orientation by IMU(or Compass), focal length from EXIF etc. However these auxiliary information, though informative and valuable, is usually too noisy to be directly usable. In this paper, we present a global method by taking advantage of such noisy auxiliary information to improve SfM solving. More specifically, we introduce two effective iterative optimization algorithms directly initiated with such noisy auxiliary information. One is a robust iterative rotation estimation algorithm to deal with contaminated EG(epipolar graph), the other is a robust iterative scene reconstruction algorithm to deal with noisy GPS data for camera centers initialization. We found that by exclusively focusing on the inliers estimated at the current iteration, called potential inliers in this work, the optimization process initialized by such noisy auxiliary information could converge well and efficiently. Our proposed method is evaluated on real images captured by UAV(unmanned aerial vehicle), StreetView car and conventional digital cameras. Extensive experimental results show that our method performs similarly or better than many of the state-of-art reconstruction approaches, in terms of reconstruction accuracy and scene completeness, but more efficient and scalable for large-scale image datasets.

Accurate and efficient ground-to-aerial model alignment

Abstract To produce a complete 3D reconstruction of a large-scale architectural scene, both ground and aerial images are usually captured. A common approach is to first reconstruct the models from different image sources separately, and align them afterwards. Using this pipeline, this work proposes an accurate and efficient approach for ground-to-aerial model alignment in a coarse-to-fine manner. First, both the ground model and aerial model are transformed into the geo-referenced coordinate system using GPS meta-information for coarse alignment. Then, the coarsely aligned models are refined by a similarity transformation that is estimated based on 3D point correspondences between them, and the 3D point correspondences are determined in a 2D-image-matching manner by considering the rich textural and contextual information in the 2D images. Due to the dramatic differences in viewpoint and scale between ground and aerial images, which make matching them directly nearly impossible, we perform an intermediate view-synthesis step to mitigate the matching difficulty. To this end, the following three key issues are addressed: (a) selecting a suitable subset of aerial images to cover the ground model properly; (b) synthesizing images from the ground model under the viewpoints of the selected aerial images; and finally, (c) obtaining the 2D point matches between the synthesized images and the selected aerial images. The experimental results show that the proposed model alignment approach is quite effective and outperforms several state-of-the-art techniques in terms of both accuracy and efficiency.

Global fusion of generalized camera model for efficient large-scale structure from motion

摘要创新点1、全局式融合广义摄像机模型重建大规模城市街景和倾斜摄影数据, 使得误差均摊在整个外极几何图, 不存在因为误差累积而造成的场景漂移。2、捆绑调整的次数极大地减少, 使得整个重建系统速度相对于增量式重建得到了巨大的提升。3、充分融合有噪成像信息 (比如 GPS、 指南针夹角等) 来提升重建系统的速度和精度。

Geographic, geometrical and semantic reconstruction of urban scene from high resolution oblique aerial images

An effective approach is proposed for 3D urban scene reconstruction in the form of point cloud with semantic labeling. Starting from high resolution oblique aerial images, our approach proceeds through three main stages: geographic reconstruction, geometrical reconstruction and semantic reconstruction. The absolute position and orientation of all the cameras relative to the real world are recovered in the geographic reconstruction stage. Then, in the geometrical reconstruction stage, an improved multi-view stereo matching method is employed to produce 3D dense points with color and normal information by taking into account the prior knowledge of aerial imagery. Finally the point cloud is classified into three classes ( building, vegetation, and ground ) by a rule-based hierarchical approach in the semantic reconstruction step. Experiments on complex urban scene show that our proposed 3-stage approach could generate reasonable reconstruction result robustly and efficiently. By comparing our final semantic reconstruction result with the manually labeled ground truth, classification accuracies from 86.75% to 93.02% are obtained.

Learning stratified 3D reconstruction

Stratified 3D reconstruction, or a layer-by-layer 3D reconstruction upgraded from projective to affine, then to the final metric reconstruction, is a well-known 3D reconstruction method in computer vision. It is also a key supporting technology for various well-known applications, such as streetview, smart3D, oblique photogrammetry. Generally speaking, the existing computer vision methods in the literature can be roughly classified into either the geometry-based approaches for spatial vision or the learning-based approaches for object vision. Although deep learning has demonstrated tremendous success in object vision in recent years, learning 3D scene reconstruction from multiple images is still rare, even not existent, except for those on depth learning from single images. This study is to explore the feasibility of learning the stratified 3D reconstruction from putative point correspondences across images, and to assess whether it could also be as robust to matching outliers as the traditional geometry-based methods do. In this study, a special parsimonious neural network is designed for the learning. Our results show that it is indeed possible to learn a stratified 3D reconstruction from noisy image point correspondences, and the learnt reconstruction results appear satisfactory although they are still not on a par with the state-of-the-arts in the structure-from-motion community due to largely its lack of an explicit robust outlier detector such as random sample consensus (RANSAC). To the best of our knowledge, our study is the first attempt in the literature to learn 3D scene reconstruction from multiple images. Our results also show that how to implicitly or explicitly integrate an outlier detector in learning methods is a key problem to solve in order to learn comparable 3D scene structures to those by the current geometry-based state-of-the-arts. Otherwise any significant advancement of learning 3D structures from multiple images seems difficult, if not impossible. Besides, we even speculate that deep learning might be, in nature, not suitable for learning 3D structure from multiple images, or more generally, for solving spatial vision problems.

Robust global translation averaging with feature tracks

How to average translations is the single most difficult task in global structure-from-motion (SfM) to fully tap its potentials in terms of reconstruction efficiency and accuracy since usually only noisy translation directions can be factored out from essential matrices due to the inevitable matching outliers. To tackle this problem, this work proposes a two-step strategy. Firstly, a “2-point method” is introduced to refine the epipolar geometry by which a more accurate track set is generated. Then, translation lengths are computed by solving a convex L1 optimization according to the adjacent triangles induced by the selected tracks and translations. Extensive experiments show that our method performs similarly or better than the state-of-art SfM approaches in terms of the reconstruction accuracy, completeness and efficiency.