Tao Guan

On-Device Mobile Visual Location Recognition by Integrating Vision and Inertial Sensors

This paper deals with the problem of city scale on-device mobile visual location recognition by fusing the inertial sensors and computer vision techniques. The main contributions are as follows: Firstly, we design an efficient vector quantization strategy by combining the Transform Coding (TC) and Residual Vector Quantization (RVQ). Our method can compress a visual descriptor into only several bytes while providing reasonable searching accuracy, which makes the managing of city scale image database directly on mobile devices come true. Secondly, we integrate the information from inertial sensors into the Vector of Locally Aggregated Descriptors (VLAD) generation and image similarity evaluation processes. Our method is not only fast enough for on-device implementation, but it also can improve the location recognition accuracy obviously. Thirdly, we also release a set of 1.295 million geo-tagged street view images with the information from inertial sensors, as well as a difficult set of query images. These resources can be used as a new benchmark to facilitate further research in the area. Experimental results prove the validity of the proposed methods for on-device mobile visual location recognition applications.

Efficient BOF Generation and Compression for On-Device Mobile Visual Location Recognition

Existing mobile visual location recognition (MVLR) applications typically rely on bag-of-features (BOF) representation, which shows superior performance in retrieval accuracy. However, although the BOF framework is promising, it is not compact enough for on-device MVLR. The authors have made two contributions to the design of a BOF-based on-device MVLR system. First, to generate BOF descriptors, they propose a memory-efficient approximate nearest-neighbor search algorithm by combining residual vector quantization (RVQ) and tree-structured RVQ (TSRVQ). Second, they implemented a GPS-based and heading-aware RankBoost algorithm to reduce the dimensionality of the BOF descriptors. The authors evaluate the effectiveness of the proposed algorithms on an HTC mobile phone. Their work applies to on-device MVLR in city-scale workspaces.

Projected Residual Vector Quantization for ANN Search

In this research, we propose Projected Residual Vector Quantization (PRVQ) to deal with the problem of large-scale approximate nearest neighbor (ANN) search in a high-dimensional space. A lot of quantization-based ANN search algorithms have been proposed in the past few years. However, most of the existing methods discard the projection errors generated in the dimension reduction process, which inevitably decreases the search accuracy. In view of that, the authors propose a method of projected residual vector quantization for ANN search that considers the projection errors in the quantization process. They also design three simple and effective optimization strategies to improve the performance of the PRVQ algorithm. The authors have integrated the proposed PRVQ algorithm into a mobile landmark recognition system to prove its effectiveness.

Inertial sensors supported visual descriptors encoding and geometric verification for mobile visual location recognition applications

In this paper, two contributions are introduced to improve the accuracy of city scale on-device Mobile Visual Location Recognition (MVLR) systems. Firstly, to compress image descriptors, we design an improved Transform Coding (TC) algorithm for the implementation of a location aware encoding strategy. Compared with traditional encoding algorithms, our algorithm can provide reasonable searching accuracy with very low memory consumption, which makes the implementation of the location aware image descriptors encoding directly on a mobile device come true. Secondly, to perform Geometric Verification (GV) directly on a mobile device, we design a gravity aligned geometric encoding algorithm. The algorithm is not only memory and computation efficient, but also can improve the location recognition accuracy obviously. Experiments on city scale datasets demonstrate the effectiveness of the proposed algorithms. We investigate the problem of city scale on device mobile visual search applications.We propose an improved transform coding algorithm to compress city scale datasets.A new algorithm is designed to perform Geometric Verification on a mobile device.

Affection arousal based highlight extraction for soccer video

Highlight extraction, as one of the key technologies in soccer video retrieval and summarization, has great academic and application value. According to the principle that the observer’s affection state would fluctuate with the evolution of game process when watching soccer match video, a novel highlight extraction approach based on the improved affection arousal model is proposed. Compared with the existing works, our main contributions include the following. A novel feature – shot intensity is exploited to replace the motion activity, which greatly improves the computational performance of affection arousal model. Another helpful feature – replay factor is designed and successfully fused into the affection arousal model. This makes the affection arousal model reflect the variation of the true match process more accurately. In addition, event temporal transition pattern (ETTP) in soccer video is utilized to detect highlights boundaries effectively combined with the affection arousal curve. Experiments conducted on real-world soccer game videos have demonstrated the efficiency and effectiveness of the proposed approach.

Fast terrain mapping from low altitude digital imagery

We present a linear time Real Terrain Reconstruction (RTR) framework for fixed-wing micro aerial vehicles (MAVs) in this paper. Single-shot aerial images labeled with GPS and IMU signals are acquired by a fixed-wing MAV in several flights. Then these images are fed into our structure from motion (SfM) processing to generate accuracy pose estimation and 3D points. RTR improves existing state of the art algorithms VisualSFM 1] in multiaspect so as to make it more suitable for large-scale terrain reconstruction from aerial imagery. Firstly, we present a novel strategy of combining signals from airborne sensors (GPS/IMU) with the traditional SfM method, which can improve speed and accuracy of pose estimation observably. Secondly, a delayed aerial triangular method is designed to reconstruct a point visible in more than two cameras with an appropriate baseline. Thirdly, we also release 5 aerial imagery datasets which contain over 15 thousands images totally with the detailed MAV pose information from airborne sensors (GPS/IMU). These resources can be used as a new benchmark to facilitate further research in the area. We test our algorithm on these aerial image sets with various settings, and show that RTR offers state of the art performance for large-scale terrain reconstructions.

High-dimensional indexing technologies for large scale content-based image retrieval: a review

The boom of Internet and multimedia technology leads to the explosion of multimedia information, especially image, which has created an urgent need of quickly retrieving similar and interested images from huge image collections. The content-based high-dimensional indexing mechanism holds the key to achieving this goal by efficiently organizing the content of images and storing them in computer memory. In the past decades, many important developments in high-dimensional image indexing technologies have occurred to cope with the ‘curse of dimensionality’. The high-dimensional indexing mechanisms can mainly be divided into three categories: tree-based index, hashing-based index, and visual words based inverted index. In this paper we review the technologies with respect to these three categories of mechanisms, and make several recommendations for future research issues.

Registration Based on Scene Recognition and Natural Features Tracking Techniques for Wide-Area Augmented Reality Systems

This research focuses on designing a robust and flexible registration method for wide-area augmented reality applications using scene recognition and natural features tracking techniques. Instead of building a global map of the wide-area scene, we propose to partition the whole scene into several sub-maps according to the users preference or the requirements of the augmented reality (AR) applications. Random classification trees are used to learn and recognize the reconstructed scenes because they naturally handle multi-class problems, while being both robust and fast. The result is a system that can deal with large scale scene that previous methods cannot cope with. We also propose a hybrid natural features tracking strategy combining both wide and narrow baseline techniques. While providing seamless registration, our system can recover from registration failures and switch between different sub-maps automatically. Experimental results demonstrate the validity of the proposed method for wide-area augmented reality applications.

Dense 3D reconstruction combining depth and RGB information

Dense 3D reconstruction has important applications in many fields. The existing depth information based methods are typically constrained in their effective camera-object distance which should be from 0.4m to 4m. We present a novel method that can achieve a more accurate dense 3D reconstruction with an RGB-D camera when the distance between the camera and object is less than 0.4m, which enlarges the application range. Our approach combines a depth information based 3D model method with a RGB information based method to refine the reconstruction results when the camera fails to acquire the correct depth information. Rich RGB information captured from a color camera along with feature detection and triangulation methods are used to obtain accurate camera poses and 3D points when the camera is close to the object. Compared with the reconstruction results obtained from depth information only, quantitative experimental results show that our method is more effective, particularly when the camera is close to the object in the scene.

Optimized residual vector quantization for efficient approximate nearest neighbor search

In this paper, an optimized residual vector quantization-based approach is presented for improving the quality of vector quantization and approximate nearest neighbor search. The main contributions are as follows. Based on residual vector quantization (RVQ), a joint optimization process called enhanced RVQ (ERVQ) is introduced. Each stage codebook is iteratively optimized by the others aiming at minimizing the overall quantization errors. Thus, an input vector is approximated by its quantization outputs more accurately. Consequently, the precision of approximate nearest neighbor search is improved. To efficiently find nearest centroids when quantizing vectors, a non-linear vector quantization method is proposed. The vectors are embedded into 2-dimensional space where the lower bounds of Euclidean distances between the vectors and centroids are calculated. The lower bound is used to filter non-nearest centroids for the purpose of reducing computational costs. ERVQ is noticeably optimized in terms of time efficiency on quantizing vectors when combining with this method. To evaluate the accuracy that vectors are approximated by their quantization outputs, an ERVQ-based exhaustive method for approximate nearest neighbor search is implemented. Experimental results on three datasets demonstrate that our approaches outperform the state-of-the-art methods over vector quantization and approximate nearest neighbor search.