Shanmin Pang

Democratic Diffusion Aggregation for Image Retrieval

Content-based image retrieval is an important research topic in the multimedia field. In large-scale image search using local features, image features are encoded and aggregated into a compact vector to avoid indexing each feature individually. In the aggregation step, sum-aggregation is wildly used in many existing works and demonstrates promising performance. However, it is based on a strong and implicit assumption that the local descriptors of an image are identically and independently distributed in descriptor space and image plane. To address this problem, we propose a new aggregation method named democratic diffusion aggregation (DDA) with weak spatial context embedded. The main idea of our aggregation method is to re-weight the embedded vectors before sum-aggregation by considering the relevance among local descriptors. Different from previous work, by conducting a diffusion process on the improved kernel matrix, we calculate the weighting coefficients more efficiently without any iterative optimization. Besides considering the relevance of local descriptors from different images, we also discuss an efficient query fusion strategy which uses the initial top-ranked image vectors to enhance the retrieval performance. Experimental results show that our aggregation method exhibits much higher efficiency (about × 14 faster) and better retrieval accuracy compared with previous methods, and the query fusion strategy consistently improves the retrieval quality.

Exploiting local linear geometric structure for identifying correct matches

Abstract Selecting correct matches from a set of tentative feature point correspondences plays a vital important role in many tasks, such as structure from motion (SfM), wide baseline stereo and image search. In this paper, we propose an efficient and effective method for identifying correct matches from an initial batch of feature correspondences. The proposed method first obtains a subset of correct matches based on the assumption that the local geometric structure among a feature point and its nearest neighbors in an image cannot be easily affected by both geometric and photometric transformations, and thus should be observed in the matched images. For efficiency, we model this local geometric structure by a set of linear coefficients that reconstruct the point from its neighbors. After obtaining a portion of correct matches, we then provide two ways to accurately estimate the correctness of each match and to efficiently estimate the number of correct matches, respectively. The proposed method is evaluated on both applications including image matching and image re-ranking. Experimental results on several public datasets show that our method outperforms state-of-the-art techniques in terms of speed and accuracy.

Image re-ranking with an alternating optimization

In this work, we propose an efficient image re-ranking method, without regional information of each indexed feature stored in the inverted file, to re-rank all retrieved images. The motivation of the proposed method is that, there are usually many visual words in the query image that only give votes to irrelevant images. With this observation, we propose to use visual words which are actually useful in finding relevant images to re-rank the retrieved images. To achieve this goal, we first initialize some images similar to the query by maximizing a quadratic function when giving an initial ranking of the retrieved images. The quadratic function is constructed by storing the similarities among a short-list of top ranked images into an affinity matrix, where the similarity between any two images is computed by the proposed graph diffusion. Then, we select a subset of visual words in the query with an alternating optimization strategy: (1) at each iteration, visual words are selected based on the set of similar images that we have found, (2) and in turn, the set of similar images is updated with the set of selected words. These two steps are repeated until convergence. Experimental results on standard benchmark datasets show that the proposed method achieves an order of magnitude speedups over the state-of-the-art spatial based re-ranking techniques, and obtains much better retrieval quality as well.

Fast Democratic Aggregation and Query Fusion for Image Search

In image search using local features, to avoid indexing each feature individually, encoding methods are popularly adopted to embed and aggregate local features of an image into a compact vector. Democratic aggregation with triangulation embedding (T-embedding) exhibits significant retrieval accuracy improvement over previous works. However, it suffers high computational complexity. To address this problem and consistently improve the retrieval performance, we propose a new democratic method to accelerate aggregating step without accuracy lost. We also embed weak spatial context in the kernel construction to depress co-occurrence caused by local feature detector. Furthermore, we enhance the retrieval performance with an efficient query fusion strategy. The evaluation on public datasets shows that our democratic aggregation is an order of magnitude faster than the original democratic aggregation with comparable retrieval accuracy, and the query fusion achieves a significant accuracy improvement over previous works.

Isometric hashing for image retrieval

Abstract Hashing has been attracting much attention in computer vision recently, since it can provide efficient similarity comparison in massive multimedia databases with fast query speed and low storage cost. Since the distance metric is an explicit description of similarity, in this paper, a novel hashing method is proposed for image retrieval, dubbed Isometric Hashing (IH). IH aims to minimize the difference between the distance in input space and the distance of the corresponding binary codes. To tackle the discrete optimization in a computationally tractable manner, IH adopts some mathematical tricks to transform the original problem into a multi-objective optimization problem. The usage of linear-projection-based hash functions enables efficient generating hash codes for unseen data points. Furthermore, utilizing different distance metrics could produce corresponding hashing algorithms, thus IH can be seen as a framework for developing new hashing methods. Extensive experiments performed on four benchmark datasets validate that IH can achieve comparable to or even better results than some state-of-the-art hashing methods.

Locality preserving verification for image search

Establishing correct correspondences between two images has a wide range of applications, such as 2D and 3D registration, structure from motion, and image retrieval. In this paper, we propose a new matching method based on spatial constraints. The proposed method has linear time complexity, and is efficient when applying it to image retrieval. The main assumption behind our method is that, the local geometric structure among a feature point and its neighbors, is not easily affected by both geometric and photometric transformations, and thus should be preserved in their corresponding images. We model this local geometric structure by linear coefficients that reconstruct the point from its neighbors. The method is flexible, as it can not only estimate the number of correct matches between two images efficiently, but also determine the correctness of each match accurately. Furthermore, it is simple and easy to be implemented. When applying the proposed method on re-ranking images in an image search engine, it outperforms the-state-of-the-art techniques.

Large-scale vocabularies with local graph diffusion and mode seeking

Abstract In this work, we propose a large-scale clustering method that captures the intrinsic manifold structure of local features by graph diffusion for image retrieval. The proposed method is a mode seeking like algorithm, and it finds the mode of each data point with the defined stochastic matrix resulted by a same local graph diffusion process. While mode seeking algorithms are normally costly, our method is efficient to generate large-scale vocabularies as it is not iterative, and the major computational steps are done in parallel. Furthermore, unlike other clustering methods, such as k-means and spectral clustering, the proposed clustering algorithm does not need to empirically appoint the number of clusters beforehand, and its time complexity is independent on the number of clusters. Experimental results on standard image retrieval datasets demonstrate that the proposed method compares favorably to previous large-scale clustering methods.

Beyond Sum and Weighted Aggregation: An Efficient Mixed Aggregation Method with Multiple Weights for Image Search

Image search with local descriptors represents an image usually by embedding and aggregating a set of patch descriptors into a single vector. Standard aggregation operations include sum and weighted aggregations. While showing high efficiency, sum aggregation lacks discriminative power. In contrast, weighted aggregation shows promising retrieval performance but suffers extremely high time cost. In this paper, we present a general mixed aggregation method that unifies sum and weighted aggregation methods. Owing to its general formulation, our method is able to balance the trade-off between quality and speed. Furthermore, we propose to compute multiple weighting coefficients rather than one for each to be aggregated vector by partitioning it into several components. Experimental results demonstrate that, while showing over ten times speedup over baselines, the image search frameworks with our mixed aggregation method achieve the state-of-the-art performance. Inspired by our aggregation method, we also present a new embedding strategy. Different from the existing embedding methods that individually map each descriptor into a single embedded vector, our embedding method maps a group of local descriptors into a single vector, which significantly benefits the aggregation step in terms of speed. As demonstrated by the experiments, the retrieval frameworks with our embedding method are more than fifty times faster than baselines, while maintaining competitive retrieval performance.

Image Re-ranking with an Alternating Optimization

In this work, we propose an efficient image re-ranking method, without additional memory cost compared with the baseline method~\cite{philbin2007object}, to re-rank all retrieved images. The motivation of the proposed method is that, there are usually many visual words in the query image that only give votes to irrelevant images. With this observation, we propose to only use visual words which can help to find relevant images to re-rank the retrieved images. To achieve the goal, we first find some similar images to the query by maximizing a quadratic function when given an initial ranking of the retrieved images. Then we select query visual words with an alternating optimization strategy: (1) at each iteration, select words based on the similar images that we have found and (2) in turn, update the similar images with the selected words. These two steps are repeated until convergence. Experimental results on standard benchmark datasets show that the proposed method outperforms spatial based re-ranking methods.

Building discriminative CNN image representations for object retrieval using the replicator equation

Abstract We present a generic unsupervised method to increase the discriminative power of image vectors obtained from a broad family of deep neural networks for object retrieval. This goal is accomplished by simultaneously selecting and weighting informative deep convolutional features using the replicator equation, commonly used to capture the essence of selection in evolutionary game theory. The proposed method includes three major steps: First, efficiently detecting features within Regions of Interest (ROIs) using a simple algorithm, as well as trivially collecting a subset of background features. Second, assigning unassigned features by optimizing a standard quadratic problem using the replicator equation. Finally, using the replicator equation again in order to partially address the issue of feature burstiness. We provide theoretical time complexity analysis to show that our method is efficient. Experimental results on several common object retrieval benchmarks using both pre-trained and fine-tuned deep networks show that our method compares favorably to the state-of-the-art. We also publish an easy-to-use Matlab implementation of the proposed method for reproducing our results.