Haiyan Fu

A balanced semi-supervised hashing method for CBIR

Hashing methods have attracted much attention in large scale image research in recent years, because they are not only fast, but also needing a little memory. This paper proposed a balanced semi-supervised hashing method by dividing image into several blocks. With the help of improved semi-supervised hashing, we obtain a short hash code of each block, which jointed together forms a hash code of an integrated image. In the improved semi-supervised hashing, the supervised information is completed by combining the similarity of image pairs and label information. Extensive experiments demonstrate that our method can get more balanced result between retrieval speed, saving storage of original data and retrieval accuracy in CBIR than the state-of-the-art hashing methods.

BHoG: binary descriptor for sketch-based image retrieval

Due to the popularity of devices with touch screens, it is convenient to match images with a hand-drawn sketch query. However, existing methods usually care little about memory space and time efficiency thus is inadequate for the rapid growth of multimedia resources. In this paper, a BHoG descriptor is proposed for sketch-based image retrieval. Firstly, the boundary image is detected from natural image using Berkeley boundary detector, and then divided into many blocks. Secondly, we calculate the gradient feature of each block, and find the principal gradient orientation. Finally, the principal gradient orientation is encoded to binary codes, which is proved to be efficient and discriminative. We evaluated the performance of BHoG on a large-scale social media dataset. The experimental results have shown that BHoG not only has a better performance on flexibility and efficiency, but also occupies small memory.

Feature extraction via multi-view non-negative matrix factorization with local graph regularization

Feature extraction is a crucial and difficult issue in pattern recognition tasks with the high-dimensional and multiple features. To extract the latent structure of multiple features without label information, multi-view learning algorithms have been developed. In this paper, motivated by manifold learning and multi-view Non-negative Matrix Factorization (NM-F), we introduce a novel feature extraction method via multi-view NMF with local graph regularization, where the inner-view relatedness between data is taken into consideration. We propose the matrix factorization objective function by constructing a nearest neighbor graph to integrate local geometrical information of each view and apply two iterative updating rules to effectively solve the optimization problem. In the experiment, we use the extracted feature to cluster several realistic datasets. The experimental results demonstrate the effectiveness of our proposed feature extraction approach.

Binary code reranking method with weighted hamming distance

Due to its compact binary codes and efficient search scheme, image hashing method is suitable for large-scale image retrieval. In image hashing methods, Hamming distance is used to measure similarity between two points. For K-bit binary codes, the Hamming distance is an int and bounded by K. Therefore, there are many returned images sharing the same Hamming distances with the query. In this paper, we propose two efficient image ranking methods, which are distance weights based reranking method (DWR) and bit importance based reranking method (BIR). DWR method aim to rerank PCA hash codes. DWR averages Euclidean distance of equal hash bits to these bits with different values, so as to obtain the weights of hash codes. BIR method is suitable for all type of binary codes. Firstly, feedback technology is adopted to detect the importance of each binary bit, and then big weights are assigned to important bits and small weights are assigned to minor bits. The advantage of this proposed method is calculation efficiency. Evaluations on two large-scale image data sets demonstrate the efficacy of our methods.

Semi-supervised learning based on group sparse for relative attributes

Relative attributes provide accurate information for image processing to describe which image is more natural, more open, etc. Robustness of relative attribute learning depends on the labeled comparative image pairs. However, manually labeling is a labor intensive and time-consuming task. In this paper, a semi-supervised learning approach based on group sparse is proposed to discover pairwise comparisons automatically. We generate an initial level division of the labeled training images for the basic of new constraints. Then, group sparse representation for the unlabeled images is introduced by embedding the level information into the dictionary. The semi-supervised process is conducted by selecting samples which have minimum reconstruction errors and adding new constraints to the model by comparing the selected ones with the samples in dictionary. Experiments on three public datasets demonstrate the effectiveness of our proposed method.

Aggregating hierarchical binary activations for image retrieval

Abstract Convolutional Neural Networks (CNNs) have achieved a breakthrough on a large number of image retrieval benchmarks. However, most previous works make use of the CNNs following the image classification strategy, where the last fully connected layer activations of the whole image are occupied as a single holistic feature vector. To improve the representation power of CNNs, this paper proposes a Multi-layer Fusion (MF) approach to aggregate deep activations for image retrieval task. The key insight of our approach is that different layers of a CNN are sensitive to specific patterns, and are complementary with each other for image representation. Specifically, our approach transforms CNN activations to deep binary codes embedded in the inverted index of Bag-of-Words structure for fast retrieval. Those activations are derived from multiple layers of a CNN on local patches, for features from orderless local areas have proved superior to global ones in the low level handcrafted cases. Corresponding weights and diffusion process are thereafter utilized to penalize and re-rank the individual similarity scores of layers. Our method is efficient, which extracts visual features from different layers only once. Furthermore, the proposed MF approach can be easily extended to include SIFT features to enhance the representation power. Extensive experiments on four public retrieval datasets quantitatively evaluate the effectiveness of our contributions, and the proposed algorithm prove to be the new state-of-the-art on the Holidays and UKBench datasets.

A loss combination based deep model for person re-identification

The Convolutional Neural Network (CNN) has significantly improved the state-of-the-art in person re-identification (re-ID). In the existing available identification CNN model, the softmax loss function is employed as the supervision signal to train the CNN model. However, the softmax loss only encourages the separability of the learned deep features between different identities. The distinguishing intra-class variations have not been considered during the training process of CNN model. In order to minimize the intra-class variations and then improve the discriminative ability of CNN model, this paper combines a new supervision signal with original softmax loss for person re-ID. Specifically, during the training process, a center of deep features is learned for each pedestrian identity and the deep features are subtracted from the corresponding identity centers, simultaneously. So that, the deep features of the same identity to the center will be pulled efficiently. With the combination of loss functions, the inter-class dispersion and intra-class aggregation can be constrained as much as possible. In this way, a more discriminative CNN model, which has two key learning objectives, can be learned to extract deep features for person re-ID task. We evaluate our method in two identification CNN models (i.e., CaffeNet and ResNet-50). It is encouraging to see that our method has a stable improvement compared with the baseline and yields a competitive performance to the state-of-the-art person re-ID methods on three important person re-ID benchmarks (i.e., Market-1501, CUHK03 and MARS).

Deep hashing with top similarity preserving for image retrieval

Hashing has drawn more and more attention in image retrieval due to its high search speed and low storage cost. Traditional hashing methods project the high-dimensional hand-crafted visual features to compact binary codes by linear or non-linear hashing functions. Deep hashing methods, which integrate image representation learning and hash functions learning into a unified framework, have shown more superior performance. Most of existing supervised deep hashing methods mainly consider the semantic similarities among images by using pair-wise or triplet-wise constraints as supervision information. However, as a kind of crucial information, the rankings of the retrieval results, are neglected. Consequently, the produced hash codes may be suboptimal. In this paper, a new Deep Hashing with Top Similarity Preserving (DHTSP) method is proposed to optimize the quality of hash codes for image retrieval. Specifically, we utilize AlexNet to extract discriminative image representations directly from the raw image pixels and learn hash functions simultaneously. Then a top similarity preserving loss function is designed to preserve the similarity of returned images at the top of the ranking list. Experimental results on three benchmark datasets show that our proposed method outperforms most of state-of-the-art deep hashing methods and traditional hashing methods.

Exploring geometric information in CNN for image retrieval

Convolutional Neural Network (CNN) has brought significant improvements for various multimedia tasks. In contrast, image retrieval has not yet benefited as much since no training database is available. In this paper, we propose an unsupervised weighting scheme for pre-trained CNN models to adaptively emphasize image center. Different from the general preference for fully connected layers which represent abstract semantics, we aggregate the activations of convolutional layers on image patches to depict local patterns in details. It is an empirical observation that the target of searching is naturally the focus of an image. Thus we pooling the features with respect to their positions, since they innately maintain the geometric layout of an image. Experimental results on two benchmarks prove the effectiveness of our methods.

A novel two-stream saliency image fusion CNN architecture for person re-identification

Background interference, which arises from complex environment, is a critical problem for a robust person re-identification (re-ID) system. The background noise may significantly compromise the feature learning and matching process. To reduce the background interference, this paper proposes a saliency image embedding as a pedestrian descriptor. First, to eliminate the background for each pedestrian image, the saliency image is constructed, which is implemented through an unsupervised manifold ranking-based saliency detection algorithm. Second, to reduce some errors and details missing of pedestrian during the saliency image construction process, a saliency image fusion (SIF) convolutional neural network (CNN) architecture is well designed, in which the original pedestrian image and saliency image are both employed as input. We implement our idea in the identification models based on some state-of-the-art backbone CNN models (i.e., CaffeNet, VGGNet-16, GoogLeNet and ResNet-50). We show that the learned pedestrian descriptor by the proposed SIF CNN architecture provides a significant improvement over the baselines and produces a competitive performance compared with the state-of-the-art person re-ID methods on three large-scale person re-ID benchmarks (i.e., Market-1501, DukeMTMC-reID and MARS).