Zongmin Li

Integrating multi-level deep learning and concept ontology for large-scale visual recognition

Abstract To support large-scale visual recognition (i.e., recognizing thousands or even tens of thousands of object classes), a multi-level deep learning algorithm is developed to learn multiple deep networks and a tree classifier jointly, where a concept ontology is constructed to organize large numbers of object classes hierarchically in a coarse-to-fine fashion and determine the inter-related learning tasks automatically. Our multi-level deep learning algorithm can: (a) train multiple deep networks simultaneously to achieve more discriminative representations of both coarse-grained groups and fine-grained object classes at different levels of the concept ontology (i.e., learning multiple sets of deep features simultaneously for different tasks); (b) leverage multi-task learning to train more discriminative classifiers for the fine-grained object classes in the same group to enhance their separability significantly and enable inter-class knowledge transferring; and (c) learn multiple deep networks and the tree classifier jointly in an end-to-end fashion. Our experimental results on three image sets have demonstrated that our multi-level deep learning algorithm can achieve very competitive results on both the accuracy rates and the computational efficiency for large-scale visual recognition.

Evidence-based SVM fusion for 3D model retrieval

Many existing 3D model retrieval use KNN (k-nearest neighbor) method for similarity search, but it is inefficient in high-dimension space search. In this paper, the classification tools are integrated for supporting more effective 3D model search in the high-dimensional feature space. Our proposed algorithm used multiple SVM classifiers to predict 3D models for a given query and D-S Evidence theory is used to fuse all the prediction results. Experimental results show that our proposed 3D model retrieval algorithm can improve the accuracy significantly compared with the traditional kNN method.

Extended cone-curvature based salient points detection and 3D model retrieval

Local feature extraction of 3D model has become a more and more important aspect in terms of 3D model shape feature extraction. Compared with the global feature, it is more suitable to do the partial retrieval and more robust to the model deformation. In this paper, a local feature called extended cone-curvature feature is proposed to describe the local shape feature of 3D model mesh. Based on the extended cone-curvature feature, salient points and salient regions are extracted by using a new salient point detection method. Then extended cone-curvature feature and local shape distribution feature calculated on the salient regions are used together as shape index, and the earth mover’s distance is employed to accomplish similarity measure. After many times’ retrieval experiments, the new extended cone-curvature descriptor we propose has more efficient and effective performance than shape distribution descriptor and light field descriptor especially on deformable model retrieval.

Deep Multi-task Learning for Large-Scale Image Classification

To achieve more effective solution for large-scale image classification (i.e., classifying millions of images into thousands or even tens of thousands of object classes or categories), a deep multi-task learning algorithm is developed by seamlessly integrating deep CNNs with multi-task learning over the concept ontology, where the concept ontology is used to organize large numbers of object classes or categories hierarchically and determine the inter-related learning tasks automatically. Our deep multi-task learning algorithm can integrate the deep CNNs to learn more discriminative high-level features for image representation, and it can also leverage multi-task learning and inter-level relationship constraint to train more discriminative tree classifier over the concept ontology and control the inter-level error propagation effectively. In our deep multi-task learning algorithm, we can use back propagation to simultaneously refine both the relevant node classifiers (at different levels of the concept ontology) and the deep CNNs according to a joint objective function. The experimental results have demonstrated that our deep multi-task learning algorithm can achieve very competitive results on both the accuracy and the cost of feature extraction for large-scale image classification.

Discovering the Latent Similarities of the KNN Graph by Metric Transformation

The manifold of the dataset turns out to be quite useful in refining the retrieval results, and the diffusion process provides an efficient solution by careful selection of the similarity neighborhood which is usually modeled as the K-nearest neighborhood (KNN) graph. However, existing works are sensitive to the topology noises induced by the first K neighbors. In this paper, we tackle the problem by studying metric transformation which aims at finding new functional relationship to dig the latent similarity. The advantage of the approach lies in its robustness towards the varying K values; that is to say, it could preserve high similarity performances even if K is very large. Except for discussing only the global KNN (i.e. the same K for all neighborhoods) graph, we also investigate to specify a different K for each neighborhood by incorporating the new penalized consensus information (PCI). We show that PCI works superior compared with the original consensus information for denoising. Experiments on multiple affinity matrices have corroborated the superiority of our method with surprising good results.

TOP-SIFT: A New Method for SIFT Descriptor Selection

The large amount of SIFT descriptors in an image and the high dimensionality of SIFT descriptor has made problems for large-scale image dataset in terms of speed and scalability. In this paper, we propose a descriptor selection algorithm via dictionary learning and only a small set of features are reserved, which we refer to as TOP-SIFT. We discover the inner relativity between the problem of descriptor selection and dictionary learning for sparse representation, and then turn our problem into dictionary learning. Compared with the earlier methods, our method is neither relying on the dataset nor losing important information, and the experiments have shown that our algorithm can save memory space and increase the retrieval speed efficiently while maintain the recognition performance as well.

TOP-SIFT: the selected SIFT descriptor based on dictionary learning

The large amount of SIFT descriptors in an image and the high dimensionality of SIFT descriptor have made problems for the large-scale image database in terms of speed and scalability. In this paper, we present a descriptor selection algorithm based on dictionary learning to remove the redundant features and reserve only a small set of features, which we refer to as TOP-SIFTs. During the experiment, we discovered the inner relativity between the problem of descriptor selection and dictionary learning in sparse representation, and then turned our problem into dictionary learning. We designed a new dictionary learning method to adapt our problem and employed the simulated annealing algorithm to obtain the optimal solution. During the process of learning, we added the sparsity constraint and spatial distribution characteristic of SIFT points. And lastly selected the small representative feature set with good spatial distribution. Compared with the earlier methods, our method is neither relying on the database nor losing important information, and the experiments have shown that our algorithm can save memory space a lot and increase time efficiency while maintaining the accuracy as well.

Automatic Privacy Prediction to Accelerate Social Image Sharing

The manual process for privacy setting could be very time-consuming and challenging for common users. By assuming that there are hidden correlations between the visual properties of images (i.e., visual features) or object classes and the privacy settings for image sharing, an effective algorithm is developed in this paper to achieve automatic prediction of image privacy, so that the best-matching privacy setting can be recommended automatically for each single image being shared. Our algorithm for automatic image privacy prediction contains two approaches: (a) feature-based approach by learning more representative deep features and discriminative classifier for assigning each single image being shared into one of two categories: private vs. public, (b) object-based approach by detecting large numbers of privacy-sensitive object classes and events automatically and leveraging them to achieve more discriminative characterization of image privacy, so that we can support more explainable solution for automatic image privacy prediction. We have also conducted extensive experimental studies on large-scale social images, which have demonstrated both efficiency and effectiveness of our proposed algorithm.

The Multi-layer Spherical Helix Curve-Based 3-D Fourier Feature

This paper proposes an extended 3-D shape retrieval feature, which is based on a new scheme. In our method, continuous principal component analysis is used to align 3-D models into a canonical position. We then represent 3-D polygonal mesh as a 3-D multi-layer closed curve. It can capture the more feature of the 3-D models. The feature of the 3-D model is extracted from the 3-D closed curve by the Fourier transform. And the experiments show it is more efficient.

3D models’ indexing and compression storage in digital library

3D digital library is a hotspot in recent years. With the continuing development of 3D graphics technology, 3D model as the fourth-generation media has been one of the digital resources in digital library. This article analyzes the progress of two directions: 3D model indexing and compression. At last, we propose a simple frame of 3D model indexing and compression in digital library.