Donglin Cao

Re-ranking Person Re-identification with k-Reciprocal Encoding

When considering person re-identification (re-ID) as a retrieval process, re-ranking is a critical step to improve its accuracy. Yet in the re-ID community, limited effort has been devoted to re-ranking, especially those fully automatic, unsupervised solutions. In this paper, we propose a k-reciprocal encoding method to re-rank the re-ID results. Our hypothesis is that if a gallery image is similar to the probe in the k-reciprocal nearest neighbors, it is more likely to be a true match. Specifically, given an image, a k-reciprocal feature is calculated by encoding its k-reciprocal nearest neighbors into a single vector, which is used for re-ranking under the Jaccard distance. The final distance is computed as the combination of the original distance and the Jaccard distance. Our re-ranking method does not require any human interaction or any labeled data, so it is applicable to large-scale datasets. Experiments on the large-scale Market-1501, CUHK03, MARS, and PRW datasets confirm the effectiveness of our method.

A cross-media public sentiment analysis system for microblog

Since classical public sentiment analysis systems for microblog are based on the text sentiment analysis, it is difficult to determine the sentiment of short text without clear sentiment words in microblog posts. Fortunately, a lot of microblog posts contain images which also represent users’ sentiment. To fully understand users’ sentiment, we propose a cross-media public sentiment analysis system for microblog. The best advantage of this novel system is the unified cross-media public sentiment analysis framework which fuses the text sentiment and image sentiment not only from sentiment results, but also from sentiment ontology. To enhance presentation effects, this system presents sentiment results from macroscopic view and microscopic view which details the sentiment results in region, topic, microblog content and user diffusion. In our knowledge, this is the first unified cross-media public sentiment analysis system.

Visual sentiment topic model based microblog image sentiment analysis

With a growing number of images being used to express opinions in Microblog, text based sentiment analysis is not enough to understand the sentiments of users. To obtain the sentiments implied in Microblog images, we propose a Visual Sentiment Topic Model (VSTM) which gathers images in the same Microblog topic to enhance the visual sentiment analysis results. First, we obtain the visual sentiment features by using Visual Sentiment Ontology (VSO); then, we build a Visual Sentiment Topic Model by using all images in the same topic; finally, we choose better visual sentiment features according to the visual sentiment features distribution in a topic. The best advantage of our approach is that the discriminative visual sentiment ontology features are selected according to the sentiment topic model. The experiment results show that the performance of our approach is better than VSO based model.

Multimodal hypergraph learning for microblog sentiment prediction

Microblog sentiment analysis has attracted extensive research attention in the recent literature. However, most existing works mainly focus on the textual modality, while ignore the contribution of visual information that contributes ever increasing proportion in expressing user emotions. In this paper, we propose to employ a hypergraph structure to formulate textual, visual and emoticon information jointly for sentiment prediction. The constructed hypergraph captures the similarities of tweets on different modalities where each vertex represents a tweet and the hyperedge is formed by the “centroid” vertex and its k-nearest neighbors on each modality. Then, the transductive inference is conducted to learn the relevance score among tweets for sentiment prediction. In this way, both intra- and inter- modality dependencies are taken into consideration in sentiment prediction. Experiments conducted on over 6,000 microblog tweets demonstrate the superiority of our method by 86.77% accuracy and 7% improvement compared to the state-of-the-art methods.

A two-level model for automatic image annotation

Image automatic annotation is a significant and challenging problem in pattern recognition and computer vision. Current image annotation models almost used all the training images to estimate joint generation probabilities between images and keywords, which would inevitably bring a lot of irrelevant images. To solve the above problem, we propose a hierarchical image annotation model which combines advantages of discriminative model and generative model. In first annotation layer, discriminative model is used to assign topic annotations to unlabeled images, and then relevant image set corresponding to each unlabeled image is obtained. In second annotation layer, we propose a keywords-oriented method to establish links between images and keywords, and then our iterative algorithm is used to expand relevant image sets. Candidate labels will be given higher weights by using our method based on visual keywords. Finally, generative model is used to assign detailed annotations to unlabeled images on expanded relevant image sets. Experiments conducted on Corel 5K datasets verify the effectiveness of our hierarchical image annotation model.

Hacking Chinese Touclick CAPTCHA by Multi-Scale Corner Structure Model with Fast Pattern Matching

In this paper, we tackle the challenge of hacking CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart), which is widely used to identify machine and human in webpage registration and authorization [17]. More specially, we target at touclick Chinese CAPTCHA that is recently popular in mobile application scenario. Hacking such CAPTCHA is much more challenging and left unexploited in the literature. Our main idea is a multi-scale Corner based Structure Model, termed CSM, with a very efficient pattern matching scheme. CSM can accurately capture the intrinsic statistics of touclick Chinese CAPTCHA against background clutters. We demonstrate the efficiency and effectiveness of the proposed approach by extensive experiments on a Chinese touclick CAPTCHA dataset collected from Internet forums. We report encouraging results with an overall success rate of almost 100% and an averaged detection speed of 170 millisecond. Upon our work, we also provide suggestions on improving the current CAPTCHA-based human-machine identification systems.

Class-specific object proposals re-ranking for object detection in automatic driving

Object proposal generation is an important step in object detection, obtaining high-quality proposals can effectively improve the performance of detection. In this paper, we propose a semantic, class-specific approach to re-rank object proposals, which can consistently improve the recall performance even with fewer proposals. Specifically, we first extract features for each proposal including semantic segmentation, stereo information, contextual information, CNN-based objectness and low-level cue, and then score them using class-specific weights learned by Structured SVM. The advantages of the proposed model are two-fold: 1) it can be easily merged to existing generators with few computational costs, and 2) it can achieve high recall rate under strict critical even using fewer proposals. Experimental evaluation on the KITTI benchmark demonstrates that our approach significantly improves existing popular generators on recall performance. Moreover, in the experiment conducted for object detection, even with 1500 proposals, our approach can still have higher average precision (AP) than baselines with 5000 proposals.

Cross-Modality Sentiment Analysis for Social Multimedia

Sentiment analysis is important for understanding the social media contents and user opinions. Along with the development of social media applications, an increasing number of people combine texts and images to express their opinions. However, text based sentiment analysis methods cannot process other medias except texts. Therefore, visual sentiment analysis is born at the right moment. In this article, we review two multimodal-based visual sentiment analysis models proposed in our group. Both model exploit the multimodal content from correlation and hyper graph view respectively. In the Multimodal Correlation Model (MCM), we observe the correlation among different modalities and model then through a probabilistic graphical model. In the Hyper graph Learning Model (HLM), we use hyper graph to model the independence of each modality. We further discuss the underneath challenges and foresee potential opportunities of this direction.

A new camera self-calibration method based on CSA

A large number of computer vision applications rely on camera calibration. Camera self-calibration which only depends on the relationship between corresponding points of a pair of images draws much attention for its simplicity. Almost all the camera self-calibration methods rely on the solution of Kruppa equations which are difficult to be directly solved. The state-of-the-art self-calibration algorithms usually convert the solution of these equations to non-linear optimization problem, traditional optimization methods usually have the drawback of convergent to local extreme. Artificial immune system (AIS) has the ability to fast convergent to global extreme. To address this problem, we proposed an artificial immune system based method which can fast convergent to the global optimization solutions. We demonstrate the performance of the proposed method with synthetic and real data.

Making intelligent business decisions by mining the implicit relation from bloggers' posts

More and more people use blogs to write down their ideas, opinions and individual thoughts. Mining blogs will obtain useful information, which can support business policy and decision-making, especially in analyzing product popularity. In this study, we mine the features of persons as implicit relations between the content of the bloggers’ posts and the bloggers. These kinds of implicit relations, which are also semantic relations, is called the Blogger Role. To mine this semantic relation, Wordnet is used to extract the Blogger Role and the features of Blogger Role. To get an appropriate Blogger Role, we cluster all bloggers’ posts and use the clustering result to revise the Blogger Role obtained by single document analysis. To get more relevant retrieval results, we combine this implicit relation with the classical retrieval model by the Blogger Role-based model. The combination is performed by the explicit model and implicit model. Results of experiments on TREC corpus show that Blogger Role reveals bloggers’ characters and mining Blogger Roles is useful in analyzing the popularity of products.