Richang Hong

Learning Visual Semantic Relationships for Efficient Visual Retrieval

In this paper, we investigate how to establish the relationship between semantic concepts based on the large-scale realworld click data from image commercial engine, which is a challenging topic because the click data suffers from the noise such as typos, the same concept with different queries, etc. We first define five specific relationships between concepts. We then extract some concept relationship features in textual and visual domain to train the concept relationship models. The relationship of each pair of concepts will thus be classified into one of the five special relationships. We study the efficacy of the conceptual relationships by applying them to augment imperfect image tags, i.e., improve representative power. We further employ a sophisticated hashing approach to transform augmented image tags into binary codes, which are subsequently used for content-based image retrieval task. Experimental results on NUS-WIDE dataset demonstrate the superiority of our proposed approach as compared to state-of-the-art methods.

Unified Photo Enhancement by Discovering Aesthetic Communities From Flickr

Photo enhancement refers to the process of increasing the aesthetic appeal of a photo, such as changing the photo aspect ratio and spatial recomposition. It is a widely used technique in the printing industry, graphic design, and cinematography. In this paper, we propose a unified and socially aware photo enhancement framework which can leverage the experience of photographers with various aesthetic topics (e.g., portrait and landscape). We focus on photos from the image hosting site Flickr, which has 87 million users and to which more than 3.5 million photos are uploaded daily. First, a tagwise regularized topic model is proposed to describe the aesthetic topic of each Flickr user, and coherent and interpretable topics are discovered by leveraging both the visual features and tags of photos. Next, a graph is constructed to describe the similarities in aesthetic topics between the users. Noticeably, densely connected users have similar aesthetic topics, which are categorized into different communities by a dense subgraph mining algorithm. Finally, a probabilistic model is exploited to enhance the aesthetic attractiveness of a test photo by leveraging the photographic experiences of Flickr users from the corresponding communities of that photo. Paired-comparison-based user studies show that our method performs competitively on photo retargeting and recomposition. Moreover, our approach accurately detects aesthetic communities in a photo set crawled from nearly 100000 Flickr users.

Coherent Semantic-Visual Indexing for Large-Scale Image Retrieval in the Cloud

The rapidly increasing number of images on the internet has further increased the need for efficient indexing for digital image searching of large databases. The design of a cloud service that provides high efficiency but compact image indexing remains challenging, partly due to the well-known semantic gap between user queries and the rich semantics of large-scale data sets. In this paper, we construct a novel joint semantic-visual space by leveraging visual descriptors and semantic attributes, which narrows the semantic gap by combining both attributes and indexing into a single framework. Such a joint space embraces the flexibility of coherent semantic-visual indexing, which employs binary codes to boost retrieval speed while maintaining accuracy. To solve the proposed model, we make the following contributions. First, we propose an interactive optimization method to find the joint semantic and visual descriptor space. Second, we prove convergence of our optimization algorithm, which guarantees a good solution after a certain number of iterations. Third, we integrate the semantic-visual joint space system with spectral hashing, which finds an efficient solution to search up to billion-scale data sets. Finally, we design an online cloud service to provide a more efficient online multimedia service. Experiments on two standard retrieval datasets (i.e., Holidays1M, Oxford5K) show that the proposed method is promising compared with the current state-of-the-art and that the cloud system significantly improves performance.

Computational Social Indicators: A Case Study of Chinese University Ranking

Many professional organizations produce regular reports of social indicators to monitor social progress. Despite their reasonable results and societal value, early efforts on social indicator computing suffer from three problems: 1) labor-intensive data gathering, 2) insufficient data, and 3) expert-relied data fusion. Towards this end, we present a novel graph-based multi-channel ranking scheme for social indicator computation by exploring the rich multi-channel Web data. For each channel, this scheme presents the semi-structured and unstructured data with simple graphs and hypergraphs, respectively. It then groups the channels into different clusters according to their correlations. After that, it uses a unified model to learn the cluster-wise common spaces, perform ranking separately upon each space, and fuse these rankings to produce the final one. We take Chinese university ranking as a case study and validate our scheme over a real-world dataset. It is worth emphasizing that our scheme is applicable to computation of other social indicators, such as Educational attainment.

A Spatial-Temporal Probabilistic Matrix Factorization Model for Point-of-Interest Recommendation.

With the rapid development of Location-based Social Network (LBSN) services, a large number of Point-of-Interests (POIs) have been available, which consequently raises a great demand of building personalized POI recommender systems. A personalized POI recommender system can significantly help users to find their preferred POIs and assist POI owners to attract more customers. However, due to the complexity of users’ checkin decision making process that is influenced by many different factors such as POI distance and region’s prosperity, and the dynamics of user’s preference, POI recommender systems usually suffer from many challenges. Although different latent factor based methods (e.g., probabilistic matrix factorization) have been proposed, most of them do not successfully incorporate both geographical influence and temporal effect together into latent factor models. To this end, in this paper, we propose a new Spatial-Temporal Probabilistic Matrix Factorization (STPMF) model that models a user’s preference for POI as the combination of his geographical preference and other general interest in POI. Furthermore, in addition to static general interest of user, we capture the temporal dynamics of user’s interest as well by modeling checkin data in a unique way. To evaluate the proposed STPMF model, we conduct extensive experiments with many state-of-the-art baseline methods and evaluation metrics on two real-world data sets. The experimental results clearly demonstrate the effectiveness of our proposed STPMF model.

Generative Models for Mining Latent Aspects and Their Ratings from Short Reviews

A large number of online reviews have been accumulated on the Web, such as Amazon.com and Cnet.com. It is increasingly challenging to digest these reviews for both consumers and firms as the volume of reviews increases. A promising direction to ease such a burden is to automatically identify aspects of a product and reveal each individuals ratings on them from these reviews. The identified and rated aspects can help consumers understand the pros and cons of a product and make their purchase decisions, and help firms learn user feedbacks and improve their products and marketing strategy. While different methods have been introduced to tackle this problem in the past, few of them successfully model the intrinsic connection between aspect and aspect rating particularly in short reviews. To this end, in this paper, we first propose the Aspect Identification and Rating (AIR) model to model observed textual reviews and overall ratings in a generative way, where the sampled aspect rating influences the sampling of sentimental words on this aspect. Furthermore, we enhance AIR model to particularly address one unique characteristic of short reviews that aspects mentioned in reviews may be quite unbalanced, and develop another model namely AIRS. Within AIRS model, we allow an aspect to directly affect the sampling of a latent rating on this aspect in order to capture the mutual influence between aspect and aspect rating through the whole generative process. Finally, we examine our two models and compare them with other methods based on multiple real world data sets, including hotel reviews, beer reviews and app reviews. Experimental results clearly demonstrate the effectiveness and improvement of our models. Other potential applications driven by our results are also shown in the experiments.

Product Adoption Rate Prediction in a Competitive Market

As the worlds of commerce and the Internet technology become more inextricably linked, a large number of user consumption series become available for online market intelligence analysis. A critical demand along this line is to predict the future product adoption state of each user, which enables a wide range of applications such as targeted marketing. Nevertheless, previous works only aimed at predicting if a user would adopt a particular product or not with a binary buy-or-not representation. The problem of tracking and predicting users’ adoption rates, i.e., the frequency and regularity of using each product over time, is still under-explored. To this end, we present a comprehensive study of product adoption rate prediction in a competitive market. This task is nontrivial as there are three major challenges in modeling users’ complex adoption states: the heterogeneous data sources around users, the unique user preference and the competitive product selection. To deal with these challenges, we first introduce a flexible factor-based decision function to capture the change of users’ product adoption rate over time, where various factors that may influence users’ decisions from heterogeneous data sources can be leveraged. Using this factor-based decision function, we then provide two corresponding models to learn the parameters of the decision function with both generalized and personalized assumptions of users’ preferences. We further study how to leverage the competition among different products and simultaneously learn product competition and users’ preferences with both generalized and personalized assumptions. Finally, extensive experiments on two real-world datasets show the superiority of our proposed models.