Jitao Sang

Exploiting Social-Mobile Information for Location Visualization

With a smart phone at hand, it becomes easy now to snap pictures and publish them online with few lines of texts. The GPS coordinates and User-Generated Content (UGC) data embedded in the shared photos provide opportunities to exploit important knowledge to tackle interesting tasks like geographically organizing photos and location visualization. In this work, we propose to organize photos both geographically and semantically, and investigate the problem of location visualization from multiple semantic themes. The novel visualization scheme provides a rich display landscape for geographical exploration from versatile views. A two-level solution is presented, where we first identify the highly photographed places of interest (POI) and discover their focused themes, and then aggregate the lower-level POI themes to generate the higher-level city themes for location visualization. We have conducted experiments on crawled Flickr and Instagram data and exhibited the visualization for the cities of Singapore and Sydney. The experimental results have validated the proposed method and demonstrated the potentials of location visualization from multiple themes.

Folksonomy-Based Visual Ontology Construction and Its Applications

An ontology hierarchically encodes concepts and concept relationships, and has a variety of applications such as semantic understanding and information retrieval. Previous work for building ontologies has primarily relied on labor-intensive human contributions or focused on text-based extraction. In this paper, we consider the problem of automatically constructing a folksonomy-based visual ontology (FBVO) from the user-generated annotated images. A systematic framework is proposed consisting of three stages as concept discovery, concept relationship extraction, and concept hierarchy construction. The noisy issues of the user-generated tags are carefully addressed to guarantee the quality of derived FBVO. The constructed FBVO finally consists of 139 825 concept nodes and millions of concept relationships by mining more than 2.4 million Flickr images. Experimental evaluations show that the derived FBVO is of high quality and consistent with human perception. We further demonstrate the utility of the derived FBVO in applications of complex visual recognition and exploratory image search.

A Unified Video Recommendation by Cross-Network User Modeling

Online video sharing sites are increasingly encouraging their users to connect to the social network venues such as Facebook and Twitter, with goals to boost user interaction and better disseminate the high-quality video content. This in turn provides huge possibilities to conduct cross-network collaboration for personalized video recommendation. However, very few efforts have been devoted to leveraging users’ social media profiles in the auxiliary network to capture and personalize their video preferences, so as to recommend videos of interest. In this article, we propose a unified YouTube video recommendation solution by transferring and integrating users’ rich social and content information in Twitter network. While general recommender systems often suffer from typical problems like cold-start and data sparsity, our proposed recommendation solution is able to effectively learn from users’ abundant auxiliary information on Twitter for enhanced user modeling and well address the typical problems in a unified framework. In this framework, two stages are mainly involved: (1) auxiliary-network data transfer, where user preferences are transferred from an auxiliary network by learning cross-network knowledge associations; and (2) cross-network data integration, where transferred user preferences are integrated with the observed behaviors on a target network in an adaptive fashion. Experimental results show that the proposed cross-network collaborative solution achieves superior performance not only in terms of accuracy, but also in improving the diversity and novelty of the recommended videos.

Bundled Local Features for Image Representation

Local features have been widely used for image representation. Traditional methods often treat each local feature independently or simply model the correlations of local features with spatial partition. However, local features are correlated and should be jointly modeled. Besides, due to the variety of images, predefined partition rules will probably introduce noisy information. To solve these problems, in this paper we propose a novel bundled local features method for efficient image representation and apply it for classification. Specially, we first extract local features and bundle them together with over-complete spatial shapes by viewing each local feature as the central point. Then, the most discriminatively bundling features are selected by reconstruction error minimization. The encoding parameters are then used for image representations in a matrix form. Finally, we train bi-linear classifiers with quadratic hinge loss to predict the classes of images. The proposed method can combine local features appropriately and efficiently for discriminative representations. Experimental results on three image data sets show the effectiveness of the proposed method compared with other local features combination strategies.

Demographic Attribute Inference from Social Multimedia Behaviors: a Cross-OSN Approach

This study focuses on exploiting the dynamic social multimedia behaviors to infer the stable demographic attributes. Existing demographic attribute inference studies are devoted to developing advanced features/models or exploiting external information and knowledge. The conflicts between dynamicity of behaviors and the steadiness of demographic attributes are largely ignored. To address this issue, we introduce a cross-OSN approach to discover the shared stable patterns from users’ social multimedia behaviors on multiple Online Social Networks (OSNs). The basic assumption for the proposed approach is that, the same user’s cross-OSN behaviors are the reflection of his/her demographic attributes in different scenarios. Based on this, a coupled projection matrix extraction method is proposed for solution, where the cross-OSN behaviors are collectively projected onto the same space for demographic attribute inference. Experimental evaluation is conducted on a self-collected Google+ and Twitter dataset consisting of four types of demographic attributes as gender, age, relationship and occupation. The experimental results demonstrate the effectiveness of cross-OSN based demographic attribute inference.

Towards SMP Challenge: Stacking of Diverse Models for Social Image Popularity Prediction

Popularity prediction on social media has attracted extensive attention nowadays due to its widespread applications, such as online marketing and economical trends. In this paper, we describe a solution of our team CASIA-NLPR-MMC for Social Media Prediction (SMP) challenge. This challenge is designed to predict the popularity of social media posts. We present a stacking framework by combining a diverse set of models to predict the popularity of images on Flickr using user-centered, image content and image context features. Several individual models are employed for scoring popularity of an image at earlier stage, and then a stacking model of Support Vector Regression (SVR) is utilized to train a meta model of different individual models trained beforehand. The Spearmans Rho of this Stacking model is 0.88 and the mean absolute error is about 0.75 on our test set. On the official final-released test set, the Spearmans Rho is 0.7927 and mean absolute error is about 1.1783. The results on provided dataset demonstrate the effectiveness of our proposed approach for image popularity prediction.

Who Are Your “Real” Friends: Analyzing and Distinguishing Between Offline and Online Friendships From Social Multimedia Data

The Internet has extended the physical boundary of peoples social circles to manage an inordinate number of online friends. It is recognized that only a fraction of these online friends are also known with each other in offline circumstances, i.e., the offline friends. An important type of offline friend, onsite offline friend, is defined and addressed in this paper. We explores the possibility of utilizing users’ online photo sharing-related behaviors and network topologies to analyze and distinguish between online and onsite offline friendships. Different from traditional social science studies which rely on survey-based data, we employ users’ tagged people on the shared Instagram photos as the ground-truth for onsite offline friends. This enables a large-scale and objective analysis and experimental evaluation, which compares between different factors and identifies the features that are key to onsite offline friend identification.

Hashtag-centric Immersive Search on Social Media

Social media information distributes in different Online Social Networks (OSNs). This paper addresses the problem integrating the cross-OSN information to facilitate an immersive social media search experience. We exploit hashtag, which is widely used to annotate and organize multi-modal items in different OSNs, as the bridge for information aggregation and organization. A three-stage solution framework is proposed for hashtag representation, clustering and demonstration. Given an event query, the related items from three OSNs, Twitter, Flickr and YouTube, are organized in cluster-hashtag-item hierarchy for display. The effectiveness of the proposed solution is validated by qualitative and quantitative experiments on hundreds of trending event queries.

CSAN: Contextual Self-Attention Network for User Sequential Recommendation

The sequential recommendation is an important task for online user-oriented services, such as purchasing products, watching videos, and social media consumption. Recent work usually used RNN-based methods to derive an overall embedding of the whole behavior sequence, which fails to discriminate the significance of individual user behaviors and thus decreases the recommendation performance. Besides, RNN-based encoding has fixed size and makes further recommendation application inefficient and inflexible. The online sequential behaviors of a user are generally heterogeneous, polysemous, and dynamically context-dependent. In this paper, we propose a unified Contextual Self-Attention Network (CSAN) to address the three properties. Heterogeneous user behaviors are considered in our model that are projected into a common latent semantic space. Then the output is fed into the feature-wise self-attention network to capture the polysemy of user behaviors. In addition, the forward and backward position encoding matrices are proposed to model dynamic contextual dependency. Through extensive experiments on two real-world datasets, we demonstrate the superior performance of the proposed model compared with other state-of-the-art algorithms.