Qiusha Zhu

Web media semantic concept retrieval via tag removal and model fusion

Multimedia data on social websites contain rich semantics and are often accompanied with user-defined tags. To enhance Web media semantic concept retrieval, the fusion of tag-based and content-based models can be used, though it is very challenging. In this article, a novel semantic concept retrieval framework that incorporates tag removal and model fusion is proposed to tackle such a challenge. Tags with useful information can facilitate media search, but they are often imprecise, which makes it important to apply noisy tag removal (by deleting uncorrelated tags) to improve the performance of semantic concept retrieval. Therefore, a multiple correspondence analysis (MCA)-based tag removal algorithm is proposed, which utilizes MCAs ability to capture the relationships among nominal features and identify representative and discriminative tags holding strong correlations with the target semantic concepts. To further improve the retrieval performance, a novel model fusion method is also proposed to combine ranking scores from both tag-based and content-based models, where the adjustment of ranking scores, the reliability of models, and the correlations between the intervals divided on the ranking scores and the semantic concepts are all considered. Comparative results with extensive experiments on the NUS-WIDE-LITE as well as the NUS-WIDE-270K benchmark datasets with 81 semantic concepts show that the proposed framework outperforms baseline results and the other comparison methods with each component being evaluated separately.

Effective supervised discretization for classification based on correlation maximization

In many real-world applications, there are features (or attributes) that are continuous or numerical in the data. However, many classification models only take nominal features as the inputs. Therefore, it is necessary to apply discretization as a pre-processing step to transform numerical data into nominal data for such models. Well-discretized data should not only characterize the original data to produce a concise summarization, but also improve the classification performance. In this paper, a novel and effective supervised discretization algorithm based on correlation maximization (CM) is proposed by using multiple correspondence analysis (MCA) which is a technique to capture the correlations between multiple variables. For each numeric feature, the correlation information generated from MCA is used to build the discretization algorithm that maximizes the correlations between feature intervals/items and classes. Empirical comparisons with four other commonly used discretization algorithms are conducted using six well-known classifiers. Results on five UCI datasets and five TRECVID datasets demonstrate that our proposed discretization algorithm can automatically generate a better set of features (feature intervals) by maximizing their correlations with the classes and thus improve the classification performance.

VideoTopic: Content-Based Video Recommendation Using a Topic Model

Most video recommender systems limit the content to the metadata associated with the videos, which could lead to poor results since metadata is not always available or correct. Meanwhile, the visual information of videos is typically not fully explored, which is especially important for recommending new items with limited metadata information. In this paper, a novel content-based video recommendation framework, called Video Topic, that utilizes a topic model is proposed. It decomposes the recommendation process into video representation and recommendation generation. It aims to capture user interests in videos by using a topic model to represent the videos, and then generates recommendations by finding those videos that most fit to the topic distribution of the user interests. Experimental results on the Movie Lens dataset validate the effectiveness of Video Topic by evaluating each of its components and the whole framework.

Moving Object Detection under Object Occlusion Situations in Video Sequences

It is a great challenge to detect an object that is overlapped or occluded by other objects in images. For moving objects in a video sequence, their movements can bring extra spatio-temporal information of successive frames, which helps object detection, especially for occluded objects. This paper proposes a moving object detection approach for occluded objects in a video sequence with the assist of the SPCPE (Simultaneous Partition and Class Parameter Estimation) unsupervised video segmentation method. Based on the preliminary foreground estimation result from SPCPE and object detection information from the previous frame, an n-steps search (NSS) method is utilized to identify the location of the moving objects, followed by a size-adjustment method that adjusts the bounding boxes of the objects. Several experimental results show that our proposed approach achieves good detection performance under object occlusion situations in serial frames of a video sequence.

Sparse Linear Integration of Content and Context Modalities for Semantic Concept Retrieval

The semantic gap between low-level visual features and high-level semantics is a well-known challenge in content-based multimedia information retrieval. With the rapid popularization of social media, which allows users to assign tags to describe images and videos, attention is naturally drawn to take advantage of these metadata in order to bridge the semantic gap. This paper proposes a sparse linear integration (SLI) model that focuses on integrating visual content and its associated metadata, which are referred to as the content and the context modalities, respectively, for semantic concept retrieval. An optimization problem is formulated to approximate an instance using a sparse linear combination of other instances and minimize the difference between them. The prediction score of a concept for a test instance measures how well it can be reconstructed by the positive instances of that concept. Two benchmark image data sets and their associated tags are used to evaluate the SLI model. Experimental results show promising performance by comparing with the approaches based on a single modality and approaches based on popular fusion methods.

Negative Correlation Discovery for Big Multimedia Data Semantic Concept Mining and Retrieval

With massive amounts of data producing each day in almost every field, traditional data processing techniques have become more and more inadequate. However, the research of effectively managing and retrieving these big data is still under development. Multimedia high-level semantic concept mining and retrieval in big data is one of the most challenging research topics, which requires joint efforts from researchers in both big data mining and multimedia domains. In order to bridge the semantic gap between high-level concepts and low-level visual features, correlation discovery in semantic concept mining is worth exploring. Meanwhile, correlation discovery is a computationally intensive task in the sense that it requires a deep analysis of very large and growing repositories. This paper presents a novel system of discovering negative correlation for semantic concept mining and retrieval. It is designed to adapt to Hadoop MapReduce framework, which is further extended to utilize Spark, a more efficient and general cluster computing engine. The experimental results demonstrate the feasibility of utilizing big data technologies in negative correlation discovery.

Multimodal Sparse Linear Integration for Content-Based Item Recommendation

Most content-based recommender systems focus on analyzing the textual information of items. For items with images, the images can be treated as another information modality. In this paper, an effective method called MSLIM is proposed to integrate multimodal information for content-based item recommendation. It formalizes the probelm into a regularized optimization problem in the least-squares sense and the coordinate gradient descent is applied to solve the problem. The aggregation coefficients of the items are learned in an unsupervised manner during this process, based on which the k-nearest neighbor (k-NN) algorithm is used to generate the top-N recommendations of each item by finding its k nearest neighbors. A framework of using MSLIM for item recommendation is proposed accordingly. The experimental results on a self-collected handbag dataset show that MSLIM outperforms the selected comparison methods and show how the model parameters affect the final recommendation results.

Supporting Semantic Concept Retrieval with Negative Correlations in a Multimedia Big Data Mining System

With the extensive use of smart devices and blooming popularity of social media websites such as Flickr, YouTube, Twitter, and Facebook, we have witnessed an explosion of multimedia data. The amount of data nowadays is formidable without effective big data technologies. It is well-acknowledged that multimedia high-level semantic concept mining and retrieval has become an important research topic; while the semantic gap (i.e., the gap between the low-level features and high-level concepts) makes it even more challenging. To address these challenges, it requires the joint research efforts from both big data mining and multimedia areas. In particular, the correlations among the classes can provide important context cues to help bridge the semantic gap. However, correlation discovery is computationally expensive due to the huge amount of data. In this paper, a novel multimedia big data mining system based on the MapReduce framework is proposed to discover negative correlations for semantic concept mining and retrieval. Furthermore, the proposed multimedia big data mining system consists of a big data processing platform with Mesos for efficient resource management and with Cassandra for handling data across multiple data centers. Experimental results on the TRECVID benchmark datasets demonstrate the feasibility and the effectiveness of the proposed multimedia big data mining system with negative correlation discovery for semantic concept mining and retrieval.

VideoTopic: Modeling User Interests for Content-Based Video Recommendation

With the vast amount of video data uploaded to the Internet every day, how to analyze user interests and recommend videos that they are potentially interested in is a big challenge. Most video recommender systems limit the content to metadata associated with videos, which could lead to poor recommendation results since the metadata is not always available or correct. On the other side, visual content of videos contain information of different granularities, from a whole video, to portions of a video, and to an object in a video, which are not fully explored. This extra information is especially important for recommending new items when no user profile is available. In this paper, a novel recommendation framework, called VideoTopic, that targets at cold-start items is proposed. VideoTopic focuses on user interest modeling and decomposes the recommendation process into interest representation, interest discovery, and recommendation generation. It aims to model user interests by using a topic model to represent the interests in the videos and then discover user interests from user watch histories. A personalized list is generated to maximize the recommendation accuracy by finding the videos that most fit the users interests under the constraints of some criteria. The optimal solution and a practical system of VideoTopic are presented. Experiments on a public benchmark data set demonstrate the promising results of VideoTopic.

Correlation maximisation-based discretisation for supervised classification

This paper proposes a novel supervised discretisation algorithm based on Correlation Maximisation (CM) using Multiple Correspondence Analysis (MCA). MCA is an effective technique to capture the correlation between multiple variables. For each numeric feature, the proposed discretisation algorithm utilises MCA to measure the correlations between feature intervals/items and classes, and the set of cut-points yielding the maximum correlation is chosen as the discretisation scheme for that feature. Therefore, the discretised feature can not only produce a concise summarisation of the original numeric feature but also provide the maximum correlation information to predict class labels. Experiments are conducted by comparing to seven state-of-the-art supervised discretisation algorithms using six well-known classifiers on 19 UCI data sets. Experimental results demonstrate that the proposed discretisation algorithm can automatically generate a set of features (feature intervals) that produce the best classification results on average.