Xueliang Liu

Finding media illustrating events

We present a method combining semantic inferencing and visual analysis for finding automatically media (photos and videos) illustrating events. We report on experiments validating our heuristic for mining media sharing platforms and large event directories in order to mutually enrich the descriptions of the content they host. Our overall goal is to design a web-based environment that allows users to explore and select events, to inspect associated media, and to discover meaningful, surprising or entertaining connections between events, media and people participating in events. We present a large dataset composed of semantic descriptions of events, photos and videos interlinked with the larger Linked Open Data cloud and we show the benefits of using semantic web technologies for integrating multimedia metadata.

Heterogeneous features and model selection for event-based media classification

With the rapid development of social media sites, a lot of user generated content is being shared in the Web, leading to new challenges for traditional media retrieval techniques. An event describes the happening at a specific time and place in real-world, and it is one of the most important cues for people to recall past memories. The reminder value of an event makes it extremely helpful in organizing human life. Thus, organizing media by events has recently drawn much attention within the multimedia research community. In this paper, we focus on two fundamental problems related to event based social media analysis: the study of feature importance for modeling the relation between events and media, and how to deal with missing and erroneous metadata often present in social media data. These issues are studied within an event-based media classification framework. Different learning approaches are employed to train the event models on different features. We find, through experiments on a large set of events, that the best discriminant features are tags, spatial and temporal feature. We address the missing value problem by extending the feature with an extra attribute to indicate if the values are missing. Promising results are achieved demonstrating the effectiveness of the proposed method.

Event-Based Media Enrichment Using an Adaptive Probabilistic Hypergraph Model

Nowadays, with the continual development of digital capture technologies and social media services, a vast number of media documents are captured and shared online to help attendees record their experience during events. In this paper, we present a method combining semantic inference and multimodal analysis for automatically finding media content to illustrate events using an adaptive probabilistic hypergraph model. In this model, media items are taken as vertices in the weighted hypergraph and the task of enriching media to illustrate events is formulated as a ranking problem. In our method, each hyperedge is constructed using the K-nearest neighbors of a given media document. We also employ a probabilistic representation, which assigns each vertex to a hyperedge in a probabilistic way, to further exploit the correlation among media data. Furthermore, we optimize the hypergraph weights in a regularization framework, which is solved as a second-order cone problem. The approach is initiated by seed media and then used to rank the media documents using a transductive inference process. The results obtained from validating the approach on an event dataset collected from EventMedia demonstrate the effectiveness of the proposed approach.

Event Representation and Visualization from Social Media

The user generated content, available in massive amounts in social media, is receiving increased attention due to its many potential applications. One of such applications is the representation of events with multimedia data. This paper addresses the problem of retrieving and summarizing events on a given topic, and propose a novel and original framework for leveraging social media data to extract and illustrate social events automatically on any given query. The problem is tackled in three steps. First, the input query is parsed semantically to identify the topic, location, and time information related to the event of interest. Then, we use the parsed information to mine the latest and hottest related events from social news web services. In the end, for each event, we retrieve both relevant tweets on Twitter and compelling images from Google image search. The resulting documents are shown within a vivid interface featuring both event description, tag cloud and photo collage.

Event-based cross media question answering

User generated content, available in massive amounts on the Internet, is receiving increased attention due to its many potential applications. One of such applications is the representation of events using multimedia data. In this paper, an event-based cross media question answering system, which retrieves and summarizes events on a given topic is proposed. In other words, we present a framework for leveraging social media data to extract and illustrate social events automatically on any given query. The system is built in three steps. First, the input query is parsed semantically to identify the topic, location, and time information related to the News of interest. Then, we use the parsed information to mine the latest and hottest related News from social news web services. Third, to identify a unique event, we model the News content by latent Dirichlet Allocation and cluster the News using the DBSCAN algorithm. In the end, for each event, we retrieve both textual and visual content of News that refer the same event. The resulting documents are shown within a vivid interface featuring both event description, tag cloud and photo collage.

Event analysis in social multimedia: a survey

Recent years have witnessed the rapid growth of social multimedia data available over the Internet. The age of huge amount of media collection provides users facilities to share and access data, while it also demands the revolution of data management techniques, since the exponential growth of social multimedia requires more scalable, effective and robust technologies to manage and index them. The event is one of the most important cues to recall people’s past memory. The reminder value of an event makes it extremely helpful in organizing data. The study of event based analysis on social multimedia data has drawn intensive attention in research community. In this article, we provide a comprehensive survey on event based analysis over social multimedia data, including event enrichment, detection, and categorization. We introduce each paradigm and summarize related research efforts. In addition, we also suggest the emerging trends in this research area.

EventEnricher: a novel way to collect media illustrating events

Exploiting event context to organize social media draws lots of interest from the multimedia community. In this paper, we present our system, called EventEnricher, to infer the semantics behind events and explore social media to illustrate events. We extend the set of illustrating images for a particular event by querying social media with diverse multi-modal features and subsequently pruning the results using content based visual analysis. We integrate the solution into an intelligent interface that enables the user to browse the media collection illustrating events in an easy, effective and informative way.

Social event discovery by topic inference

With the keen interest of people for social media sharing websites the multimedia research community faces new challenges and compelling opportunities. In this paper, we address the problem of discovering specific events from social media data automatically. Our proposed approach assumes that events are conjoint distribution over the latent topics in a given place. Based on this assumption, topics are learned from large amounts of automatically collected social data using a LDA model. Then, event distribution estimation over a topic is solved using least mean square optimization. We evaluate our methods on locations scattered around the world and show via our experimental results that the proposed framework offers promising performance for detecting events based on social media.

Gathering training sample automatically for social event visual modeling

In recent years, the emergence of social media on the Internet has derived many of interesting research and applications. In this paper, a novel framework is proposed to model the visual appearance of social events using automatically collected training samples on the basis of photo context analysis. While collecting positive samples can be achieved easily thanks to explicitly identifying tags, finding representative negative samples from the vast amount of irrelevant multimedia documents is a more challenging task. Here, we argue and demonstrate that the most common negative sample, originating from the same location as the event to be modeled, are best suited for the task. A novel ranking approach is devised to select a set of negative samples. The visual event models are learned from automatically collected samples using SVM. The results reported here show that the event models are effective to filter out irrelevant photos and perform with a high accuracy on various social events categories.

Deep Graph Laplacian Hashing for Image Retrieval

Due to the storage and retrieval efficiency, hashing has been widely deployed to approximate nearest neighbor search for fast image retrieval on large-scale datasets. It aims to map images to compact binary codes that approximately preserve the data relations in the Hamming space. However, most of existing approaches learn hashing functions using the hand-craft features, which cannot optimally capture the underlying semantic information of images. Inspired by the fast progress of deep learning techniques, in this paper we design a novel Deep Graph Laplacian Hashing (DGLH) method to simultaneously learn robust image features and hash functions in an unsupervised manner. Specifically, we devise a deep network architecture with graph Laplacian regularization to preserve the neighborhood structure in the learned Hamming space. At the top layer of the deep network, we minimize the quantization errors, and enforce the bits to be balanced and uncorrelated, which makes the learned hash codes more efficient. We further utilize back-propagation to optimize the parameters of the networks. It should be noted that our approach does not require labeled training data and is more practical to real-world applications in comparison to supervised hashing methods. Experimental results on three benchmark datasets demonstrate that DGLH can outperform the state-of-the-art unsupervised hashing methods in image retrieval tasks.