Mengfan Tang

Using Photos as Micro-Reports of Events

Photos serve dual role. Photos are important for capturing, saving, sharing, and reminiscing memories of events and people. Modern photos, however, are becoming more spontaneous, objective, compelling, and universal reports of a moment in an event also. In this paper our focus is on millions of photos being captured as informative reports and using them for emerging applications including situation recognition, trend analysis, and cultural dynamics. EventShop is an open source platform for situation recognition. Utilizing this platform and using a stream of photo reports from various sources as one of the data streams in this platform, we build a visual analytics system to understand the information that could be gleaned from such photo report streams. Our early experiments are based on the Yahoo Flickr Creative Commons 100 Million photos set released recently. We are also using other sources to import and understand the efficacy of these reports for various important applications.

Geospatial interpolation analytics for data streams in eventshop

EventShop is an open-source software which provides a generic infrastructure for the analysis of heterogeneous spatio-temporal data streams. Efficient interpolation of data from spatially sparse sources is critical but currently missing in EventShop. To address this challenge, we implement a Spatial Gaussian Process based statistical operator into the EventShop framework. Spectral analysis is employed to generate features at higher spatial resolution and to improve interpolation accuracy at unsampled locations. Further, we test this operator by interpolating air pollution levels in California. The evaluations of multiple metrics demonstrate that our operators outperform earlier EventShop operators, chemical transportation models, and state-of-the-art methods.

A graph based multimodal geospatial interpolation framework

Recent multimedia research has increasingly focused on large scale multimodal data from disparate geospatial sensors. In addition to the volume of the data, the diversity and granularity of the data poses a major challenge in extracting meaningful and actionable information. To address this, we present a novel spatial interpolation framework, capable of incorporating multimodal data sources and modeling the spatial processes comprehensively at multiple resolutions. The framework transforms the spatial interpolation problem into a graph structure learning problem, based on the latent structure of the data. This enables more efficient and accurate predictions at unobserved locations. We demonstrate the effectiveness of our approach by testing it on air pollution interpolation.

Integration of Diverse Data Sources for Spatial PM2.5 Data Interpolation

Heterogeneous data fusion from disparate geospatial sensors has drawn increasing attention in multimedia. Unfortunately, environmental sensors are usually sparsely and preferentially located, which restricts situation recognition of geographical regions and results in uncertainty in derived inferences. Spatial interpolation is an effective way to solve the problem of data sparsity, which demands the availability of related data sources. However, these data sources are usually in different resolutions, distributions, scales, and densities, which poses a major challenge in data integration. To address this problem, we present a novel spatial interpolation framework to incorporate diverse data sources and model the spatial processes explicitly at multiple resolutions. Spectral analysis is deployed to generate features at multiple spatial resolutions and to improve the interpolation accuracy at unobserved locations. A statistical operator based on the spatial Gaussian process is implemented and integrated into a geospatial situation recognition system, which can analyze heterogeneous spatio-temporal data streams derived from sensors. To verify the effectiveness and efficiency of the proposed framework, this framework is applied to the PM2.5 air pollution application. Experiments conducted in California, USA, demonstrate that the proposed method outperforms state-of-the-art approaches.

Capped Lp-Norm Graph Embedding for Photo Clustering

Photos are a predominant source of information on a global scale. Cluster analysis of photos can be applied to situation recognition and understanding cultural dynamics. Graph-based learning provides a current approach for modeling data in clustering problems. However, the performance of this framework depends heavily on initial graph construction by input data. Data outliers degrade graph quality, leading to poor clustering results. We designed a new capped lp-norm graph-based model to reduce the impact of outliers. This is accomplished by allowing the data graph to self adjust as part of the graph embedding. Furthermore, we derive an iterative algorithm to solve the objective function optimization problem. Experiments on four real-world benchmark data sets and Yahoo Flickr Creative Commons data set show the effectiveness of this new graph-based capped lp-norm clustering method.

Habits vs Environment: What Really Causes Asthma?

Despite considerable number of studies on risk factors for asthma onset, very little is known about their relative importance. To have a full picture of these factors, both categories, personal and environmental data, have to be taken into account simultaneously, which is missing in previous studies. We propose a framework to rank the risk factors from heterogeneous data sources of the two categories. Established on top of EventShop and Personal EventShop, this framework extracts about 400 features, and analyzes them by employing a gradient boosting tree. The features come from sources including personal profile and life-event data, and environmental data on air pollution, weather and PM2.5 emission sources. The top ranked risk factors derived from our framework agree well with the general medical consensus. Thus, our framework is a reliable approach, and the discovered rankings of relative importance of risk factors can provide insights for the prevention of asthma.

Research Challenges in Developing Multimedia Systems for Managing Emergency Situations

With an increasing amount of diverse heterogeneous data and information, the methodology of multimedia analysis has become increasingly relevant in solving challenging societal problems such as managing emergency situations during disasters. Using cybernetic principles combined with multimedia technology, researchers can develop effective frameworks for using diverse multimedia (including traditional multimedia as well as diverse multimodal) data for situation recognition, and determining and communicating appropriate actions to people stranded during disasters. We present known issues in disaster management and then focus on emergency situations. We show that an emergency management problem is fundamentally a multimedia information assimilation problem for situation recognition and for connecting peoples needs to available resources effectively, efficiently, and promptly. Major research challenges for managing emergency situations are identified and discussed. We also present a intelligently detecting evolving environmental situations, and discuss the role of multimedia micro-reports as spontaneous participatory sensing data streams in emergency responses. Given enormous progress in concept recognition using machine learning in the last few years, situation recognition may be the next major challenge for learning approaches in multimedia contextual big data. The data needed for developing such approaches is now easily available on the Web and many challenging research problems in this area are ripe for exploration in order to positively impact our society during its most difficult times.

A graph regularized dimension reduction method for out-of-sample data

Among various dimension reduction techniques, Principal Component Analysis (PCA) is specialized in treating vector data, whereas Laplacian embedding is often employed for embedding graph data. Moreover, graph regularized PCA, a combination of both techniques, has also been developed to assist the learning of a low dimensional representation of vector data by incorporating graph data. However, these approaches are confronted by the out-of-sample problem: each time when new data is added, it has to be combined with the old data before being fed into the algorithm to re-compute the eigenvectors, leading to enormous computational cost. In order to address this problem, we extend the graph regularized PCA to the graphźregularized linear regression PCA (grlrPCA). grlrPCA eliminates the redundant calculation on the old data by first learning a linear function and then directly applying it to the new data for its dimension reduction. Furthermore, we derive an efficient iterative algorithm to solve grlrPCA optimization problem and show the close relatedness of grlrPCA and unsupervised Linear Discriminant Analysis at infinite regularization parameter limit. The evaluations of multiple metrics on seven realistic datasets demonstrate that grlrPCA outperforms established unsupervised dimension reduction algorithms.

Semi-supervised learning on large-scale geotagged photos for situation recognition

Abstract Photos are becoming spontaneous, objective, and universal sources of information. This paper explores evolving situation recognition using photo streams coming from disparate sources combined with the advances of deep learning. Using visual concepts in photos together with space and time information, we formulate the situation detection into a semi-supervised learning framework and propose new graph-based models to solve the problem. To extend the method for unknown situations, we introduce a soft label method that enables the traditional semi-supervised learning framework to accurately predict predefined labels as well as effectively form new clusters. To overcome the noisy data which degrades graph quality, leading to poor recognition results, we take advantage of two kinds of noise-robust norms which can eliminate the adverse effects of outliers in visual concepts and improve the accuracy of situation recognition. Finally, we demonstrate the idea and the effectiveness of the proposed models on Yahoo Flickr Creative Commons 100 Million.

Geospatial Multimedia Data for Situation Recognition

Many emerging problems increasingly rely on integrating complex heterogeneous sensor streams, ranging from photos, texts, environmental data streams and other participating human sensors. The diversity of data types from disparate sensors poses a major challenge in data aggregation and assimilation. EventShop was originally designed for situation recognition using diverse data sources. We propose and develop new interpolation and prediction models on top of EventShop, allowing for effective ingesting and combining appropriate data streams to improve data quality and predict specific situations. We also incorporate data from participatory sensing into the system. The synergy of data gives powerful insight into better understanding of evolving situations, in which participatory sensing is integrated with the surrounding environment. Furthermore, the enhanced system is used for two real-world problems: asthma risk management and smart city.