Mingyan Gao

Situation recognition: an evolving problem for heterogeneous dynamic big multimedia data

With the growth in social media, internet of things, and planetary-scale sensing there is an unprecedented need to assimilate spatio-temporally distributed multimedia streams into actionable information. Consequently the concepts like objects, scenes, and events, need to be extended to recognize situations (e.g. epidemics, traffic jams, seasons, flash mobs). This paper motivates and computationally grounds the problem of situation recognition. It describes a systematic approach for combining multimodal real-time big data into actionable situations. Specifically it presents a generic approach for modeling and recognizing situations. A set of generic building blocks and guidelines help the domain experts model their situations of interest. The created models can be tested, refined, and deployed into practice using a developed system (EventShop). Results of applying this approach to create multiple situation-aware applications by combining heterogeneous streams (e.g. Twitter, Google Insights, Satellite imagery, Census) are presented.

Eventshop: from heterogeneous web streams to personalized situation detection and control

The Web now has enormous volume of heterogeneous data being continuously reported by different sensors and humans from different locations. These data flows can be considered as spatio-temporal-thematic streams. Combined effectively, these streams can be used for detecting situations and saving lives and resources. We describe a system to combine streams from heterogeneous data sources, process them to detect situations, and use the detected situations to aid millions of users. This system uses a unified data model to integrate different web streams, and provides a set of generic operators to detect spatio-temporal characteristics of individual or combined data streams to detect complex situations. The detected situations can be combined with user parameters to provide personalized information and action alerts.

EventShop: recognizing situations in web data streams

Web Observatories must address fundamental societal challenges using enormous volumes of data being created due to the significant progress in technology. The proliferation of heterogeneous data streams generated by social media, sensor networks, internet of things, and digitalization of transactions in all aspect of humans? life presents an opportunity to establish a new era of networks called Social Life Networks (SLN). The main goal of SLN is to connect People to Resources effectively, efficiently, and promptly in given Situations. Towards this goal, we present a computing framework, called EventShop, to recognize evolving situations from massive web streams in real-time. These web streams can be fundamentally considered as spatio-temporal-thematic streams and can be combined using a set of generic spatio-temporal analysis operators to recognize evolving situations. Based on the detected situations, the relevant information and alerts can be provided to both individuals and organizations. Several examples from the real world problems have been developed to test the efficacy of EventShop framework.

WonderWhat: real-time event determination from photos

How often did you feel disappointed in a foreign country, when you had been craving for participating in authentic native events but miserably ended up with being lost in the crowd, due to ignorance of the local culture? Have you ever imagined that with merely a simple click, a tool can identify the events that are right in front of you? As a step in this direction, in this paper, we propose a system that provides users with information of the public events that they are attending by analyzing in real time their photos taken at the event, leveraging both spatio-temporal context and photo content. To fulfill the task, we designed the system to collect event information, maintain dedicated event database, build photo content model for event types, and rank the final results. Extensive experiments were conducted to prove the effectiveness of each component.

Spatio-temporal event stream processing in multimedia communication systems

Emerging multimedia communication environments, such as Environment-to-Environment (E2E) systems, require detecting complex events in environments using multimodal sensory data. Based on these spatio-temporal events, systems select and send data from appropriate sensors. Most existing stream processing systems consider temporal streams of alpha-numeric data and provide efficient approach to deal with queries in these environments. In cases where events are detected in different sensory data types, including audio and video collected at different locations, new approaches need to be developed to represent, combine, and process events to answer queries. In this paper, we present our approach in managing event stream processing to address the needs of a real time E2E system being developed in our laboratory. We introduce the modeling of our problem, and describe in detail the filtering and matching algorithms for querying spatio-temporal event stream. Experimental results demonstrate the efficacy and efficiency of our approach.

MEDIALIFE: from images to a life chronicle

Amarnath Gupta Univ. of California Irvine 9500 Gilman Drive La Jolla, CA 92093, USA gupta@sdsc.edu Setareh Rafatirad Dept. of Computer Science Univ. of California Irvine Irvine, USA, CA 92697 srafatirad@gmail.com Mingyan Gao Dept. of Computer Science Univ. of California Irvine Irvine, USA, CA 92697 gaomingyan@gmail.com Ramesh Jain Dept. of Computer Science Univ. of California Irvine Irvine, USA, CA 92697 jain@ics.uci.edu

Extractable mobile photo tags

Mobile phones are resulting in a major shift in how people shoot photos. Just a little more than a decade ago consumer behavior was plan-shoot-process-share-organize-reflect; but rapid proliferation of mobile phone cameras have resulted in shoot-share-forget behavior. This trend will be replaced soon because photos are more important than that - people treasure their memories in visual form. Fortunately, a plethora of sensors combined with access to powerful Web may allow effortless organize and reflect environment without much, if any, cognitive load on the consumer. We propose new approaches for determining attributes that we call Extractable Mobile Photo Tags (EMPT) for processing and organizing photos and videos on mobile phones. We present approaches to populate EMPT and use it for applications.