Seon Ho Kim

Viewable scene modeling for geospatial video search

Video sensors are becoming ubiquitous and the volume of captured video material is very large. Therefore, tools for searching video databases are indispensable. Current techniques that extract features purely based on the visual signals of a video are struggling to achieve good results. By considering video related meta-information, more relevant and precisely delimited search results can be obtained. In this study we propose a novel approach for querying videos based on the notion that the geographical location of the captured scene in addition to the location of a camera can provide valuable information and may be used as a search criterion in many applications. This study provides an estimation model of the viewable area of a scene for indexing and searching and reports on a prototype implementation. Among our objectives is to stimulate a discussion of these topics in the research community as information fusion of different georeferenced data sources is becoming increasingly important. Initial results illustrate the feasibility of the proposed approach.

On configuring a single disk continuous media server

The past decade has witnessed a proliferation of repositories that store and retrieve continuous media data types, e.g., audio and video objects. These repositories are expected to play a major role in several emerging applications, e.g., library information systems, educational applications, entertainment industry, etc. To support the display of a video object, the system partitions each object into fixed size blocks. All blocks of an object reside permanently on the disk drive. When displaying an object, the system stages the blocks of the object into memory one at a time for immediate display. In the presence of multiple displays referencing different objects, the bandwidth of the disk drive is multiplexed among requests, introducing disk seeks. Disk seeks reduce the useful utilization of the disk bandwidth and result in a lower number of simultaneous displays (throughput).This paper characterizes the impact of disk seeks on the throughput of the system. It describes REBECA as a mechanism that maximizes the throughput of the system by minimizing the time attributed to each incurred seek. A limitation of REBECA is that it increases the latency observed by each request. We quantify this throughput vs latency tradeoff of REBECA and, develop an efficient technique that computes its configuration parameters to realize the performance requirements (desired latency and throughput) of an application.

Automatic tag generation and ranking for sensor-rich outdoor videos

Video tag annotations have become a useful and powerful feature to facilitate video search in many social media and web applications. The majority of tags assigned to videos are supplied by users - a task which is time consuming and may result in annotations that are subjective and lack precision. A number of studies have utilized content-based extraction techniques to automate tag generation. However, these methods are compute-intensive and challenging to apply across domains. Here, we describe a complementary approach for generating tags based on the geographic properties of videos. With todays sensor-equipped smartphones, the location and orientation of a camera can be continuously acquired in conjunction with the captured video stream. Our novel technique utilizes these sensor meta-data to automatically tag outdoor videos in a two step process. First, we model the viewable scenes of the video as geometric shapes by means of its accompanied sensor data and determine the geographic objects that are visible in the video by querying geo-information databases through the viewable scene descriptions. Subsequently we extract textual information about the visible objects to serve as tags. Second, we define six criteria to score the tag relevance and rank the obtained tags based on these scores. Then we associate the tags with the video and the accurately delimited segments of the video. To evaluate the proposed technique we implemented a prototype tag generator and conducted a user study. The results demonstrate significant benefits of our method in terms of automation and tag utility.

Relevance ranking in georeferenced video search

The rapid adoption and deployment of ubiquitous video cameras has led to the collection of voluminous amounts of media data. However, indexing and searching of large video databases remain a very challenging task. Recently, some recorded video data are automatically annotated with meta-data collected from various sensors such as Global Positioning System (GPS) and compass devices. In our earlier work, we proposed the notion of a viewable scene model derived from the fusion of location and direction sensor information with a video stream. Such georeferenced media streams are useful in many applications and, very importantly, they can effectively be searched via their meta-data on a large scale. Consequently, search by geo-properties complements traditional content-based retrieval methods. The result of a georeferenced video query will in general consist of a number of video segments that satisfy the query conditions, but with more or less relevance. For example, a building of interest may appear in a video segment, but may only be visible in a corner. Therefore, an essential and integral part of a video query is the ranking of the result set according to the relevance of each clip. An effective result ranking is even more important for video than it is for text search, since the browsing of results can only be achieved by viewing each clip, which is very time consuming. In this study, we investigate and present three ranking algorithms that use spatial and temporal properties of georeferenced videos to effectively rank search results. To allow our techniques to scale to large video databases, we further introduce a histogram-based approach that allows fast online computations. An experimental evaluation demonstrates the utility of the proposed methods.

Minimizing start-up latency in scalable continuous media servers

In a scalable server that supports the retrieval and display of continuous media, both the number of simultaneous displays and the expected startup latency of a display increases as a function of additional disk bandwidth. Based on a striping technique and around-robin placement of data, this paper describes object replication and request migration as two alternative techniques to minimize startup latency. In addition to developing analytical models for these two techniques, we report on their implementation using a scalable server. The results obtained from both the analytical models and the experimental system demonstrate the effectiveness of the proposed techniques.

Generating synthetic meta-data for georeferenced video management

Recently various sensors, such as GPS and compass devices, can be cost-effectively manufactured and this allows their deployment in conjunction with mobile video cameras. Hence, recorded clips can automatically be annotated with geospatial information and the resulting georeferenced videos may be used in various Geographic Information System (GIS) applications. However, the research community is lacking large-scale and realistic test datasets of such sensor-fused information to evaluate their techniques since collecting real-world test data requires considerable time and effort. To fill this void, we propose an approach for generating synthetic video meta-data with realistic geospatial properties for mobile video management research. We highlight the essential aspects of the georeferenced video meta-data and present an approach to simulate the behavioral patterns of mobile cameras in the synthetic data. The data generation process can be customized through user parameters for a variety of GIS applications that use mobile videos. We demonstrate the feasibility and applicability of the proposed approach by providing comparisons with real-world data.

Design and implementation of geo-tagged video search framework

User generated video content is experiencing significant growth which is expected to continue and further accelerate. As an example, users are currently uploading 20h of video per minute to YouTube. Making such video archives effectively searchable is one of the most critical challenges of multimedia management. Current search techniques that utilize signal-level content extraction from video struggle to scale. Here we present a framework based on the complementary idea of acquiring sensor streams automatically in conjunction with video content. Of special interest are geographic properties of mobile videos. The meta-data from sensors can be used to model the coverage area of scenes as spatial objects such that videos can effectively, and on a large scale, be organized, indexed and searched based on their field-of-views. We present an overall framework that is augmented with our design and implementation ideas to illustrate the feasibility of this concept of managing geo-tagged video.

GRVS: a georeferenced video search engine

An increasing number of recorded videos are being tagged with geographic properties of the camera scenes. This meta-data is of significant use for storing, indexing and searching large collections of videos. By considering video related meta-information, more relevant and precisely delimited search results can be returned. Our system implementation demonstrates a prototype of a georeferenced video search engine (GRVS) that utilizes an estimation model of a cameras viewable scene for efficient video search. For video acquisition, our system provides an automated annotation software that captures videos and their respective field of views (FOV). The acquisition software allows community-driven data contributions to the search engine.

Janus - Multi Source Event Detection and Collection System for Effective Surveillance of Criminal Activity

Recent technological advances provide the opportunity to use large amounts of multimedia data from a multitude of sensors with different modalities (e.g., video, text) for the detection and characterization of criminal activity. Their integration can compensate for sensor and modality deficiencies by using data from other available sensors and modalities. However, building such an integrated system at the scale of neighborhood and cities is challenging due to the large amount of data to be considered and the need to ensure a short response time to potential criminal activity. In this paper, we present a system that enables multi-modal data collection at scale and automates the detection of events of interest for the surveillance and reconnaissance of criminal activity. The proposed system showcases novel analytical tools that fuse multimedia data streams to automatically detect and identify specific criminal events and activities. More specifically, the system detects and analyzes series of incidents (an incident is an occurrence or artifact relevant to a criminal activity extracted from a single media stream) in the spatiotemporal domain to extract events (actual instances of criminal events) while cross-referencing multimodal media streams and incidents in time and space to provide a comprehensive view to a human operator while avoiding information overload. We present several case studies that demonstrate how the proposed system can provide law enforcement personnel with forensic and real time tools to identify and track potential criminal activity. Keywords— multi-source, multi-modal event detection, law enforcement, criminal activity, surveillance, security, safety

Text region extraction and text segmentation on camera-captured document style images

In this paper, we propose a text extraction method from camera-captured document style images and propose a text segmentation method based on a color clustering method. The proposed extraction method detects text regions from the images using two low-level image features and verifies the regions through a high-level text stroke feature. The two level features are combined hierarchically. The low-level features are intensity variation and color variance. And, we use text strokes as a high-level feature using multi-resolution wavelet transforms on local image areas. The stroke feature vector is an input to a SVM (support vector machine) for verification, when needed. The proposed text segmentation method uses color clustering to the extracted text regions. We improved K-means clustering method and it selects K and initial seed values automatically. We tested the proposed methods with various document style images captured by three different cameras. We confirmed that the extraction rates are good enough to be used in real-life applications.