Guanfeng Wang

Eddy: an error-bounded delay-bounded real-time map matching algorithm using HMM and online Viterbi decoder

Real-time map matching is a fundamental but challenging problem with various applications in Geographic Information Systems (GIS), Intelligent Transportation Systems (ITS) and beyond. It aims to align a sequence of measured latitude/longitude positions with the road network on a digital map in real-time. There exist a number of statistical matching approaches that unfortunately either process trajectory data offline or provide an online solution without an infimum analysis. Here we propose a novel statistics-based online map matching algorithm called Eddy with a solid error-and delay-bound analysis. More specifically, Eddy employs a Hidden Markov Model (HMM) to represent the spatio-temporal data as state chains, which elucidates the road networks topology, observation noises and their underlying relations. After modeling, we shape the decoding phase as a ski-rental problem, and an improved online-version Viterbi decoding algorithm is proposed to find the most likely sequence of hidden states (road routes) in real-time. We reduce the candidate routes search range during the decoding for efficiency reasons. Moreover, our deterministic decoder trades off latency for expected accuracy dynamically, without having to choose a fixed window size beforehand. We also provide the competitive analysis and the proof that our online algorithm is error-bounded (with a competitive ratio of 2) and latency-bounded. Our experimental results show that the proposed algorithm outperforms widely used existing approaches on both accuracy and latency.

Multi-video summary and skim generation of sensor-rich videos in geo-space

User-generated videos have become increasingly popular in recent years. Due to advances in camera technology it is now very easy and convenient to record videos with mobile devices, such as smartphones. Here we consider an application where users collect and share a large set of videos that are related to a geographic area, say a city. Such a repository can be a great source of information for prospective tourists when they plan to visit a city and would like to get a preview of its main areas. The challenge that we address is how to automatically create a preview video summary from a large set of source videos. The main features of our technique are that it is fully automatic and leverages meta-data sensor information which is acquired in conjunction with videos. The meta-data is collected from GPS and compass sensors and is used to describe the viewable scenes of the videos. Our method then proceeds in three steps through the analysis of the sensor data. First, we generate a single video summary. Shot boundaries are detected based on different motion types of camera movements and key frames are extracted related to motion patterns. Second, we build video skims for popular places (i.e., hotspots) aiming to provide maximal coverage of hotspot areas with minimal redundancy (per-spot multi-video summary). Finally, the individual hotspot skims are linked together to generate a pleasant video tour that visits all the popular places (multi-spot multi-video summary).

Keyframe presentation for browsing of user-generated videos on map interfaces

To present user-generated videos that relate to geographic areas for easy access and browsing it is often natural to use maps as interfaces. A common approach is to place thumbnail images of video keyframes in appropriate locations. Here we consider the challenge of determining which keyframes to select and where to place them on the map. Our proposed technique leverages sensor-collected meta-data which are automatically acquired as a continuous stream together with the video. Our approach is able to detect interesting regions and objects (hotspots) and their distances from the camera in a fully automated way. Meaningful keyframes are adaptively selected based on the popularity of the hotspots. Our experiments show very promising results and demonstrate excellent utility for the users.

Key Frame Selection Algorithms for Automatic Generation of Panoramic Images from Crowdsourced Geo-tagged Videos

Currently, an increasing number of user-generated videos (UGVs) are being collected – a trend that is driven by the ubiquitous availability of smartphones. Additionally, it has become easy to continuously acquire and fuse various sensor data (e.g., geospatial metadata) together with video to create sensor-rich mobile videos. As a result, large repositories of media contents can be automatically geo-tagged at the fine granularity of frames during video recording. Thus, UGVs have great potential to be utilized in various geographic information system (GIS) applications, for example, as source media to automatically generate panoramic images. However, large amounts of crowdsourced media data are currently underutilized because it is very challenging to manage, browse and explore UGVs.

Sensor-rich video exploration on a map interface

Result presentations from searches into video repositories is still a challenging problem. Current systems usually display a ranked list that shows the first frame of each video. Users then explore the videos one-by-one. In our recent work we have investigated the fusion of captured video with a continuous stream of sensor meta-data. These so-called sensor-rich videos can conveniently be captured with todays smartphones. Importantly, the recorded sensor-data streams enable processing and result resentation in novel and useful ways. In this demonstration we present a system that provides an integrated solution to present videos based on keyframe extraction and interactive, map-based browsing. As a key feature, the system automatically computes popular places based on the collective information from all the available videos. For each video it then extracts keyframes and renders them at their proper location on the map synchronously with the video playback. All the processing is performed in real-time, which allows for an interactive exploration of all the videos in a geographic area.

Active key frame selection for 3D model reconstruction from crowdsourced geo-tagged videos

Automatic reconstruction of 3D models is attracting increasing attention in the multimedia community. Scene recovery from video sequences requires a selection of representative video frames. Most prior work adopted content-based techniques to automate key frame extraction. However, these methods take no frame geo-information into consideration and are still compute-intensive. Here we propose a new approach for key frame selection based on the geographic properties of videos. Currently, an increasing number of user-generated videos (UGVs) are collected - a trend that is driven by the ubiquitous availability of smartphones. Additionally, it has become easy to continuously acquire and fuse various sensor data (e.g., geo-spatial metadata) with video to create geo-tagged mobile videos. Our novel technique utilizes these underlying geo-metadata to select the most representative frames. Specifically, a key frame subset with minimal spatial coverage gain difference is extracted by incorporating a manifold structure into reproducing a kernel Hilbert space to analyze the spatial relationship among the frames. Our experimental results illustrate that the execution time of the 3D reconstruction is shortened while the model quality is preserved.

Orientation data correction with georeferenced mobile videos

Similar to positioning data, camera orientation information has become a powerful contextual feature utilized by a number of GIS and social media applications. Such auxiliary information facilitates higher-level semantic analysis and management of video assets in such applications, e.g., video summarization and video indexing systems. However, it is problematic that raw sensor data collected from current mobile devices is often not accurate enough for subsequent geospatial analysis. To date, an effective orientation data correction system for mobile video content has been lacking. Here we present a content-based approach that improves the accuracy of noisy orientation sensor measurements generated by mobile devices in conjunction with video acquisition. Our preliminary experimental results demonstrate significant accuracy enhancements which benefit upstream sensor-aided GIS applications to access video content more precisely.

Automatic positioning data correction for sensor-annotated mobile videos

Video associated positioning data has become a useful contextual feature to facilitate analysis and management of media assets in GIS and social media applications. Moreover, with todays sensor-equipped mobile devices, the location of a camera can be continuously acquired in conjunction with the captured video stream without much difficulty. However, most sensor information collected from mobile devices is not highly accurate due to two main reasons: (a) the varying surrounding environmental conditions during data acquisition, and (b) the use of low-cost, consumer-grade sensors in current mobile devices. In this paper, we enhance the noisy positioning data generated by smartphones during video recording by analyzing typical error patterns for real collected data and introducing two robust algorithms, based on Kalman filtering and weighted linear least square regression, respectively. Our experimental results demonstrate significant benefits of our methods, which help upstream sensor-aided applications to access media content precisely.

Point of Interest Detection and Visual Distance Estimation for Sensor-Rich Video

Due to technological advances and the popularity of camera sensors, it is now straightforward for users to capture and share videos. A large number of geo-tagged photos and videos have been accumulating continuously on the web, posing a challenging problem for mining this type of media data. In one application scenario, users might desire to know what the Points of Interest (POI) are which contain important objects or places in a video. Existing solutions attempt to examine the content of the videos and recognize objects and events. This is typically time-consuming and computationally expensive and the results can be uneven. Therefore these methods face challenges when applied to large video repositories. We propose a novel technique that leverages sensor-generated meta-data (camera locations and viewing directions) which are automatically acquired as continuous streams together with the video frames. Existing smartphones can easily accommodate such integrated recording tasks. By considering a collective set of videos and leveraging the acquired auxiliary meta-data, our approach is able to detect interesting regions and objects (POIs) and their distances from the camera positions in a fully automated way. Because of its computational efficiency, the proposed method scales well and our experiments show very promising results.

Sensor-assisted camera motion analysis and motion estimation improvement for H.264/AVC video encoding

Camera motion information is one aspect that helps to infer higher-level semantic descriptions in many video applications, e.g., in video retrieval. However, an efficient methodology for annotating camera motion information is still an elusive goal. Here we propose and present a novel and efficient approach for the task of partitioning a video document into sub-shots and characterizing their camera motion. By leveraging location (GPS) and digital compass data, which are available from most current smartphone handsets, we exploit the geographical sensor information to detect transitions between two sub-shots based on the variations of both the camera location and the shooting direction. The advantage of our method lies in its considerable accuracy. Additionally, the computational efficiency of our scheme enables it to be deployed on mobile devices and to process videos while recording. We utilize this capability to show how the HEX motion estimation algorithm in the H.264/AVC encoder can be simplified with the aid of our camera motion information. Our experimental results show that we can reduce the computation of the HEX algorithm by up to 50% while achieving comparable video quality.