Roman Fedorov

A Framework for Outdoor Mobile Augmented Reality and Its Application to Mountain Peak Detection

Outdoor augmented reality applications project information of interest onto views of the world in real-time. Their core challenge is recognizing the meaningful objects present in the current view and retrieving and overlaying pertinent information onto such objects. In this paper we report on the development of a framework for mobile outdoor augmented reality application, applied to the overlay of peak information onto views of mountain landscapes. The resulting app operates by estimating the virtual panorama visible from the viewpoint of the user, using an online Digital Terrain Model (DEM), and by matching such panorama to the actual image framed by the camera. When a good match is found, meta-data from the DEM (e.g., peak name, altitude, distance) are projected in real time onto the view. The application, besides providing a nice experience to the user, can be employed to crowdsource the collection of annotated mountain images for environmental applications.

Estimating Snow Cover From Publicly Available Images

In this paper, we study the problem of estimating snow cover in mountainous regions, that is, the spatial extent of the earth surface covered by snow. We argue that publicly available visual content, in the form of user-generated photographs and image feeds from outdoor webcams, can both be leveraged as additional measurement sources, complementing existing ground, satellite, and airborne sensor data. To this end, we describe two content acquisition and processing pipelines that are tailored to such sources, addressing the specific challenges posed by each of them, e.g., identifying the mountain peaks, filtering out images taken in bad weather conditions, and handling varying illumination conditions. The final outcome is summarized in a snow cover index, which indicates for a specific mountain and day of the year the fraction of visible area covered by snow, possibly at different elevations. We created a manually labeled dataset to assess the accuracy of the image snow covered area estimation, achieving 90.0% precision at 91.1% recall. In addition, we show that seasonal trends related to air temperature are captured by the snow cover index.

Mountain Peak Identification in Visual Content Based on Coarse Digital Elevation Models

We present a method for the identification of mountain peaks in geo-tagged photos. The key tenet is to perform an edge-based matching between the visual content of each photo and a terrain view synthesized from a Digital Elevation Model (DEM). The latter is generated as if a virtual observer is located at the coordinates indicated by the geo-tag. The key property of the method is the ability to reach a highly accurate estimation of the position of mountain peaks with a coarse resolution DEM available in the corresponding geographical area, which is sampled at a spatial resolution between 30m and 90m. This is the case for publicly available DEMs that cover almost the totality of the Earth surface (such as SRTM CGIAR and ASTER GDEM). The method is fully unsupervised, thus it can be applied to the analysis of massive amounts of user generated content available, e.g., on Flickr and Panoramio. We evaluated our method on a dataset of manually annotated images of mountain landscapes, containing peaks of the Italian and Swiss Alps. Our results show that it is possible to accurately identify the peaks in 75.0% of the cases. This result increases to 81.6% when considering only photos with mountain slopes far from the observer.

Snow phenomena modeling through online public media

We propose a method for the environmental monitoring through the publicly available media User Generated Content (UGC). In particular we address the problem of the snow cover and level estimation by analyzing the social media data such as geotagged photographs and public webcams installed in mountain regions. The entire pipeline of the process is presented to the audience: from the data crawling and automatic relevance classification (does or does not the photograph contain a significant mountain profile) to the image content analysis and environmental models (identification of the snow covered area on the photograph). Each presented component is self-contained and can be inspected individually, the connections between the components however are strongly highlighted allowing the viewer to understand intuitively the entire pipeline structure.

Multimedia on the Mountaintop: Using Public Snow Images to Improve Water Systems Operation

This paper merges multimedia and environmental research to verify the utility of public web images for improving water management in periods of water scarcity, an increasingly critical event due to climate change. A multimedia processing pipeline fetches mountain images from multiple sources and extracts virtual snow indexes correlated to the amount of water accumulated in the snow pack. Such indexes are used to predict water availability and design the operating policy of Lake Como, Italy. The performance of this informed policy is contrasted, via simulation, with the current operation, which depends only on lake water level and day of the year, and with a policy that exploits official Snow Water Equivalent (SWE) estimated from ground stations data and satellite imagery. Virtual snow indexes allow improving the system performance by 11.6% w.r.t. the baseline operation, and yield further improvement when coupled with official SWE information, showing that the two data sources are complementary. The proposed approach exemplifies the opportunities and challenges of applying multimedia content analysis methods to complex environmental problems.

SnowWatch: A Multi-modal Citizen Science Application

The demo presents SnowWatch, a citizen science system that supports the acquisition and processing of mountain images for the purpose of extracting snow information, predicting the amount of water available in the dry season, and supporting a multi-objective lake regulation problem. We discuss how the proposed architecture has been rapidly prototyped using a general-purpose architecture to collect sensor and user-generated Web content from heterogeneous sources, process it for knowledge extraction, relying on the contribution of voluntary crowds, engaged and retained with gamification techniques.

A framework for regression testing of outdoor mobile applications

Outdoor mobile applications are becoming popular in fields such as gaming, tourism and environment monitoring. They rely on the input of multiple, possibly noisy, sensors, such as the camera, GPS, compass and gyroscope. The regression testing of such applications requires the reproduction of the real conditions in which the application works, which are hard to reproduce without automated support. We present a capture replay framework that automates regression testing of mobile outdoor applications, by recording data streams in real-time on the field from multiple sensors, replays them in lab and computes quality metrics to trace regression errors.

Compressing Web Geodata for Real-Time Environmental Applications

The advent of connected mobile devices has caused an unprecedented availability of geo-referenced user-generated content, which can be exploited for environment monitoring. In particular, Augmented Reality (AR) mobile applications can be designed to enable citizens collect observations, by overlaying relevant meta-data on their current view. This class of applications rely on multiple meta-data, which must be properly compressed for transmission and real-time usage. This paper presents a two-stage approach for the compression of Digital Elevation Model (DEM) data and geographic entities for a mountain environment monitoring mobile AR application. The proposed method is generic and could be applied to other types of geographical data.