Kashif Ahmad

The JORD System: Linking Sky and Social Multimedia Data to Natural Disasters

Being able to automatically link social media information and data to remote-sensed data holds large possibilities for society and research. In this paper, we present a system called JORD that is able to autonomously collect social media data about technological and environmental disasters, and link it automatically to remote-sensed data. In addition, we demonstrate that queries in local languages that are relevant to the exact position of natural disasters retrieve more accurate information about a disaster event. To show the capabilities of the system, we present some examples of disaster events detected by the system. To evaluate the quality of the provided information and usefulness of JORD from the potential users point of view we include a crowdsourced user study.

JORD: A System for Collecting Information and Monitoring Natural Disasters by Linking Social Media with Satellite Imagery

Gathering information, and continuously monitoring the affected areas after a natural disaster can be crucial to assess the damage, and speed up the recovery process. Satellite imagery is being considered as one of the most productive sources to monitor the after effects of a natural disaster; however, it also comes with a lot of challenges and limitations, due to slow update. It would be beneficiary to link remote sensed data with social media for the damage assessment, and obtaining detailed information about a disaster. The additional information, which are obtainable by social media, can enrich remote-sensed data, and overcome its limitations. To tackle this, we present a system called JORD that is able to autonomously collect social media data about natural disasters, and link it automatically to remote-sensed data. In addition, we demonstrate that queries in local languages that are relevant to the exact position of natural disasters retrieve more accurate information about a disaster event. We also provide content based analysis along with temporal and geo-location based filtering to provide more accurate information to the users. To show the capabilities of the system, we demonstrate that a large number of disaster events can be detected by the system. In addition, we use crowdsourcing to demonstrate the quality of the provided information about the disasters, and usefulness of JORD from potential users point of view

A hierarchical approach to event discovery from single images using MIL framework

In this paper we propose to face the problem of event detection from single images, by exploiting both background information often containing revealing contextual clues and details, which are salient for recognizing the event. Such details are visual objects critical to understand the underlying event depicted in the image and were recently defined in the literature as “event-saliency”. Adopting the Multiple-Instance Learning (MIL) paradigm we propose a hierarchical approach analyzing first the entire picture and then refining the decision on the basis of the event-salient objects. Validation of the proposed method is carried out on two benchmarking datasets and it demonstrates the effectiveness of the proposed hierarchical approach to event discovery from single images.

USED: a large-scale social event detection dataset

Event discovery from single pictures is a challenging problem that has raised significant interest in the last decade. During this time, a number of interesting solutions have been proposed to tackle event discovery in still images. However, a large scale benchmarking image dataset for the evaluation and comparison of event discovery algorithms from single images is still lagging behind. To this aim, in this paper we provide a large-scale properly annotated and balanced dataset of 490,000 images, covering every aspect of 14 different types of social events, selected among the most shared ones in the social network. Such a large scale collection of event-related images is intended to become a powerful support tool for the research community in multimedia analysis by providing a common benchmark for training, testing, validation and comparison of existing and novel algorithms. In this paper, we provide a detailed description of how the dataset is collected, organized and how it can be beneficial for the researchers in the multimedia analysis domain. Moreover, a deep learning based approach is introduced into event discovery from single images as one of the possible applications of this dataset with a belief that deep learning can prove to be a breakthrough also in this research area. By providing this dataset, we hope to gather research community in the multimedia and signal processing domains to advance this application.

A saliency-based approach to event recognition

Abstract Over the last few years, a number of interesting solutions covering different aspects of event recognition have been proposed for event-based multimedia analysis. Existing approaches mostly focus on an efficient representation of the image and advanced classification schemes. However, it would be desirable to focus on the event-specific information available in the image, namely the so-called event saliency. In this paper, we propose a novel approach based on multiple instance learning (MIL) to learn the visual features contained in event-salient regions, extracted through a crowd-sourcing study. In total, we collect the salient regions for 76 different events from 4 large-scale datasets. The experimental results demonstrate the efficacy of using only event-related regions by achieving a significant gain in performance over the state-of-the-art.

Effect of Salient Features in Object Recognition

the SIFT and SURF based recognition, the paper presents the impact of salient features in object recognition. We use the two well-known image descriptors in the bag of words framework on five online available standard datasets. Experiments show that by introducing saliency in the bag of words model, state-of-the-art performance can still be retained while reducing considerable amount of data processing and thus achieving faster execution times.

Event recognition in personal photo collections via multiple instance learning-based classification of multiple images

Over the last few years, a rapid growth has been witnessed in the number of digital photos produced per year. This rapid process poses challenges in the organization and management of multimedia collections, and one viable solution consists of arranging the media on the basis of the underlying events. However, album-level annotation and the presence of irrelevant pictures in photo collections make event-based organization of personal photo albums a more challenging task. To tackle these challenges, in contrast to conventional approaches relying on supervised learning, we propose a pipeline for event recognition in personal photo collections relying on a multiple instance-learning (MIL) strategy. MIL is a modified form of supervised learning and fits well for such applications with weakly labeled data. The experimental evaluation of the proposed approach is carried out on two large-scale datasets including a self-collected and a benchmark dataset. On both, our approach significantly outperforms the existing state-of-the-art.

Social media and satellites

Being able to automatically link social media and satellite imagery holds large opportunities for research, with a potentially considerable impact on society. The possibility of integrating different information sources opens in fact to new scenarios where the wide coverage of satellite imaging can be used as a collector of the fine-grained details provided by the social media. Remote-sensed data and social media data can well complement each other, integrating the wide perspective provided by the satellite view with the information collected locally, being it textual, audio, or visual. Among the possible applications, natural disasters are certainly one of the most interesting scenarios, where global and local perspectives are needed at the same time. In this paper, we present a system called JORD that is able to autonomously collect social media data (including the text analysis in local languages) about technological and environmental disasters, and link it automatically to remote-sensed data. Moreover, in order to ensure the quality of retrieved information, JORD is equipped with a hierarchical filtering mechanism relying on the temporal information and the content analysis of retrieved multimedia data. To show the capabilities of the system, we present a large number of disaster events detected by the system, and we evaluate both the quality of the provided information about the events and the usefulness of JORD from potential users viewpoint, using crowdsourcing.

Ensemble of Deep Models for Event Recognition

In this article, we address the problem of recognizing an event from a single related picture. Given the large number of event classes and the limited information contained in a single shot, the problem is known to be particularly hard. To achieve a reliable detection, we propose a combination of multiple classifiers, and we compare three alternative strategies to fuse the results of each classifier, namely: (i) induced order weighted averaging operators, (ii) genetic algorithms, and (iii) particle swarm optimization. Each method is aimed at determining the optimal weights to be assigned to the decision scores yielded by different deep models, according to the relevant optimization strategy. Experimental tests have been performed on three event recognition datasets, evaluating the performance of various deep models, both alone and selectively combined. Experimental results demonstrate that the proposed approach outperforms traditional multiple classifier solutions based on uniform weighting, and outperforms recent state-of-the-art approaches.

Computer vision based room interior design

This paper introduces a new application of computer vision. To the best of the author’s knowledge, it is the first attempt to incorporate computer vision techniques into room interior designing. The computer vision based interior designing is achieved in two steps: object identification and color assignment. The image segmentation approach is used for the identification of the objects in the room and different color schemes are used for color assignment to these objects. The proposed approach is applied to simple as well as complex images from online sources. The proposed approach not only accelerated the process of interior designing but also made it very efficient by giving multiple alternatives.