Massimiliano Ruocco

A scalable algorithm for extraction and clustering of event-related pictures

The event detection problem, which is closely related to clustering, has gained a lot of attentions within event detection for textual documents. However, although image clustering is a problem that has been treated extensively in both Content-Based Image Retrieval (CBIR) and Text-Based Image Retrieval (TBIR) systems, event detection within image management is a relatively new area. Having this in mind, we propose a novel approach for event extraction and clustering of images, taking into account textual annotations, time and geographical positions. Our goal is to develop a clustering method based on the fact that an image may belong to an event cluster. Here, we stress the necessity of having an event clustering and cluster extraction algorithm that are both scalable and allow online applications. To achieve this, we extend a well-known clustering algorithm called Suffix Tree Clustering (STC), originally developed to cluster text documents using document snippets. The idea is that we consider an image along with its annotation as a document. Further, we extend it to also include time and geographical position so that we can capture the contextual information from each image during the clustering process. This has appeared to be particularly useful on images gathered from online photo-sharing applications such as Flickr. Hence, our STC-based approach is aimed at dealing with the challenges induced by capturing contextual information from Flickr images and extracting related events. We evaluate our algorithm using different annotated datasets mainly gathered from Flickr. As part of this evaluation we investigate the effects of using different parameters, such as time and space granularities, and compare these effects. In addition, we evaluate the performance of our algorithm with respect to mining events from image collections. Our experimental results clearly demonstrate the effectiveness of our STC-based algorithm in extracting and clustering events.

Event Clusters Detection on Flickr Images Using a Suffix-Tree Structure

Image clustering is a problem that has been treated extensively in both Content-Based (CBIR) and Text-Based (TBIR) Image Retrieval Systems. In this paper, we propose a new image clustering approach that takes both annotation, time and geographical position into account. Our goal is to develop a clustering method that allows an image to be part of an event cluster. We extend a well-known clustering algorithm called Suffix Tree Clustering (STC), which was originally developed to cluster text documents using a document snippet. To be able to use this algorithm, we consider an image with annotation as a document. Then, we extend it to also include time and geographical position. This appears to be particularly useful on the images gathered from online photo-sharing applications such as Flickr. Here image tags are often subjective and incomplete. For this reason, clustering based on textual annotations alone is not enough to capture all context information related to an image. Our approach has been suggested to address this challenge. In addition, we propose a novel algorithm to extract event clusters. The algorithm is evaluated using an annotated dataset from Flickr, and a comparison between different granularity of time and space is provided.

Inter-Session Modeling for Session-Based Recommendation

In recent years, research has been done on applying Recurrent Neural Networks (RNNs) as recommender systems. Results have been promising, especially in the session-based setting where RNNs have been shown to outperform state-of-the-art models. In many of these experiments, the RNN could potentially improve the recommendations by utilizing information about the users past sessions, in addition to its own interactions in the current session. A problem for session-based recommendation, is how to produce accurate recommendations at the start of a session, before the system has learned much about the users current interests. We propose a novel approach that extends an RNN recommender to be able to process the users recent sessions, in order to improve recommendations. This is done by using a second RNN to learn from recent sessions, and predict the users interest in the current session. By feeding this information to the original RNN, it is able to improve its recommendations. Our experiments on two different datasets show that the proposed approach can significantly improve recommendations throughout the sessions, compared to a single RNN working only on the current session. The proposed model especially improves recommendations at the start of sessions, and is therefore able to deal with the cold start problem within sessions.

Exploratory analysis on heterogeneous tag-point patterns for ranking and extracting hot-spot related tags

The availability of a huge amount of geotagged resources on the web can be exploited to extract new useful information. We propose a set of estimators that are able to evaluate the degree of clustering of the spatial distribution of terms used to tag such geotagged resources. We introduce the concept of tag point pattern to derive indexes from the exploratory analysis by taking advantage of the second order Ripleys K-function, previously used in epidemiology, geo-statistics and ecology. The derived model estimates the degree of aggregation of the geotagged resources, taking into account the heterogeneity of the spatial distribution of the underlying population. Further, thanks to subsampling techniques, our approach is able to handle large datasets. Without losing of generality, we perform our experiments on a dataset derived Flickr pictures, as a use case. This dataset consists of tags that were extracted from a set of 1.2 million of pictures. We evaluate our proposed indexes with respect to their ability to extract tags related to geographical landmarks and hotspots. Our experiments show that we get good results using our estimators.

Geo-temporal distribution of tag terms for event-related image retrieval

Abstract Media sharing applications, such as Flickr and Panoramio, contain a large amount of pictures related to real life events. For this reason, the development of effective methods to retrieve these pictures is important, but still a challenging task. Recognizing this importance, and to improve the retrieval effectiveness of tag-based event retrieval systems, we propose a new method to extract a set of geographical tag features from raw geo-spatial profiles of user tags. The main idea is to use these features to select the best expansion terms in a machine learning-based query expansion approach. Specifically, we apply rigorous statistical exploratory analysis of spatial point patterns to extract the geo-spatial features. We use the features both to summarize the spatial characteristics of the spatial distribution of a single term, and to determine the similarity between the spatial profiles of two terms – i.e., term-to-term spatial similarity. To further improve our approach, we investigate the effect of combining our geo-spatial features with temporal features on choosing the expansion terms. To evaluate our method, we perform several experiments, including well-known feature analyzes. Such analyzes show how much our proposed geo-spatial features contribute to improve the overall retrieval performance. The results from our experiments demonstrate the effectiveness and viability of our method.

Exploring temporal proximity and spatial distribution of terms in web-based search of event-related images

Pictures in media sharing applications are increasingly accompanied with geotags. For this reason, we stress the importance of exploring the possibility of applying spatial, as well as the temporal dimensions in searching event-related pictures. Specifically, we propose extended query expansion models that exploit the information about the temporal neighbourhoods among pictures in a collection and leverage on the spatio-temporal distribution of the candidate expansion terms to re-weight and expand the initial query. To evaluate our approach, we conduct extensive experiments on a large dataset consisting of 88 million pictures from Flickr. The results from these experiments demonstrate the viability and effectiveness of our method with respect to retrieval performance, considering both a large dataset and query pictures with restricted size of terms.

Context-aware image semantic extraction in the social web

Media sharing applications such as Panoramio and Flickr contain a huge amount of pictures that need to be organized to facilitate browsing and retrieval. Such pictures are often surrounded by a set of metadata or image tags, constituting the image context. With the advent of the paradigm of Web 2.0 especially the past five years, the concept of image context has further evolved, allowing users to tag their own and other peoples pictures. Focusing on tagging, we distinguish between static and dynamic features. The set of static features include textual and visual features, as well as the contextual information. Further, we may identify other features belonging to the social context as a result of the usage within the media sharing applications. Due to their dynamic nature, we call these the dynamic set of features. In this work, we assume that every media uploaded contains both static and dynamic features. In addition, a user may be linked with other users with whom he/she shares common interests. This has resulted in a new series of challenges within the research field of semantic understanding. One of the main goals of this work is to address these challenges.

Geo-Temporal Mining and Searching of Events from Web-based Image Collections

The proliferation of Web- and social media-based photo-sharing applications have not only opened many possibilities but also resulted in new needs and challenges. They have resulted in a large amount of personal photos being available for public access. One of the most interesting characteristics of these data is that they are surrounded by 1) textual annotations, also called tags, which are intended to describe and categorize, by collective user efforts, the uploaded resources 2) temporal information referring to when a pictures has been taken and often by 3) a locational information describing where the picture has been taken. Despite the recent developments and technological advances in Web-based mediasharing applications, the continuously increasing amount of available information has made the access to the photos a demanding task. In general, we can address this challenge by allowing photo collections to be organized and browsed through the concept of events. We also believe most users are familiar with searching photo collections using events as starting points. Aiming at supporting the detection and search of event-related photos, this thesis proposes a novel framework for extracting pictures related to real-life events from a collection of Web-images by leveraging on their temporal geographical and textual annotations and comparing the proposed approach with existing related state-of-the-art approaches. Second, a set of geographical features is proposed describing the characteristics of the geographical profile of query terms deriving concepts from exploratory analysis. Third, the thesis provides two different tag-based search framework to improve the effectiveness of searching images related to events. The first framework is based on temporal and geographical proximity of query terms to the temporal neighbours of a given timestamped query, while the second framework is based on a novel machine-learning based query expansion method combining the heterogeneous textual, temporal and geographical similarity between query terms and candidate expansion terms for the selection of the expansion terms given a free text textual query. All the proposed methods have been evaluated by performing extensive experiments on real data gathered from media-sharing applications on the Web. Where possible, comparison with related techniques has been performed to reinforce the validity of the presented approaches. The proposed methods have shown promising results in both the extraction and clustering of event-related images and searching event-related pictures by using metrics from the state of the art. This doctoral work has been performed at the Department of Computer and Information Science at the Norwegian University of Science and Technology (NTNU) under advise of Associate Professor Heri Ramampiaro and with co-supervisors Associate Professor Roger Midtstraum and Associate Professor Randi Karlsen. Professor Ramesh Jain (University of California Irvine), Assistant Professor Claudia Hauff (TU Delft) and Associate Professor Trond Aalberg (NTNU) served as dissertation committee members. Available online at: http://www.idi.ntnu.no/research/doctor_theses/ruocco.pdf.

Event-related image retrieval: exploring geographical and temporal distribution of user tags

Providing effective tools to retrieve event-related pictures within media-sharing applications, such as Flickr, is an important but challenging task. One interesting aspect is to search pictures related to a specific event with a given annotated image. Most existing methods have focused on doing this by extracting visual features from the pictures. However, pictures in media-sharing applications increasingly come with location information, such as geotags. Therefore, we stress the importance of exploring the possibility to leverage on the geographical and temporal distribution of terms in a tag-based search process, within event-related image retrieval. Specifically, we propose extended query expansion models that exploit the information about the temporal neighborhoods among pictures in a collection, and leverage on the geo-temporal distribution of the candidate expansion terms to reweight and expand the initial query. To evaluate our approach, we conduct extensive experiments on a dataset consisting of pictures from Flickr. The results from these experiments demonstrate the effectiveness of our method with respect to retrieval performance.