Evaggelos Spyrou

Retrieving landmark and non-landmark images from community photo collections

State of the art data mining and image retrieval in community photo collections typically focus on popular subsets, e.g. images containing landmarks or associated to Wikipedia articles. We propose an image clustering scheme that, seen as vector quantization compresses a large corpus of images by grouping visually consistent ones while providing a guaranteed distortion bound. This allows us, for instance, to represent the visual content of all thousands of images depicting the Parthenon in just a few dozens of scene maps and still be able to retrieve any single, isolated, non-landmark image like a house or graffiti on a wall. Starting from a geo-tagged dataset, we first group images geographically and then visually, where each visual cluster is assumed to depict different views of the the same scene. We align all views to one reference image and construct a 2D scene map by preserving details from all images while discarding repeating visual features. Our indexing, retrieval and spatial matching scheme then operates directly on scene maps. We evaluate the precision of the proposed method on a challenging one-million urban image dataset.

VIRaL: Visual Image Retrieval and Localization

New applications are emerging every day exploiting the huge data volume in community photo collections. Most focus on popular subsets, e.g., images containing landmarks or associated to Wikipedia articles. In this work we are concerned with the problem of accurately finding the location where a photo is taken without needing any metadata, that is, solely by its visual content. We also recognize landmarks where applicable, automatically linking them to Wikipedia. We show that the time is right for automating the geo-tagging process, and we show how this can work at large scale. In doing so, we do exploit redundancy of content in popular locations—but unlike most existing solutions, we do not restrict to landmarks. In other words, we can compactly represent the visual content of all thousands of images depicting e.g., the Parthenon and still retrieve any single, isolated, non-landmark image like a house or a graffiti on a wall. Starting from an existing, geo-tagged dataset, we cluster images into sets of different views of the same scene. This is a very efficient, scalable, and fully automated mining process. We then align all views in a set to one reference image and construct a 2D scene map. Our indexing scheme operates directly on scene maps. We evaluate our solution on a challenging one million urban image dataset and provide public access to our service through our online application, VIRaL.

Using Visual Context and Region Semantics for High-Level Concept Detection

In this paper we investigate detection of high-level concepts in multimedia content through an integrated approach of visual thesaurus analysis and visual context. In the former, detection is based on model vectors that represent image composition in terms of region types, obtained through clustering over a large data set. The latter deals with two aspects, namely high-level concepts and region types of the thesaurus, employing a model of a priori specified semantic relations among concepts and automatically extracted topological relations among region types; thus it combines both conceptual and topological context. A set of algorithms is presented, which modify either the confidence values of detected concepts, or the model vectors based on which detection is performed. Visual context exploitation is evaluated on TRECVID and Corel data sets and compared to a number of related visual thesaurus approaches.

Video-based measurements for wireless capsule endoscope tracking

The wireless capsule endoscope is a swallowable medical device equipped with a miniature camera enabling the visual examination of the gastrointestinal (GI) tract. It wirelessly transmits thousands of images to an external video recording system, while its location and orientation are being tracked approximately by external sensor arrays. In this paper we investigate a video-based approach to tracking the capsule endoscope without requiring any external equipment. The proposed method involves extraction of speeded up robust features from video frames, registration of consecutive frames based on the random sample consensus algorithm, and estimation of the displacement and rotation of interest points within these frames. The results obtained by the application of this method on wireless capsule endoscopy videos indicate its effectiveness and improved performance over the state of the art. The findings of this research pave the way for a cost-effective localization and travel distance measurement of capsule endoscopes in the GI tract, which could contribute in the planning of more accurate surgical interventions.

Concept detection and keyframe extraction using a visual thesaurus

This paper presents a video analysis approach based on concept detection and keyframe extraction employing a visual thesaurus representation. Color and texture descriptors are extracted from coarse regions of each frame and a visual thesaurus is constructed after clustering regions. The clusters, called region types, are used as basis for representing local material information through the construction of a model vector for each frame, which reflects the composition of the image in terms of region types. Model vector representation is used for keyframe selection either in each video shot or across an entire sequence. The selection process ensures that all region types are represented. A number of high-level concept detectors is then trained using global annotation and Latent Semantic Analysis is applied. To enhance detection performance per shot, detection is employed on the selected keyframes of each shot, and a framework is proposed for working on very large data sets.

Capsule endoscope localization based on visual features

Computational analysis of wireless capsule endoscopy (WCE) videos has already proved its potentials in the discovery or characterization of lesions and in the reduction of the time required by the endoscopists to perform the examination. An open problem that has only partially been addressed is the localization of the capsule endoscope in the gastrointestinal (GI) tract. Previous works have been based mainly on external, wearable, sensors. In this paper we propose a novel approach based solely on visual information extracted from WCE videos. This approach is based on a feature tracking method for visual odometry, which enables the estimation of both the rotation and the displacement of a capsule endoscope from reference anatomical landmarks. Its implementation is based on a novel, open access Java Video Analysis (JVA) framework, which enables quick and standardized development of intelligent video analysis applications. The experimental evaluation presented in this paper, indicates the feasibility of the proposed methodological approach and the efficiency of its implementation.

A region thesaurus approach for high-level concept detection in the natural disaster domain

This paper presents an approach on high-level feature detection using a region thesaurus. MPEG-7 features are locally extracted from segmented regions and for a large set of images. A hierarchical clustering approach is applied and a relatively small number of region types is selected. This set of region types defines the region thesaurus. Using this thesaurus, low-level features are mapped to high-level concepts as model vectors. This representation is then used to train support vector machine-based feature detectors. As a next step, latent semantic analysis is applied on the model vectors, to further improve the analysis performance. High-level concepts detected derive from the natural disaster domain.

A survey on Flickr multimedia research challenges

Multimedia content sharing within social networks has become one of the most interesting and trending research fields over the last few years. This undoubted emerge of related research works is rather twofold, namely it includes both the analysis and management techniques of the content itself, as well as new ways for its accompanied meaningful interpretation and exploitation. In this paper, we review the recent advances in the above fields in the humanistic framework of the popular Flickr social network. In addition, the major research challenges in the area are demonstrated and discussed, which include current state-of-the-art approaches with respect to interesting humanistic data collection and interpretation research fields, such as multimedia information retrieval, (semi-) automatic tag manipulation, travel applications, semantic knowledge extraction, human activity tracking, as well as related benchmarking efforts. At the end of this survey, we also discuss the main challenges and propose a number of future research directions for interested fellow researchers to continue investigation in the field.

A Semantic Multimedia Analysis Approach Utilizing a Region Thesaurus and LSA

This paper presents an approach on high-level feature detection within video documents, using a region thesaurus and latent semantic analysis. A video shot is represented by a single keyframe. MPEG-7 features are extracted from coarse regions of it. A clustering algorithm is applied on all extracted regions and a region thesaurus is constructed. Its use is to assist to the mapping of low- to high-level features by a model vector representation. Latent semantic analysis is then applied on the model vectors to exploit the latent relations among region types aiming to improve detection performance. The proposed approach is thoroughly examined using TRECVID 2007 development data.

Keyframe Extraction Using Local Visual Semantics in the Form of a Region Thesaurus

Edge directions histograms are widely used as an image descriptor for image retrieval and recognition applications. Edges represent textures and are also representative of the image shapes. In this work a histogram of the edge pixel directions is defined for image description. The edges detected with the canny algorithm will be described in two different scales in four directions. In the lower scale the image is divided into 16 sub-images, and a descriptor with 64 bins results. In the higher scale, as no image division is done because only the most important image features will be present, 4 bins result. A total of 68 bins are used to describe the image in scale-space. Images will be compared using the Euclidean distance between histograms. The provided results will be compared with the ones that result from the use of the histogram in the low scale only. Improved classification using the nearest class mean and neural networks will be used. A higher level semantic annotation, based on this low level descriptor that results from the multiscale image analysis, will be extracted.This paper presents an approach for efficient keyframe extraction, using local semantics in form of a region thesaurus. More specifically, certain MPEG-7 color and texture features are locally extracted from keyframe regions. Then, using a hierarchical clustering approach a local region thesaurus is constructed to facilitate the description of each frame in terms of higher semantic features. The thesaurus consists of the most common region types that are encountered within the video shot, along with their synonyms. These region types carry semantic information. Each keyframe is represented by a vector consisting of the degrees of confidence of the existence of all region types within this shot. Using this keyframe representation, the most representative keyframe is then selected for each shot. Where a single keyframe is not adequate, using the same algorithm and exploiting the presence of the region types of the visual thesaurus, more keyframes are extracted.