Spiros Nikolopoulos

Evidence-Driven Image Interpretation by Combining Implicit and Explicit Knowledge in a Bayesian Network

Computer vision techniques have made considerable progress in recognizing object categories by learning models that normally rely on a set of discriminative features. However, in contrast to human perception that makes extensive use of logic-based rules, these models fail to benefit from knowledge that is explicitly provided. In this paper, we propose a framework that can perform knowledge-assisted analysis of visual content. We use ontologies to model the domain knowledge and a set of conditional probabilities to model the application context. Then, a Bayesian network is used for integrating statistical and explicit knowledge and performing hypothesis testing using evidence-driven probabilistic inference. In addition, we propose the use of a focus-of-attention (FoA) mechanism that is based on the mutual information between concepts. This mechanism selects the most prominent hypotheses to be verified/tested by the BN, hence removing the need to exhaustively test all possible combinations of the hypotheses set. We experimentally evaluate our framework using content from three domains and for the following three tasks: 1) image categorization; 2) localized region labeling; and 3) weak annotation of video shot keyframes. The results obtained demonstrate the improvement in performance compared to a set of baseline concept classifiers that are not aware of any context or domain knowledge. Finally, we also demonstrate the ability of the proposed FoA mechanism to significantly reduce the computational cost of visual inference while obtaining results comparable to the exhaustive case.

Deep Learning Advances in Computer Vision with 3D Data: A Survey

Deep learning has recently gained popularity achieving state-of-the-art performance in tasks involving text, sound, or image processing. Due to its outstanding performance, there have been efforts to apply it in more challenging scenarios, for example, 3D data processing. This article surveys methods applying deep learning on 3D data and provides a classification based on how they exploit them. From the results of the examined works, we conclude that systems employing 2D views of 3D data typically surpass voxel-based (3D) deep models, which however, can perform better with more layers and severe data augmentation. Therefore, larger-scale datasets and increased resolutions are required.

High order pLSA for indexing tagged images

This work presents a method for the efficient indexing of tagged images. Tagged images are a common resource of social networks and occupy a large portion of the social media stream. Their basic characteristic is the co-existence of two heterogeneous information modalities, i.e. visual and tag, which refer to the same abstract meaning. This multi-modal nature of tagged images makes their efficient indexing a challenging task that apart from dealing with the heterogeneity of modalities, it needs to also exploit their complementary information capacity. Towards this objective, we propose the extension of probabilistic latent semantic analysis to higher order, so as to become applicable for more than two observable variables. Then, by treating images, visual features and tags as the three observable variables of an aspect model, we learn a space of latent topics that incorporates the semantics of both visual and tag information. Our novelty is on using the cross-modal dependencies learned from a corpus of images to approximate the joint distribution of the observable variables. By penalizing the co-existence of visual content and tags that are known from experience to exhibit low dependency, we manage to filter out the effect of noisy content in the resulting latent space.

ImproveMyCity: an open source platform for direct citizen-government communication

ImproveMyCity is an open source platform that enables residents to directly report to their public administration local issues about their neighborhood such as discarded trash bins, faulty street lights, broken tiles on sidewalks, illegal advertising boards, etc. The reported issues are automatically transmitted to the appropriate office in public administration so as to schedule their settlement. Reporting is feasible both through a web- and a smartphone-based front-end that adopt a map-based visualization, which makes reporting a user-friendly and intriguing process. The management and routing of incoming issues is performed through a back-end infrastructure that serves as an integrated management system with easy to use interfaces. Apart from reporting a new issue, both front-ends allow the citizens to add comments or vote on existing issues, which adds a social dimension on the collected content. Finally, the platform makes also provision for informing the citizens about the progress status of the reported issue and in this way facilitate the establishment of a two-way dialogue between the citizen and public administration.

Image replica detection system utilizing R-trees and linear discriminant analysis

This manuscript introduces a novel system for content-based identification of image replicas. The proposed approach utilizes image resemblance for deciding whether a test image has been replicated from a certain original or not. We formulate replica detection as a classification problem and show that we can optimize efficiency on a per query basis by dynamically solving a reduced multiclass problem. For this purpose, we investigate the effective coupling of multidimensional indexing and machine learning approaches and we aim to achieve replica detection through the training of classifiers with distortions expected in a replica. Visual descriptors are indexed using an R-tree based multidimensional structure for fast image retrieval. Cases unsuccessfully handled by the R-tree are resolved by a multiclass classifier operating on the transformed feature space that results from the application of linear discriminant analysis (LDA) and principal component analysis (PCA). Experimental results show that the proposed system can identify replicas with high accuracy and facilitate a wide range of applications such as copyright protection, content-based monitoring, content-aware multimedia management, etc.

Semi-supervised object recognition using flickr images

In this work we present an algorithm for extracting region level annotations from flickr images using a small set of manually labelled regions to guide the selection process. More specifically, we construct a set of flickr images that focuses on a certain concept and apply a novel graph based clustering algorithm on their regions. Then, we select the cluster or clusters that correspond to the examined concept guided by the manually labelled data. Experimental results show that although the obtained regions are of lower quality compared to the manually labelled regions, the gain in effort compensates for the loss in performance.

Combining Multi-modal Features for Social Media Analysis

In this chapter we discuss methods for efficiently modeling the diverse information carried by social media. The problem is viewed as a multi-modal analysis process where specialized techniques are used to overcome the obstacles arising from the heterogeneity of data. Focusing at the optimal combination of low-level features (i.e., early fusion), we present a bio-inspired algorithm for feature selection that weights the features based on their appropriateness to represent a resource. Under the same objective of optimal feature combination we also examine the use of pLSA-based aspect models, as the means to define a latent semantic space where heterogeneous types of information can be effectively combined. Tagged images taken from social sites have been used in the characteristic scenarios of image clustering and retrieval, to demonstrate the benefits of multi-modal analysis in social media.

A comparative study on mobile visual recognition

In this work we perform an extensive comparative study of approaches for mobile visual recognition by simultaneously evaluating the performance and the computational cost of state-of-the-art key-point detection, feature extraction and encoding algorithms. Every step is independently tested so that its contribution to the final computational cost can be measured. The widely used OpenCV library is utilized for the implementation of the algorithms, while the evaluation is performed on the PASCAL VOC 2007 dataset, a challenging real world dataset crawled from the web. Our study identifies the algorithmic configurations that manage to optimally balance performance and computational cost, and provide a viable solution for real time mobile visual recognition.

Leveraging social media for training object detectors

The fact that most users tend to tag images emotionally rather than realistically makes social datasets inherently flawed from a computer vision perspective. On the other hand they can be particularly useful due to their social context and their potential to grow arbitrary big. Our work shows how a combination of techniques operating on both tag and visual information spaces, manages to leverage the associated weak annotations and produce region-detail training samples. In this direction we make some theoretical observations relating the robustness of the resulting models, the accuracy of the analysis algorithms and the amount of processed data. Experimental evaluation performed against manually trained object detectors reveals the strengths and weaknesses of our approach.

A Novel Replica Detection System using Binary Classifiers, R-Trees, and PCA

Replica detection is a prerequisite for the discovery of copyright infringement and detection of illicit content. For this purpose, content-based systems can be an efficient alternative to watermarking. Rather than imperceptibly embedding a signal, content-based systems rely-on image similarity. Certain content-based systems use adaptive classifiers to detect replicas. In such systems, a suspect image is tested against every original, which can become computationally prohibitive as the number of original images grows. In this paper, we propose using R-tree indexing to decrease the necessary number of comparisons and rapidly select the most likely originals. Experimental results show that the proposed system performs very satisfactorily and that up to 99.3% of the originals can be discarded before applying the binary classifiers.