Eva Hörster

On the optimal placement of multiple visual sensors

Many novel multimedia systems and applications use visual sensor arrays. An important issue in designing sensor arrays is the appropriate placement of the visual sensors such that they achieve a predefined goal. In this paper we focus on the placement with respect to maximizing coverage or achieving coverage at a certain resolution. We identify and consider four different problems: maximizing coverage subject to a given number of cameras (a) or a maximum total price of the sensor array (b), optimizing camera poses given fixed locations (c), and minimizing the cost of a sensor array given a minimally required percentage of coverage (d). To solve these problems, we propose different algorithms. Our approaches can be subdivided into algorithms which give a global optimum solution and heuristics which solve the problem within reasonable time and memory consumption at the cost of not necessarily determining the global optimum. We also present a user-interface to enter and edit the spaces under analysis, the optimization problems as well as the other setup parameters. The different algorithms are experimentally evaluated and results are presented. The results show that the algorithms work well and are suited for different practical applications. For the final paper it is planned to have the user interface running as a web service.

Image retrieval on large-scale image databases

Online image repositories such as Flickr contain hundreds of millions of images and are growing quickly. Along with that the needs for supporting indexing, searching and browsing is becoming more and more pressing. In this work we will employ the image content as a source of information to retrieve images. We study the representation of images by Latent Dirichlet Allocation (LDA) models for content-based image retrieval. Image representations are learned in an unsupervised fashion, and each image is modeled as the mixture of topics/object parts depicted in the image. This allows us to put images into subspaces for higher-level reasoning which in turn can be used to find similar images. Different similarity measures based on the described image representation are studied. The presented approach is evaluated on a real world image database consisting of more than 246,000 images and compared to image models based on probabilistic Latent Semantic Analysis (pLSA). Results show the suitability of the approach for large-scale databases. Finally we incorporate active learning with user relevance feedback in our framework, which further boosts the retrieval performance.

Multilayer pLSA for multimodal image retrieval

It is current state of knowledge that our neocortex consists of six layers [10]. We take this knowledge from neuroscience as an inspiration to extend the standard single-layer probabilistic Latent Semantic Analysis (pLSA) [13] to multiple layers. As multiple layers should naturally handle multiple modalities and a hierarchy of abstractions, we denote this new approach multilayer multimodal probabilistic Latent Semantic Analysis (mm-pLSA). We derive the training and inference rules for the smallest possible non-degenerated mm-pLSA model: a model with two leaf-pLSAs (here from two different data modalities: image tags and visual image features) and a single top-level pLSA node merging the two leaf-pLSAs. From this derivation it is obvious how to extend the learning and inference rules to more modalities and more layers. We also propose a fast and strictly stepwise forward procedure to initialize bottom-up the mm-pLSA model, which in turn can then be post-optimized by the general mm-pLSA learning algorithm. We evaluate the proposed approach experimentally in a query-by-example retrieval task using 50-dimensional topic vectors as image models. We compare various variants of our mm-pLSA system to systems relying solely on visual features or tag features and analyze possible pitfalls of the mm-pLSA training. It is shown that the best variant of the the proposed mm-pLSA system outperforms the unimodal systems by approximately 19% in our query-by-example task.

Approximating Optimal Visual Sensor Placement

Many novel multimedia applications use visual sensor arrays. In this paper we address the problem of optimally placing multiple visual sensors in a given space. Our linear programming approach determines the minimum number of cameras needed to cover the space completely at a given sampling frequency. Simultaneously it determines the optimal positions and poses of the visual sensors. We also show how to account for visual sensors with different properties and costs if more than one kind is available, and report performance results.

Deep networks for image retrieval on large-scale databases

Currently there are hundreds of millions (high-quality) images in online image repositories such as Flickr. This makes is necessary to develop new algorithms that allow for searching and browsing in those large-scale databases. In this work we explore deep networks for deriving a low-dimensional image representation appropriate for image retrieval. A deep network consisting of multiple layers of features aims to capture higher order correlations between basic image features. We will evaluate our approach on a real world large-scale image database and compare it to image representations based on topic models. Our results show the suitability of the approach for very large databases.

Multimodal ranking for image search on community databases

Searching for relevant images given a query term is an important task in nowadays large-scale community databases. The image ranking approach presented in this work represents an image collection as a graph that is built using a multimodal similarity measure based on visual features and user tags. We perform a random walk on this graph to find the most common images. Further we discuss several scalability issues of the proposed approach and show how in this framework queries can be answered fast. Experimental results validate the effectiveness of the presented algorithm.

Continuous visual vocabulary modelsfor pLSA-based scene recognition

Topic models such as probabilistic Latent Semantic Analysis (pLSA) and Latent Dirichlet Allocation (LDA) have been shown to perform well in various image content analysis tasks. However, due to the origin of these models from the text domain, almost all prior work uses discrete vocabularies even when applied in the image domain. Thus in these works the continuous local features used to describe an image need to be quantized to fit the model. In this work we will propose and evaluate three different extensions to the pLSA framework so that words are modeled as continuous feature vector distributions rather than crudely quantized high-dimensional descriptors. The performance of these continuous vocabulary models are compared in an automatic scene recognition task. Our experiments clearly show that the continuous approaches outperform the standard pLSA model. In this paper all required equations for parameter estimation and inference are given for each of the three models.

Calibrating and optimizing poses of visual sensors in distributed platforms

Many novel multimedia, home entertainment, visual surveillance and health applications use multiple audio-visual sensors. We present a novel approach for position and pose calibration of visual sensors, i.e., cameras, in a distributed network of general purpose computing devices (GPCs). It complements our work on position calibration of audio sensors and actuators in a distributed computing platform (Raykar et al. in proceedings of ACM Multimedia ‘03, pp. 572–581, 2003). The approach is suitable for a wide range of possible – even mobile – setups since (a) synchronization is not required, (b) it works automatically, (c) only weak restrictions are imposed on the positions of the cameras, and (d) no upper limit on the number of cameras under calibration is imposed. Corresponding points across different camera images are established automatically. Cameras do not have to share one common view. Only a reasonable overlap between camera subgroups is necessary. The method has been sucessfully tested in numerous multi-camera environments with a varying number of cameras and has proven itself to work extremely accurate. Once all distributed visual sensors are calibrated, we focus on post-optimizing their poses to increase coverage of the space observed. A linear programming approach is derived that determines jointly for each camera the pan and tilt angle that maximizes the coverage of the space at a given sampling frequency. Experimental results clearly demonstrate the gain in visual coverage.

Multimodal pLSA on visual features and tags

This work studies a new approach for image retrieval on largescale community databases. Our proposed system explores two different modalities: visual features and communitygenerated metadata, such as tags. We use topic models to derive a high-level representation appropriate for retrieval for each of our images in the database. We evaluate the proposed approach experimentally in a query-by-example retrieval task and compare our results to systems relying solely on visual features or tag features. It is shown that the proposed multimodal system outperforms the unimodal systems by approximately 36%.

Comparing Local Feature Descriptors in pLSA-Based Image Models

Probabilistic models with hidden variables such as probabilistic Latent Semantic Analysis (pLSA) and Latent Dirichlet Allocation (LDA) have recently become popular for solving several image content analysis tasks. In this work we will use a pLSA model to represent images for performing scene classification. We evaluate the influence of the type of local feature descriptor in this context and compare three different descriptors. Moreover we also examine three different local interest region detectors with respect to their suitability for this task. Our results show that two examined local descriptors, the geometric blur and the self-similarity feature, outperform the commonly used SIFT descriptor by a large margin.