Julia Vogel

Semantic Modeling of Natural Scenes for Content-Based Image Retrieval

In this paper, we present a novel image representation that renders it possible to access natural scenes by local semantic description. Our work is motivated by the continuing effort in content-based image retrieval to extract and to model the semantic content of images. The basic idea of the semantic modeling is to classify local image regions into semantic concept classes such as water, rocks, or foliage. Images are represented through the frequency of occurrence of these local concepts. Through extensive experiments, we demonstrate that the image representation is well suited for modeling the semantic content of heterogenous scene categories, and thus for categorization and retrieval.The image representation also allows us to rank natural scenes according to their semantic similarity relative to certain scene categories. Based on human ranking data, we learn a perceptually plausible distance measure that leads to a high correlation between the human and the automatically obtained typicality ranking. This result is especially valuable for content-based image retrieval where the goal is to present retrieval results in descending semantic similarity from the query.

Natural Scene Retrieval based on a Semantic Modeling Step

In this paper, we present an approach for the retrieval of natural scenes based on a semantic modeling step. Semantic modeling stands for the classification of local image regions into semantic classes such as grass, rocks or foliage and the subsequent summary of this information in so-called concept-occurrence vectors. Using this semantic representation, images from the scene categories coasts, rivers/lakes, forests, plains, mountains and sky/clouds are retrieved. We compare two implementations of the method quantitatively on a visually diverse database of natural scenes. In addition, the semantic modeling approach is compared to retrieval based on low-level features computed directly on the image. The experiments show that semantic modeling leads in fact to better retrieval performance.

A Semantic Typicality Measure for Natural Scene Categorization

We propose an approach to categorize real-world natural scenes based on a semantic typicality measure. The proposed typicality measure allows to grade the similarity of an image with respect to a scene category. We argue that such a graded decision is appropriate and justified both from a human’s perspective as well as from the image-content point of view. The method combines bottom-up information of local semantic concepts with the typical semantic content of an image category. Using this learned category representation the proposed typicality measure also quantifies the semantic transitions between image categories such as coasts, rivers/lakes, forest, plains, mountains or sky/clouds. The method is evaluated quantitatively and qualitatively on a database of natural scenes. The experiments show that the typicality measure well represents the diversity of the given image categories as well as the ambiguity in human judgment of image categorization.

Categorization of natural scenes: Local versus global information and the role of color

Categorization of scenes is a fundamental process of human vision that allows us to efficiently and rapidly analyze our surroundings. Several studies have explored the processes underlying human scene categorization, but they have focused on processing global image information. In this study, we present both psychophysical and computational experiments that investigate the role of local versus global image information in scene categorization. In a first set of human experiments, categorization performance is tested when only local or only global image information is present. Our results suggest that humans rely on local, region-based information as much as on global, configural information. In addition, humans seem to integrate both types of information for intact scene categorization. In a set of computational experiments, human performance is compared to two state-of-the-art computer vision approaches that have been shown to be psychophysically plausible and that model either local or global information. In addition to the influence of local versus global information, in a second series of experiments, we investigated the effect of color on the categorization performance of both the human observers and the computational model. Analysis of the human data suggests that color is an additional channel of perceptual information that leads to higher categorization results at the expense of increased reaction times in the intact condition. However, it does not affect reaction times when only local information is present. When color is removed, the employed computational model follows the relative performance decrease of human observers for each scene category and can thus be seen as a perceptually plausible model for human scene categorization based on local image information.

Categorization of natural scenes: local vs. global information

Understanding the robustness and rapidness of human scene categorization has been a focus of investigation in the cognitive sciences over the last decades. At the same time, progress in the area of image understanding has prompted computer vision researchers to design computational systems that are capable of automatic scene categorization. Despite these efforts, a framework describing the processes underlying human scene categorization that would enable efficient computer vision systems is still missing. In this study, we present both psychophysical and computational experiments that aim to make a further step in this direction by investigating the processing of local and global information in scene categorization. In a set of human experiments, categorization performance is tested when only local or only global image information is present. Our results suggest that humans rely on local, region-based information as much as on global, configural information. In addition, humans seem to integrate both types of information for intact scene categorization. In a set of computational experiments, human performance is compared to two state-of-the-art computer vision approaches that model either local or global information.

On Performance Characterization and Optimization for Image Retrieval

In content-based image retrieval (CBIR) performance characterization is easily being neglected. A major difficulty lies in the fact that ground truth and the definition of benchmarks are extremely user and application dependent. This paper proposes a two-stage CBIR framework which allows to predict the behavior of the retrieval system as well as to optimize its performance. In particular, it is possible to maximize precision, recall, or jointly precision and recall. The framework is based on the detection of high-level concepts in images. These concepts correspond to vocabulary users can query the database with. Performance optimization is carried out on the basis of the user query, the performance of the concept detectors, and an estimated distribution of the concepts in the database. The optimization is transparent to the user and leads to a set of internal parameters that optimize the succeeding retrieval. Depending only on the query and the desired concept, precision and recall of the retrieval can be increased by up to 40%. The paper discusses the theoretical and empirical results of the optimization as well as its dependency on the estimate of the concept distribution.

A non-myopic approach to visual search

We show how a greedy approach to visual search - i.e., directly moving to the most likely location of the target - can be suboptimal, if the target object is hard to detect. Instead it is more efficient and leads to higher detection accuracy to first look for other related objects, that are easier to detect. These provide contextual priors for the target that make it easier to find. We demonstrate this in simulation using POMDP models, focussing on two special cases: where the target object is contained within the related object, and where the target object is spatially adjacent to the related object.

A psychophysically plausible model for typicality ranking of natural scenes

Natural scenes constitute a very heterogeneous stimulus class. Each semantic category contains exemplars of varying typicality. It is, therefore, an interesting question whether humans can categorize natural scenes consistently into a relatively small number of categories, such as, coasts, rivers/lakes, forests, plains, and mountains. This is particularly important for applications, such as, image retrieval systems. Only if typicality is consistently perceived across different individuals, a general image-retrieval system makes sense. In this study, we use psychophysics and computational modeling to gain a deeper understanding of scene typicality. In the first psychophysical experiment, we used a forced-choice categorization task in which each of 250 natural scenes had to be classified into one of the following five categories: coasts, rivers/lakes, forests, plains, and mountains. In the second experiment, the typicality of each scene had to be rated on a 50-point scale for each of the five categories. The psychophysical results show high consistency between participants not only in the categorization of natural scenes, but also in the typicality ratings. In order to model human perception, we then employ a computational approach that uses an intermediate semantic modeling step by extracting local semantic concepts, such as, rock, water, and sand. Based on the human typicality ratings, we learn a psychophysically plausible distance measure that leads to a high correlation between the computational and the human ranking of natural scenes. Interestingly, model comparisons without a semantic-modeling step correlated much less with human performance, suggesting that our model is psychophysically very plausible.

Frequency domain step-size control in non-stationary environments

The presented algorithm detects changes of the loud-speaker enclosure microphone (LEM) system in an acoustic echo cancellation setup under non-stationary noise conditions. Changes of the LEM system entail a mismatch of the cancellation filter, consequently making the beforehand cancelled echos audible again. The new algorithm avoids the reduction of the adaptation speed after these changes. System changes are detected by monitoring the different behavior of two coherences: these changes have only a small impact on the coherence between the microphone and the loudspeaker signal, whereas they significantly influence the coherence between the microphone signal and the output of the cancellation filter.

Performance of differentially detected /spl pi//4 DQPSK in the presence of IQ phase imbalance

The effect of circuit level distortions on system level performance is becoming an important issue in optimizing the design of low power receivers. Circuit level distortions which in the past were insignificant in full power designs, must be accounted for in the link budgets of low power systems. In this paper, we present an analysis that demonstrates and quantifies the effect of one such distortion. We present a closed form expression for the BER of differentially detected /spl pi//4 DQPSK in the presence of AWGN and distorted by an IQ phase imbalance in the receiver.