Stefan Kluckner

Semantic classification in aerial imagery by integrating appearance and height information

In this paper we present an efficient technique to obtain accurate semantic classification on the pixel level capable of integrating various modalities, such as color, edge responses, and height information. We propose a novel feature representation based on Sigma Points computations that enables a simple application of powerful covariance descriptors to a multi-class randomized forest framework. Additionally, we include semantic contextual knowledge using a conditional random field formulation. In order to achieve a fair comparison to state-of-the-art methods our approach is first evaluated on the MSRC image collection and is then demonstrated on three challenging aerial image datasets Dallas, Graz, and San Francisco. We obtain a full semantic classification on single aerial images within two minutes. Moreover, the computation time on large scale imagery including hundreds of images is investigated.

A 3D Teacher for Car Detection in Aerial Images

This paper demonstrates how to reduce the hand labeling effort considerably by 3D information in an object detection task. In particular, we demonstrate how an efficient car detector for aerial images with minimal hand labeling effort can be build. We use an on-line boosting algorithm to incrementally improve the detection results. Initially, we train the classifier with a single positive (car) example, randomly drawn from a fixed number of given samples. When applying this detector to an image we obtain many false positive detections. We use information from a stereo matcher to detect some of these false positives (e.g. detected cars on a facade) and feed back this information to the classifier as negative updates. This improves the detector considerably, thus reducing the number of false positives. We show that we obtain similar results to hand labeling by iteratively applying this strategy. The performance of our algorithm is demonstrated on digital aerial images of urban environments.

Recognizing cars in aerial imagery to improve orthophotos

The automatic creation of 3D models of urban spaces has become a very active field of research. This has been inspired by recent applications in the location-awareness on the Internet, as demonstrated in maps.live.com and similar websites. The level of automation in creating 3D city models has increased considerably, and has benefited from an increase in the redundancy of the source imagery, namely digital aerial photography. In this paper we argue that the next big step forward is to replace photographic texture by an interpretation of what the texture describes, and to achieve this fully automatically. One calls the result semantic knowledge. For example we want to know that a certain part of the image is a car, a person, a building, a tree, a shrub, a window, a door, instead of just a collection of 3D points or triangles with a superimposed photographic texture. We investigate object recognition methods to make this next big step. We demonstrate an early result of using the on-line variant of a Boosting algorithm to indeed detect cars in aerial digital imagery to a satisfactory and useful level of completeness. And we show that we can use this semantic knowledge to produce improved orthophotos. We expect that also the 3D models will be improved by the knowledge of cars.

Semantic Image Classification Using Consistent Regions and Individual Context

This paper proposes an efficient approach for semantic image classification by integrating additional contextual constraints such as class co-occurrences into a randomized forest classification framework. The randomized forest classifier performs an initial yet local classification on the pixel level by using powerful covariance matrix based descriptors as feature representation. Furthermore, we exploit multiple unsupervised image partitions to provide a reliable spatial region support and to capture the real object boundaries. An information theoretic driven approach detects consistently classified regions and generates a representative segmentation incorporating the classification result on the pixel level. Moreover, we use a conditional random field formulation to obtain a final labeling including context information individually generated for each test image. To illustrate state-of-the-art performance, we run experiments on the two versions of the MSRC [21] dataset with 9 and 21 object classes and on the PASCAL VOC2007 [5] image collection.

Semantic classification by covariance descriptors within a randomized forest

This paper investigates an approach to perform semantic classification in aerial imagery by compactly integrating multiple feature cues, like appearance and 3D height information. We therefore propose a novel technique to incorporate powerful covariance region descriptors into the decision nodes of a randomized forest framework efficiently. The concept of finding reliable binary splits is based on repeated random sampling of distributions that are specified by mean vectors and covariance matrices. The sampling strategy is related to Monte Carlo simulations and perfectly fits the learning strategy of randomized decision trees, while the covariance descriptors are exploited to perform a plausible feature cue integration. To show state-of-the-art performance, we first evaluate our proposed approach on the MSRC dataset including 21 object classes. Then, we illustrate how an additional integration of 3D information improves the classification accuracy in real world aerial images taken from Dallas, San Francisco, and Graz. In addition, we use the available camera data and 3D information to combine the overlapping per-image classifications into a large-scale semantic description map that is directly applicable to virtual or procedural 3D modeling of urban environments.

Aerial Computer Vision for a 3D Virtual Habitat

An Internet-embedded 3D model of a human habitat is feasible and useful. In lieu of a 2D Earth map, the authors describe a 3D model with human-scale objects in urban spaces and inside buildings. Here, they focus on information from aerial imagery.

Object Tracking by Structure Tensor Analysis

Covariance matrices have recently been a popular choice for versatile tasks like recognition and tracking due to their powerful properties as local descriptor and their low computational demands. This paper outlines similarities of covariance matrices to the well-known structure tensor. We show that the generalized version of the structure tensor is a powerful descriptor and that it can be calculated in constant time by exploiting the properties of integral images. To measure the similarities between several structure tensors, we describe an approximation scheme which allows comparison in a Euclidean space. Such an approach is also much more efficient than the common, computationally demanding Riemannian Manifold distances. Experimental evaluation proves the applicability for the task of object tracking demonstrating improved performance compared to covariance tracking.

Exploiting redundancy for aerial image fusion using convex optimization

Image fusion in high-resolution aerial imagery poses a challenging problem due to fine details and complex textures. In particular, color image fusion by using virtual orthographic cameras offers a common representation of overlapping yet perspective aerial images. This paper proposes a variational formulation for a tight integration of redundant image data showing urban environments. We introduce an efficient wavelet regularization which enables a natural-appearing recovery of fine details in the images by performing joint inpainting and denoising from a given set of input observations. Our framework is first evaluated on a setting with synthetic noise. Then, we apply our proposed approach to orthographic image generation in aerial imagery. In addition, we discuss an exemplar-based inpainting technique for an integrated removal of non-stationary objects like cars.

Efficient object detection using orthogonal NMF descriptor hierarchies

Recently descriptors based on Histograms of Oriented Gradients (HOG) and Local Binary Patterns (LBP) have shown excellent results in object detection considering the precision as well as the recall. However, since these descriptors are based on high dimensional representations such approaches suffer from enormous memory and runtime requirements. The goal of this paper is to overcome these problems by introducing hierarchies of orthogonal Non-negative Matrix Factorizations (NMF). In fact, in this way a lower dimensional feature representation can be obtained without loosing the discriminative power of the original features. Moreover, the hierarchical structure allows to represent parts of patches on different scales allowing for a more robust classification. We show the effectiveness of our approach for two publicly available datasets and compare it to existing state-of-the-art methods. In addition, we demonstrate it in context of aerial imagery, where high dimensional images have to be processed requiring efficient methods.