Marco Körner

Temporal video segmentation by event detection: A novelty detection approach

Temporal segmentation of videos into meaningful image sequences containing some particular activities is an interesting problem in computer vision. We present a novel algorithm to achieve this semantic video segmentation. The segmentation task is accomplished through event detection in a frame-by-frame processing setup. We propose using one-class classification (OCC) techniques to detect events that indicate a new segment, since they have been proved to be successful in object classification and they allow for unsupervised event detection in a natural way. Various OCC schemes have been tested and compared, and additionally, an approach based on the temporal self-similarity maps (TSSMs) is also presented. The testing was done on a challenging publicly available thermal video dataset. The results are promising and show the suitability of our approaches for the task of temporal video segmentation.

Accurate 3D Multi-marker Tracking in X-ray Cardiac Sequences Using a Two-Stage Graph Modeling Approach

The in-depth analysis of heart movements under varying conditions is an important problem of cardiac surgery. To reveal the movement of relevant muscular parts, biplanar X-ray recordings of implanted radio-opaque markers are acquired. As manually locating these markers in the images is a very time-consuming task, our goal is to automate this process. Taking into account the difficulties in the recorded data such as missing detections or 2D occlusions, we propose a two-stage graph-based approach for both 3D tracklet and 3D track generation. In the first stage of our approach, we construct a directed acyclic graph of 3D observations to obtain tracklets via shortest path optimization. Afterwards, full tracks are extracted from a tracklet graph in a similar manner. This results in a globally optimal linking of detections and tracklets, while providing a flexible framework which can easily be adapted to various tracking scenarios based on the edge cost functions. We validate our approach on an X-ray sequence of a beating sheep heart based on manually labeled ground-truth marker positions. The results show that the performance of our method is comparable to human experts, while standard 3D tracking approaches such as particle filters are outperformed.

Building instance classification using street view images

Land-use classification based on spaceborne or aerial remote sensing images has been extensively studied over the past decades. Such classification is usually a patch-wise or pixel-wise labeling over the whole image. But for many applications, such as urban population density mapping or urban utility planning, a classification map based on individual buildings is much more informative. However, such semantic classification still poses some fundamental challenges, for example, how to retrieve fine boundaries of individual buildings. In this paper, we proposed a general framework for classifying the functionality of individual buildings. The proposed method is based on Convolutional Neural Networks (CNNs) which classify facade structures from street view images, such as Google StreetView, in addition to remote sensing images which usually only show roof structures. Geographic information was utilized to mask out individual buildings, and to associate the corresponding street view images. We created a benchmark dataset which was used for training and evaluating CNNs. In addition, the method was applied to generate building classification maps on both region and city scales of several cities in Canada and the US.

Automatic Alignment of Indoor and Outdoor Building Models Using 3D Line Segments

This paper presents an approach for automatically aligning the non-overlapping interior and exterior parts of a 3D building model computed from image based 3D reconstructions. We propose a method to align the 3D reconstructions by identifying corresponding 3D structures that are part of the interior and exterior model (e.g. openings like windows). In this context, we point out the potential of using 3D line segments to enrich the information of point clouds generated by SfMs and show how this can be used for interpreting the scene and matching individual reconstructions.

Temporal Self-Similarity for Appearance-Based Action Recognition in Multi-View Setups

We present a general data-driven method for multi-view action recognition relying on the appearance of dynamic systems captured from different viewpoints. Thus, we do not depend on 3d reconstruction, foreground segmentation, or accurate detections. We extend further earlier approaches based on Temporal Self-Similarity Maps by new low-level image features and similarity measures. Gaussian Process classification in combination with Histogram Intersection Kernels serve as powerful tools in our approach. Experiments performed on our new combined multi-view dataset as well as on the widely used IXMAS dataset show promising and competing results.

Analyzing the Subspaces Obtained by Dimensionality Reduction for Human Action Recognition from 3d Data

Since depth measuring devices for real-world scenarios became available in the recent past, the use of 3d data now comes more in focus of human action recognition. Due to the increased amount of data it seems to be advisable to model the trajectory of every landmark in the context of all other landmarks which is commonly done by dimensionality reduction techniques like PCA. In this paper we present an approach to directly use the subspaces (i.e. their basis vectors) for extracting features and classification of actions instead of projecting the landmark data themselves. This yields a fixed-length description of action sequences disregarding the number of provided frames. We give a comparison of various global techniques for dimensionality reduction and analyze their suitability for our proposed scheme. Experiments performed on the CMU Motion Capture dataset show promising recognition rates as well as robustness in the presence of noise and incorrect detection of landmarks.

The TUM-DLR Multimodal Earth Observation Evaluation Benchmark

We present a new dataset for development, benchmarking, and evaluation of remote sensing and earth observation approaches with special focus on converging perspectives. In order to provide data with different modalities, we observed the same scene using satellites, airplanes, unmanned aerial vehicles (UAV), and smartphones. The dataset is further complemented by ground-truth information and baseline results for different application scenarios. The provided data can be freely used by anybody interested in remote sensing and earth observation and will be continuously augmented and updated.

Robust Object-Based Multipass InSAR Deformation Reconstruction

Deformation monitoring by multipass synthetic aperture radar (SAR) interferometry (InSAR) is, so far, the only imaging-based method to assess millimeter-level deformation over large areas from space. Past research mostly focused on the optimal retrieval of deformation parameters on the basis of a single pixel or a pixel cluster. Only until recently, the first demonstration of object-based urban infrastructure monitoring by fusing InSAR and the semantic classification labels derived from optical images was presented by Wang et al. Given such classification labels in the SAR image, we propose a general framework for object-based InSAR parameter retrieval, where the parameters of the whole object are jointly estimated by the inversion of a regularized tensor model instead of pixelwise. Our approach does not assume the stationarity of each sample in the object, which is usually assumed in other pixel cluster-based methods, such as SqueeSAR. In addition, to handle outliers in real data, a robust phase recovery step prior to parameter retrieval is also introduced. In typical settings, the proposed method outperforms the current pixelwise estimators, e.g., periodogram, by a factor of several tens in the accuracy of the linear deformation estimates. Last but not least, for a practical demonstration on bridge monitoring, we present a full workflow of long-term bridge monitoring using the proposed approach.

Object-based InSAR deformation reconstruction with application to bridge monitoring

Deformation monitoring by multi-baseline synthetic aperture radar (SAR) interferometry is so far the only imaging-based method to assess millimeter-level deformation over large areas from space. Past research mostly focused on optimal deformation parameters retrieval on a pixel-basis. Only until recently, the first demonstration of object-based urban infrastructures monitoring by fusing SAR interferometry (InSAR) and the semantic classification labels derived from optical images was presented in [1]-[3]. This paper proposes an algorithm for object-based joint InSAR deformation reconstruction using these classification labels. We derive an object-based multi-baseline InSAR reconstruction model, and propose an efficient algorithm for bridge detection in optical images.

JAR-Aibo: A Multi-view Dataset for Evaluation of Model-Free Action Recognition Systems

We present a novel multi-view dataset for evaluating model-free action recognition systems. Superior to existing datasets, it covers 56 distinct action classes. Each of them was performed ten times by remotely controlled Sony ERS-7 AIBO robot dogs observed by six distributed and synchronized cameras at 17 fps and VGA resolution. In total, our dataset contains 576 sequences. Baseline results show its applicability for benchmarking model-free action recognition methods.