Matthias Kunter

Windowed Image Registration for Robust Mosaicing of Scenes with Large Background Occlusions

We propose an enhanced window-based approach to local image registration for robust video mosaicing in scenes with arbitrarily moving foreground objects. Unlike other approaches, we estimate accurately the image transformation without any pre-segmentation even if large background regions are occluded. We apply a windowed hierarchical frame-to-frame registration based on image pyramid decomposition. In the lowest resolution level phase correlation for initial parameter estimation is used while in the next levels robust Newton-based energy minimization of the compensated image mean-squared error is conducted. To overcome the degradation error caused by spatial image interpolation due to the warping process, i.e. aliasing effects from under-sampling, final pixel values are assigned in an up-sampled image domain using a Daubechies bi-orthogonal synthesis filter. Experimental results show the excellent performance of the method compared to recently published methods. The image registration is sufficiently accurate to allow open-loop parameter accumulation for long-term motion estimation.

Stereoscopic 3D from 2D video with super-resolution capability

This paper presents a new approach for the generation of super-resolution stereoscopic and multi-view video from monocular video. Such multi-view video is used, for instance, with multi-user 3D displays or auto-stereoscopic displays with head-tracking to create a depth impression of the observed scenery. Our approach is an extension of the realistic stereo-view synthesis (RSVS) approach, which is based on structure from motion techniques and image-based rendering to generate the desired stereoscopic views for each point in time. Subjective quality measurements with 25 real and 3 synthetic sequences were carried out to test the performance of RSVS against simple time-shift and depth-image-based rendering (DIBR). Our approach heavily enhances the stereoscopic depth perception and gives a more realistic impression of the observed scenery. Simulation results applying super-resolution show that the image quality can further be improved by reducing motion blur and compression artifacts.

Motion-based Object Segmentation using Sprites and Anisotropic Diffusion

Many algorithms have been developed to recognize regions, edges, color, and objects in images and videos. For applications like surveillance or object-based video coding, it is important to segment the foreground objects from the background. The task is very challenging in the case of a moving camera. We present a foreground segmentation approach that is designed for sprite coding as well as other applications, e.g. video surveillance. Accurate frame-to- frame image registration and sprite generation build the pre-processing step. The segmentation algorithm operates on error images, which are produced by the image registration and subtraction from reconstructed background frames. It is processed in several steps including low-pass filtering using anisotropic diffusion. Experiments show excellent results with single- and multi-view test sequences.

Robust concealment for erroneous block bursts in stereoscopic images

With the increasing number of image communication applications especially in the low complexity domain, error concealment has become a very important field of research. Since many compression standards for images and videos are block-based a lot of methods were applied to conceal block losses in monocular images. The fast progress of capture, representation and display technologies for 3D image data advances the efforts on 3D concealment strategies. Because of their psycho-visual characteristics, stereoscopic images have to fulfill a very high quality demand. We propose an algorithm that makes use of the redundancies between two views of a stereo image pair. In many cases erroneous block bursts occur and can be highly disturbing, thus we mainly concentrate on these errors. In addition, we focused on the quality assessment of several error concealment strategies. Beside the objective evaluation measures, we carried out a subjective quality test following the DSCQS methodology as proposed by MPEG. The results of this test demonstrate the efficiency of the approach.

Optimal multiple sprite generation based on physical camera parameter estimation

We present a robust and computational low complex method to estimate the physical camera parameters, intrinsic and extrinsic, for scene shots captured by cameras applying pan, tilt, rotation, and zoom. These parameters are then used to split a sequence of frames into several subsequences in an optimal way to generate multiple sprites. Hereby, optimal means a minimal usage of memory while keeping or even improving the reconstruction quality of the scene background. Since wide angles between two frames of a scene shot cause geometrical distortions using a perspective mapping it is necessary to part the shot into several subsequences. In our approach it is not mandatory that all frames of a subsequence are adjacent frames in the original scene. Furthermore the angle-based classification allows frame reordering and makes our approach very powerful.

Object-Based Multiple Sprite Coding of Unsegmented Videos using H.264/AVC

In spite of recent progress in the development of hybrid block-based video codecs, it has been shown that for low-bitrate scenarios there is still coding gain applying object-based techniques. We present a sprite-based codec, based on latest H.264 features using an inbuilt segmentation approach for scenes recorded by a rotating camera. The segmentation itself is built up on reliable background estimation from the sprite and short-term image registration. Moreover, we generate multiple sprites based on physical camera parameter estimation that overcome three of the main drawbacks of sprite coding techniques. First, the coding cost for the sprite image is minimized. Second, multiple sprites allow temporal background refresh and finally, registration error accumulation is kept very small. Experimental results show that this coding approach significantly outperforms latest H.264 extensions applying hierarchical B pictures.

Extending H.264/AVC with a background sprite prediction mode

The latest standardized hybrid video codec, H.264/AVC, significantly outperforms earlier video coding standards. Despite combining improved and new algorithms within this codec, it is still possible to find methods which lead to a higher coding efficiency. We tackle the prediction problem adding a new prediction mode to the codec. It has been shown that the generation of a background sprite image containing all the background information of a certain sequence is very useful e.g. for object-based video coding. We use a pre-generated background sprite image for creating a new prediction mode in the encoder loop. For the current frame to be compensated, blocks reconstructed from the background sprite are used beside the remaining modes to calculate the residual. The rate-distortion optimization decides which mode is taken. Experimental results show the improvement using the new sprite prediction (SP) mode with the considered test sequences.

Unsupervised object segmentation for 2D to 3D conversion

In this paper, we address the handling of independently moving objects (IMOs) in automatic 2D to stereoscopic 3D conversion systems based on structure-from-motion (SfM) techniques. Exploiting the different viewing positions of a moving camera, these techniques yield excellent 3D results for static scene objects. However, the independent motion of any foreground object requires a separate conversion process. We propose a novel segmentation approach that estimates the occluded static background and segments the IMOs based on advanced change detection. The background estimation is achieved applying 2D registration and blending techniques, representing an approximation of the underlying scene geometry. The segmentation process itself uses anisotropic filtering applied on the difference image between original frame and the estimated background frame. In order to render the segmented objects into the automatically generated 3D scene properly, a small amount of user interaction will be necessary, e.g. an assignment of intra-object depth or the objects absolute z-position. Experiments show that the segmentation method achieves accurate mask results for a variety of scenes, similar to the masks obtained manually using state-of-the-art rotoscoping tools. Though, this work contributes to the extension of SfM-based automatic 3D conversion methods for the application on dynamic scenes.

Super-resolution Mosaicing using Embedded Hybrid Recursive Folow-based Segmentation

We present a new strategy for the generation of background super-resolution mosaics from videos with arbitrary camera pan, tilt, and zoom including freely moving foreground. Our main focus is directed to the automatic, embedded pre-segmentation of foreground objects. The segmentation technique is based on efficient and robust computation of the optical flow between neighboring frames in a video scene using a hybrid recursive approach, i.e. a combination of block-flow methods and spatial-temporal anisotropic diffusion-based flow field regularization. Unlike in other related publications we are able to segment moving foreground objects before the actual image-to-mosaic-registration is proceeded even if the foreground objects do not move relatively to the camera motion. Thus, every segmented background frame can be used to enhance the resolution of the composed mosaic due to an effective blending process. Additionally, the appearance of disturbing ghost objects is prevented

A gradient based approach for stereoscopic error concealment

Error concealment is an important field of research in image processing. Many methods have been applied to conceal block losses in monocular images. We present a concealment strategy for block loss in stereoscopic image pairs. Unlike the error concealment techniques used for monocular images, the information of the associated image is utilized, i.e., by means of a projective transformation model, pixel values from the associated stereo image are warped to their corresponding positions in the lost block. The stereoscopic depth perception is much less affected in our approach than using monoscopic error concealment techniques.