Andreas Krutz

Windowed Image Registration for Robust Mosaicing of Scenes with Large Background Occlusions

We propose an enhanced window-based approach to local image registration for robust video mosaicing in scenes with arbitrarily moving foreground objects. Unlike other approaches, we estimate accurately the image transformation without any pre-segmentation even if large background regions are occluded. We apply a windowed hierarchical frame-to-frame registration based on image pyramid decomposition. In the lowest resolution level phase correlation for initial parameter estimation is used while in the next levels robust Newton-based energy minimization of the compensated image mean-squared error is conducted. To overcome the degradation error caused by spatial image interpolation due to the warping process, i.e. aliasing effects from under-sampling, final pixel values are assigned in an up-sampled image domain using a Daubechies bi-orthogonal synthesis filter. Experimental results show the excellent performance of the method compared to recently published methods. The image registration is sufficiently accurate to allow open-loop parameter accumulation for long-term motion estimation.

Evaluation of pixel- and motion vector-based global motion estimation for camera motion characterization

Pixel-based and motion vector-based global motion estimation (GME) techniques are evaluated in this paper with an automatic system for camera motion characterization. First, the GME techniques are compared with a frame-by-frame PNSR measurement using five video sequences. The best motion vector-based GME method is then evaluated together with a common and a simplified pixel-based GME technique for camera motion characterization. For this, selected unedited videos from the TRECVid 2005 BBC rushes corpus are used. We evaluate how the estimation accuracy of global motion parameters affects the results for camera motion characterization in terms of retrieval measures. The results for this characterization show that the simplified pixel-based GME technique obtains results that are comparable with the common pixel-based GME method, and outperforms significantly the results of an earlier proposed motion vector-based GME approach.

Video coding using global motion temporal filtering

Recent deblocking techniques are based on spatial filtering. We present a new deblocking technique based on temporal filtering of spatially aligned frames. This approach is used in an H.264/AVC coding environment. The algorithm estimates the ideal amount of frames used for temporal filtering at the encoder side. In that way it is assured that the receiver is presented with the best possible visual quality in terms of structural similarity. Theoretical consideration of the problem proves the concept of the new approach. Experimental evaluation shows that the new temporal deblocking filter significantly improves visual quality and reduces bit rate compared to common H.264/AVC deblocking by up to 18%.

Motion-based Object Segmentation using Sprites and Anisotropic Diffusion

Many algorithms have been developed to recognize regions, edges, color, and objects in images and videos. For applications like surveillance or object-based video coding, it is important to segment the foreground objects from the background. The task is very challenging in the case of a moving camera. We present a foreground segmentation approach that is designed for sprite coding as well as other applications, e.g. video surveillance. Accurate frame-to- frame image registration and sprite generation build the pre-processing step. The segmentation algorithm operates on error images, which are produced by the image registration and subtraction from reconstructed background frames. It is processed in several steps including low-pass filtering using anisotropic diffusion. Experiments show excellent results with single- and multi-view test sequences.

Optimal multiple sprite generation based on physical camera parameter estimation

We present a robust and computational low complex method to estimate the physical camera parameters, intrinsic and extrinsic, for scene shots captured by cameras applying pan, tilt, rotation, and zoom. These parameters are then used to split a sequence of frames into several subsequences in an optimal way to generate multiple sprites. Hereby, optimal means a minimal usage of memory while keeping or even improving the reconstruction quality of the scene background. Since wide angles between two frames of a scene shot cause geometrical distortions using a perspective mapping it is necessary to part the shot into several subsequences. In our approach it is not mandatory that all frames of a subsequence are adjacent frames in the original scene. Furthermore the angle-based classification allows frame reordering and makes our approach very powerful.

Global motion estimation using variable block sizes and its application to object segmentation

Global motion is estimated either in the pixel domain or in block based domain. Until now, all the approaches regarding the latter are based on fixed sized blocks while the recent compression methods tend to use variable block sizes during motion estimation. In this paper we present a new procedure for global motion estimation based on a variable block size motion vector field. A block matching algorithm which is able to adapt the block size according to the motion complexity within the frame is used. The resulting motion vectors are employed for global motion estimation. Furthermore, binary foreground-background masks are created based on the frame-by-frame motion compensated differences by exploiting spatial conditions through anisotropic diffusion filtering. For global motion estimation the performance evaluation in terms of background PSNR shows an enhancement of more than 2.5 dB in the well-known “Stefan” sequence, compared to the conventional case of fixed block size, at a reasonable implementation complexity.

Object-Based Multiple Sprite Coding of Unsegmented Videos using H.264/AVC

In spite of recent progress in the development of hybrid block-based video codecs, it has been shown that for low-bitrate scenarios there is still coding gain applying object-based techniques. We present a sprite-based codec, based on latest H.264 features using an inbuilt segmentation approach for scenes recorded by a rotating camera. The segmentation itself is built up on reliable background estimation from the sprite and short-term image registration. Moreover, we generate multiple sprites based on physical camera parameter estimation that overcome three of the main drawbacks of sprite coding techniques. First, the coding cost for the sprite image is minimized. Second, multiple sprites allow temporal background refresh and finally, registration error accumulation is kept very small. Experimental results show that this coding approach significantly outperforms latest H.264 extensions applying hierarchical B pictures.

Robust global motion estimation using motion vectors of variable size blocks and automatic motion model selection

A new approach for highly robust and precise global motion estimation (GME) using motion vectors (MVs) is presented. We show that this approach obtains precise higher-order short-term motion parameters for global motion using motion vectors solely. The approach is general and works for different mathematical methods including least-squares and Newton-Raphson method. We show that the approach is suitable for fixed block sizes from plain full-search block-matching as well as for arbitrary block sizes from video streams compressed with H.264/AVC reference encoder. The proposed approach is compared against four other known MV-based GME (MVGME) methods. Our results show that the approach is significantly more robust and obtains higher precision for global motion parameters in terms of background motion compensation, especially if moving objects occur. In addition, the results are as good as results from precise pixel-based GME methods or even better while the presented MV-GME methods have very low computational costs.

Extending H.264/AVC with a background sprite prediction mode

The latest standardized hybrid video codec, H.264/AVC, significantly outperforms earlier video coding standards. Despite combining improved and new algorithms within this codec, it is still possible to find methods which lead to a higher coding efficiency. We tackle the prediction problem adding a new prediction mode to the codec. It has been shown that the generation of a background sprite image containing all the background information of a certain sequence is very useful e.g. for object-based video coding. We use a pre-generated background sprite image for creating a new prediction mode in the encoder loop. For the current frame to be compensated, blocks reconstructed from the background sprite are used beside the remaining modes to calculate the residual. The rate-distortion optimization decides which mode is taken. Experimental results show the improvement using the new sprite prediction (SP) mode with the considered test sequences.

A block-adaptive skip mode for inter prediction based on parametric motion models

Motion compensated prediction (MCP) in hybrid video coding estimates a translational motion vector for a given block which is then used for residual computation. However, when complex motion like zoom, rotation, and perspective transformation occur, the translational model assumption does not always hold. This may result in higher residual energy and splitting of blocks, respectively. This paper proposes a skip mode based on higher-order parametric motion models. Often, these models provide a better prediction quality resulting in lower residual energy and larger block sizes. The proposed technique estimates a higher-order motion model between two given pictures. The encoder decides in terms of rate-distortion optimization whether to use the new skip mode for a block and therefore not to transfer any additional information like coefficient data. Experimental evaluation shows that the proposed technique can improve the coding performance of next generation video coding standards significantly.