Wolfgang Schnurrer

Robust super-resolution in a multiview setup based on refined high-frequency synthesis

Increasing image sharpness and thus improving the visual quality is an important task in multiview image and video processing. We propose a novel super-resolution approach for multiview images in a mixed-resolution setup that is robust to various depth map distortions. The considered distortion scenarios may be caused by an inaccurate calibration of the depth camera or a limitation of depth range. Our method is based on a refined high-frequency synthesis that relies on a blockwise and depth-dependant low-frequency registration. The refinement step efficiently adapts the high-frequency content from a neighboring high-resolution camera to a low-resolution view and thereby compensates the displacement caused by depth inaccuracies. In case of undistorted depth maps, the results show that our algorithm leads to a PSNR gain of up to 1.33 dB with respect to a comparable unrefined super-resolution approach for a mixed-resolution multiview video plus depth format. Compared to the initial low-resolution view, a PSNR gain of up to 2.61 dB is obtained. In case of distorted depth maps, a PSNR gain of even 4.78 dB is achieved with respect to the reference superresolution algorithm. The PSNR gains get confirmed by the corresponding SSIM values which manifest a similar behaviour. The improvement of visual quality is also convincingly for all considered scenarios.

Robust Super-Resolution for Mixed-Resolution Multiview Image Plus Depth Data

Increasing spatial resolution and thus improving the image quality is a key issue in the mixed-resolution multiview image and video processing domain. Given adjacent camera perspectives with various spatial resolutions and their corresponding depth information, the high-frequency part of a high-resolution view can be used for increasing the image quality of a neighboring low-resolution camera perspective. However, a reasonable projection of high-frequency information onto the image plane of a neighboring low-resolution view typically requires pixel-wise error-free depth data for the high-resolution reference image. Starting from this, a novel image super-resolution approach is proposed that is robust against both inaccurate depth acquisition and nonperfect calibration of spatially low-resolution depth sensors. The algorithm is based on displacement-compensated high-frequency synthesis and aims at correcting the projection errors introduced by inaccurate depth information. The proposed approach is further effectively extended by a signal extrapolation technique. For a wide range of proper scenarios, the proposed framework achieves substantial objective and visual gains compared with the considered reference approaches. The improvement of quality is shown for both simulated and self-recorded experimental data.

Dimensioning of optical birefringent anti-alias filters for digital cameras

A digital camera samples the continuous real world. As with any sampling process, questions of aliasing for certain sampling frequencies and the prevention thereof arise. In this paper we will discuss the spatial domain sampling and prevention of aliasing in digital cameras. We focus on the widely used birefringent anti alias filters that are often called optical low pass filters (OLPF). We show 2D models for all contributions to spatial domain sampling and derive optimum filter parameters for minimum aliasing and best possible image sharpness. Compared to previously used selection rules, we can show that the optimum selection of filter parameters can easily deliver more sharpness and reduce aliasing by a factor of 2. The simulated results are finally confirmed in real world experiments.

Analysis of displacement compensation methods for wavelet lifting of medical 3-D thorax CT volume data

A huge advantage of the wavelet transform in image and video compression is its scalability. Wavelet-based coding of medical computed tomography (CT) data becomes more and more popular. While much effort has been spent on encoding of the wavelet coefficients, the extension of the transform by a compensation method as in video coding has not gained much attention so far. We will analyze two compensation methods for medical CT data and compare the characteristics of the displacement compensated wavelet transform with video data. We will show that for thorax CT data the transform coding gain can be improved by a factor of 2 and the quality of the lowpass band can be improved by 8 dB in terms of PSNR compared to the original transform without compensation.

Analysis of mesh-based motion compensation in wavelet lifting of dynamical 3-D+t CT data

Factorized in the lifting structure, the wavelet transform can easily be extended by arbitrary compensation methods. Thereby, the transform can be adapted to displacements in the signal without losing the ability of perfect reconstruction. This leads to an improvement of scalability. In temporal direction of dynamic medical 3-D+t volumes from Computed Tomography, displacement is mainly given by expansion and compression of tissue. We show that these smooth movements can be well compensated with a mesh-based method. We compare the properties of triangle and quadrilateral meshes. We also show that with a mesh-based compensation approach coding results are comparable to the common slice wise coding with JPEG 2000 while a scalable representation in temporal direction can be achieved.

Centroid adapted frequency selective extrapolation for reconstruction of lost image areas

Lost image areas with different size and arbitrary shape can occur in many scenarios such as error-prone communication, depth-based image rendering or motion compensated wavelet lifting. The goal of image reconstruction is to restore these lost image areas as close to the original as possible. Frequency selective extrapolation is a block-based method for efficiently reconstructing lost areas in images. So far, the actual shape of the lost area is not considered directly. We propose a centroid adaption to enhance the existing frequency selective extrapolation algorithm that takes the shape of lost areas into account. To enlarge the test set for evaluation we further propose a method to generate arbitrarily shaped lost areas. On our large test set, we obtain an average reconstruction gain of 1.29 dB.

Open source HEVC analyzer for rapid prototyping (HARP)

The design of new HEVC extensions comes with the need for careful analysis of internal HEVC codec decisions. Several bitstream analyzers have evolved for this purpose and provide a visualization of encoder decisions as seen from a decoder viewpoint. None of the existing solutions is able to provide actual insight into the encoder and its RDO decision process. With one exception, all solutions are closed source and make adaption of their code to specific implementation needs impossible. Overall, development with the HM code base remains a time-consuming task. Here, we present the HEVC Analyzer for Rapid Prototyping (HARP), which directly addresses the above issues and is freely available under www.lms.lnt.de/HARP.

Improving block-based compensated wavelet lifting by reconstructing unconnected pixels

This paper presents a new approach for improving the visual quality of the lowpass band of a compensated wavelet transform. A high quality of the lowpass band is very important as it can then be used as a downscaled version of the original signal. To adapt the transform to the signal, compensation methods can be implemented directly into the transform. We propose an improved inversion of the block-based motion compensation by processing unconnected pixels by a reconstruction method. We obtain a better subjective visual quality while furthermore saving up to 2.6% of bits for lossless coding.

3-D mesh compensated wavelet lifting for 3-D+t medical CT data

For scalable coding, a high quality of the lowpass band of a wavelet transform is crucial when it is used as a downscaled version of the original signal. However, blur and motion can lead to disturbing artifacts. By incorporating feasible compensation methods directly into the wavelet transform, the quality of the lowpass band can be improved. The displacement in dynamic medical 3-D+t volumes from Computed Tomography is mainly given by expansion and compression of tissue over time and can be modeled well by mesh-based methods. We extend a 2-D mesh-based compensation method to three dimensions to obtain a volume compensation method that can additionally compensate deforming displacements in the third dimension. We show that a 3-D mesh can obtain a higher quality of the lowpass band by 0.28 dB with less than 40% of the model parameters of a comparable 2-D mesh. Results from lossless coding with JPEG 2000 3D and SPECK3D show that the compensated subbands using a 3-D mesh need about 6% less data compared to using a 2-D mesh.

Joint shape and centroid adaptive frequency selective extrapolation for the reconstruction of arbitrarily shaped loss areas

Reconstructing missing areas of arbitrary shape and size is particularly important in error-prone communication as well as in applications where motion compensation is conducted such as multi-image super-resolution or framerate up-conversion. To that end, frequency selective extrapolation is an effective image reconstruction technique. This approach was originally designed for block losses and has recently been enhanced by a centroid adaptation to improve the reconstruction quality in the case of arbitrarily shaped loss areas. In this paper, we reuse the idea of centroid adaptation and introduce a novel shape adaptation so as to better assign higher weights to more relevant pixels. Moreover, we propose to combine both the shape adaptation and the centroid adaptation into a joint solution to further improve the reconstruction quality. To evaluate the proposed method, three different loss patterns are used. Simulation results yield an average gain in luminance PSNR of up to 0.2 dB for the high quality profile and 3.4 dB for the high efficiency profile, respectively. A visual comparison confirms these results.