Markus Jonscher

Hybrid super-resolution combining example-based single-image and interpolation-based multi-image reconstruction approaches

Achieving a higher spatial resolution is of particular interest in many applications such as video surveillance and can be realized by employing higher resolution sensors or applying super-resolution methods. Traditional super-resolution algorithms are based on either a single low resolution image or on multiple low resolution frames. In this paper, a hybrid super-resolution method is proposed which combines both a single-image and a multi-image approach using a soft decision mask. The mask is computed from the motion information utilized in the multi-image super-resolution part. This concept is shown to work for one particular setup but is also extensible toward other combinations of single-image and multi-image super-resolution algorithms as well as other merging metrics. Simulation results show an average luminance PSNR gain of up to 0.85 dB and 0.59 dB for upscaling factors of 2 and 4, respectively. Visual results substantiate the objective results.

Resampling Images to a Regular Grid From a Non-Regular Subset of Pixel Positions Using Frequency Selective Reconstruction

Even though image signals are typically defined on a regular 2D grid, there also exist many scenarios where this is not the case and the amplitude of the image signal only is available for a non-regular subset of pixel positions. In such a case, a resampling of the image to a regular grid has to be carried out. This is necessary since almost all algorithms and technologies for processing, transmitting or displaying image signals rely on the samples being available on a regular grid. Thus, it is of great importance to reconstruct the image on this regular grid, so that the reconstruction comes closest to the case that the signal has been originally acquired on the regular grid. In this paper, Frequency Selective Reconstruction is introduced for solving this challenging task. This algorithm reconstructs image signals by exploiting the property that small areas of images can be represented sparsely in the Fourier domain. By further considering the basic properties of the optical transfer function of imaging systems, a sparse model of the signal is iteratively generated. In doing so, the proposed algorithm is able to achieve a very high reconstruction quality, in terms of peak signal-to-noise ratio (PSNR) and structural similarity measure as well as in terms of visual quality. The simulation results show that the proposed algorithm is able to outperform state-of-the-art reconstruction algorithms and gains of more than 1 dB PSNR are possible.

Centroid adapted frequency selective extrapolation for reconstruction of lost image areas

Lost image areas with different size and arbitrary shape can occur in many scenarios such as error-prone communication, depth-based image rendering or motion compensated wavelet lifting. The goal of image reconstruction is to restore these lost image areas as close to the original as possible. Frequency selective extrapolation is a block-based method for efficiently reconstructing lost areas in images. So far, the actual shape of the lost area is not considered directly. We propose a centroid adaption to enhance the existing frequency selective extrapolation algorithm that takes the shape of lost areas into account. To enlarge the test set for evaluation we further propose a method to generate arbitrarily shaped lost areas. On our large test set, we obtain an average reconstruction gain of 1.29 dB.

Texture-dependent frequency selective reconstruction of non-regularly sampled images

There exist many scenarios where pixel information is available only on a non-regular subset of pixel positions. For further processing, however, it is required to reconstruct such images on a regular grid. Besides many other algorithms, frequency selective reconstruction can be applied for this task. It performs a block-wise generation of a sparse signal model as an iterative superposition of Fourier basis functions and uses this model to replace missing or corrupted pixels in an image. In this paper, it is shown that it is not required to spend the same amount of iterations on both homogeneous and heterogeneous regions. Hence, a new texture-dependent approach for frequency selective reconstruction is introduced that distributes the number of iterations depending on the texture of the regions to be reconstructed. Compared to the original frequency selective reconstruction and depending on the number of iterations, visually noticeable gains in PSNR of up to 1.47 dB can be achieved.

Reconstruction of images taken by a pair of non-regular sampling sensors using correlation based matching

Multi-view image acquisition systems with two or more cameras can be rather costly due to the number of high resolution image sensors that are required. Recently, it has been shown that by covering a low resolution sensor with a non-regular sampling mask and by using an efficient algorithm for image reconstruction, a high resolution image can be obtained. In this paper, a stereo image reconstruction setup for multi-view scenarios is proposed. A scene is captured by a pair of non-regular sampling sensors and by incorporating information from the adjacent view, the reconstruction quality can be increased. Compared to a state-of-the-art single-view reconstruction algorithm, this leads to a visually noticeable average gain in PSNR of 0.74 dB.

Recursive frequency selective reconstruction of non-regularly sampled video data

High resolution images can be acquired using a non-regular sampling sensor which consists of an underlying low resolution sensor that is covered with a non-regular sampling mask. The reconstructed high resolution image is then obtained during post-processing. Recently, it has been shown that the temporal correlation between neighboring frames can be exploited in order to enhance the reconstruction quality of non-regularly sampled video data. In this paper, a new recursive multi-frame reconstruction approach is proposed in order to further increase the reconstruction quality. By using a new reference order, previously reconstructed frames can be used for the subsequent motion estimation and a new weighting function allows for the incorporation of multiple pixels projected onto the same position. With the new recursive multi-frame approach, a visually noticeable average gain in PSNR of up to 1.13 dB with respect to a state-of-the-art single-frame reconstruction approach can be achieved. Compared to the existing multi-frame approach, a gain of 0.31 dB is possible. SSIM results show the same behavior as PSNR results. Additionally, the pre-reconstruction step of the existing multi-frame approach can be avoided and the new algorithm is, in general, capable of real-time processing.

Reconstruction of videos taken by a non-regular sampling sensor

Recently, it has been shown that a high resolution image can be obtained without the usage of a high resolution sensor. The main idea has been that a low resolution sensor is covered with a non-regular sampling mask followed by a reconstruction of the incomplete high resolution image captured this way. In this paper, a multi-frame reconstruction approach is proposed where a video is taken by a non-regular sampling sensor and fully reconstructed afterwards. By utilizing the temporal correlation between neighboring frames, the reconstruction quality can be further enhanced. Compared to a state-of-the-art single-frame reconstruction approach, this leads to a visually noticeable gain in PSNR of up to 1.19 dB on average.

Accelerated hybrid image reconstruction for non-regular sampling color sensors

Increasing the spatial resolution is an ongoing research topic in image processing. A recently presented approach applies a non-regular sampling mask on a low resolution sensor and subsequently reconstructs the masked area via an extrapolation algorithm to obtain a high resolution image. This paper introduces an acceleration of this approach for use with full color sensors. Instead of employing the effective, yet computationally expensive extrapolation algorithm on each of the three RGB channels, a color space conversion is performed and only the luminance channel is then reconstructed using this algorithm. As natural images contain much less information in the chrominance channels, a fast linear interpolation technique can here be used to accelerate the whole reconstruction procedure. Simulation results show that an average speed up factor of 2.9 is thus achieved, while the loss in visual quality stays imperceptible. Comparisons of PSNR results confirm this.

Reconstruction of multiview images taken with non-regular sampling sensors

Increasing spatial image resolution is a widely discussed area in the field of image processing. In this paper, we present an efficient reconstruction approach for high-resolution images, taken with irregularly shielded low-resolution sensors in a multiview setup. The approach is based on the sparsity assumption, meaning that natural images can be efficiently represented in a transform-domain using only few coefficients. Utilizing information from adjacent cameras results in a better reconstruction quality for the central high-resolution view. Since neighboring camera perspectives might differ in illumination, the information from adjacent views has to be adapted to the view to be reconstructed. The simulation results show that a proper incorporation of information from neighboring views leads to a PSNR gain of up to 2.20 dB compared to a state-of-the-art singleview reconstruction approach.

Reducing randomness of non-regular sampling masks for image reconstruction

Increasing spatial image resolution is an often required, yet challenging task in image acquisition. Recently, it has been shown that it is possible to obtain a high resolution image by covering a low resolution sensor with a non-regular sampling mask. Due to the masking, however, some pixel information in the resulting high resolution image is not available and has to be reconstructed by an efficient image reconstruction algorithm in order to get a fully reconstructed high resolution image. In this paper, the influence of different sampling masks with a reduced randomness of the non-regularity on the image reconstruction process is evaluated. Simulation results show that it is sufficient to use sampling masks that are non-regular only on a smaller scale. These sampling masks lead to a visually noticeable gain in PSNR compared to arbitrary chosen sampling masks which are non-regular over the whole image sensor size. At the same time, they simplify the manufacturing process and allow for efficient storage.