Andrea Eichenseer

Hybrid super-resolution combining example-based single-image and interpolation-based multi-image reconstruction approaches

Achieving a higher spatial resolution is of particular interest in many applications such as video surveillance and can be realized by employing higher resolution sensors or applying super-resolution methods. Traditional super-resolution algorithms are based on either a single low resolution image or on multiple low resolution frames. In this paper, a hybrid super-resolution method is proposed which combines both a single-image and a multi-image approach using a soft decision mask. The mask is computed from the motion information utilized in the multi-image super-resolution part. This concept is shown to work for one particular setup but is also extensible toward other combinations of single-image and multi-image super-resolution algorithms as well as other merging metrics. Simulation results show an average luminance PSNR gain of up to 0.85 dB and 0.59 dB for upscaling factors of 2 and 4, respectively. Visual results substantiate the objective results.

Modeling the energy consumption of HEVC intra decoding

Battery life is one of the major limitations to mobile device use, which makes research on energy efficient soft- and hardware an important task. This paper investigates the energy required by a CPU when decoding compressed bitstream videos on mobile platforms. A model is derived that describes the energy consumption of the new HEVC decoder for intra coded videos. We show that the relative estimation error of the model is smaller than 3.2% and that the model can be used to build encoders aiming at minimizing decoding energy.

A hybrid motion estimation technique for fisheye video sequences based on equisolid re-projection

Capturing large fields of view with only one camera is an important aspect in surveillance and automotive applications, but the wide-angle fisheye imagery thus obtained exhibits very special characteristics that may not be very well suited for typical image and video processing methods such as motion estimation. This paper introduces a motion estimation method that adapts to the typical radial characteristics of fisheye video sequences by making use of an equisolid re-projection after moving part of the motion vector search into the perspective domain via a corresponding back-projection. By combining this approach with conventional translational motion estimation and compensation, average gains in luminance PSNR of up to 1.14 dB are achieved for synthetic fish-eye sequences and up to 0.96 dB for real-world data. Maximum gains for selected frame pairs amount to 2.40 dB and 1.39 dB for synthetic and real-world data, respectively.

Motion estimation for fisheye video sequences combining perspective projection with camera calibration information

Fisheye cameras prove a convenient means in surveillance and automotive applications as they provide a very wide field of view for capturing their surroundings. Contrary to typical rectilinear imagery, however, fisheye video sequences follow a different mapping from the world coordinates to the image plane which is not considered in standard video processing techniques. In this paper, we present a motion estimation method for real-world fisheye videos by combining perspective projection with knowledge about the underlying fisheye projection. The latter is obtained by camera calibration since actual lenses rarely follow exact models. Furthermore, we introduce a re-mapping for ultra-wide angles which would otherwise lead to wrong motion compensation results for the fisheye boundary. Both concepts extend an existing hybrid motion estimation method for equisolid fisheye video sequences that decides between traditional and fisheye block matching in a block-based manner. Compared to that method, the proposed calibration and re-mapping extensions yield gains of up to 0.58 dB in luminance PSNR for real-world fisheye video sequences. Overall gains amount to up to 3.32 dB compared to traditional block matching.

Multi-image super-resolution using a dual weighting scheme based on Voronoi tessellation

Increasing spatial resolution is often required in many applications such as entertainment systems or video surveillance. Apart from using higher resolution sensors, it is also possible to apply superresolution algorithms to realize an increased resolution. Those methods can be divided into approaches that rely on only a single low resolution image or on multiple low resolution video frames. While incorporating more frames into the super-resolution is beneficial for the resolution enhancement in principle, it is also likely to introduce more artifacts from inaccurate motion estimation. To alleviate this problem, various weightings have been proposed in the literature. In this paper, we propose an extended dual weighting scheme for an interpolation-based super-resolution method based on Voronoi tessellation that relies on both a motion confidence weight and a distance weight. Compared to non-weighted super-resolution, the proposed method yields an average gain in luminance PSNR of up to 1.29 dB and 0.61 dB for upscaling factors of 2 and 4, respectively. Visual comparisons substantiate the objective results.

Multi-image super-resolution for fisheye video sequences using subpixel motion estimation based on calibrated re-projection

Super-resolution techniques are a means for reconstructing a higher spatial resolution from low resolution content, which is especially important for automotive or surveillance systems. Furthermore, being able to capture a large area with a single camera can be realized by using ultra-wide angle lenses, as employed in so-called fisheye cameras. However, the underlying non-perspective projection function of fisheye cameras introduces significant radial distortion, which is not considered by conventional super-resolution techniques. In this paper, we therefore propose the integration of a fisheye-adapted motion estimation approach that is based on a calibrated re-projection into a multi-image super-resolution framework. The proposed method is capable of taking the fisheye characteristics into account, thus improving the reconstruction quality. Simulation results show an average gain in luminance PSNR of up to 0.3 dB for upscaling factors of 2 and 4. Visual examples substantiate the objective results.

Temporal error concealment for fisheye video sequences based on equisolid re-projection

Wide-angle video sequences obtained by fisheye cameras exhibit characteristics that may not very well comply with standard image and video processing techniques such as error concealment. This paper introduces a temporal error concealment technique designed for the inherent characteristics of equisolid fisheye video sequences by applying a re-projection into the equisolid domain after conducting part of the error concealment in the perspective domain. Combining this technique with conventional decoder motion vector estimation achieves average gains of 0.71 dB compared against pure decoder motion vector estimation for the test sequences used. Maximum gains amount to up to 2.04 dB for selected frames.

Joint shape and centroid adaptive frequency selective extrapolation for the reconstruction of arbitrarily shaped loss areas

Reconstructing missing areas of arbitrary shape and size is particularly important in error-prone communication as well as in applications where motion compensation is conducted such as multi-image super-resolution or framerate up-conversion. To that end, frequency selective extrapolation is an effective image reconstruction technique. This approach was originally designed for block losses and has recently been enhanced by a centroid adaptation to improve the reconstruction quality in the case of arbitrarily shaped loss areas. In this paper, we reuse the idea of centroid adaptation and introduce a novel shape adaptation so as to better assign higher weights to more relevant pixels. Moreover, we propose to combine both the shape adaptation and the centroid adaptation into a joint solution to further improve the reconstruction quality. To evaluate the proposed method, three different loss patterns are used. Simulation results yield an average gain in luminance PSNR of up to 0.2 dB for the high quality profile and 3.4 dB for the high efficiency profile, respectively. A visual comparison confirms these results.

Accelerated hybrid image reconstruction for non-regular sampling color sensors

Increasing the spatial resolution is an ongoing research topic in image processing. A recently presented approach applies a non-regular sampling mask on a low resolution sensor and subsequently reconstructs the masked area via an extrapolation algorithm to obtain a high resolution image. This paper introduces an acceleration of this approach for use with full color sensors. Instead of employing the effective, yet computationally expensive extrapolation algorithm on each of the three RGB channels, a color space conversion is performed and only the luminance channel is then reconstructed using this algorithm. As natural images contain much less information in the chrominance channels, a fast linear interpolation technique can here be used to accelerate the whole reconstruction procedure. Simulation results show that an average speed up factor of 2.9 is thus achieved, while the loss in visual quality stays imperceptible. Comparisons of PSNR results confirm this.

Coding of distortion-corrected fisheye video sequences using H.265/HEVC

Images and videos captured by fisheye cameras exhibit strong radial distortions due to their large field of view. Conventional intra-frame as well as inter-frame prediction techniques as employed in hybrid video coding schemes are not designed to cope with such distortions, however. So far, captured fish-eye data has been coded and stored without consideration to any loss in efficiency resulting from radial distortion. This paper investigates the effects on the coding efficiency when applying distortion correction as a pre-processing step as opposed to the state-of-the-art method of post-processing. Both methods make use of the latest video coding standard H.265/HEVC and are compared with regard to objective as well as subjective video quality. It is shown that a maximum PSNR gain of 1.91 dB for intra-frame and 1.37 dB for inter-frame coding is achieved when using the pre-processing method. Average gains amount to 1.16 dB and 0.95 dB for intra-frame and inter-frame coding, respectively.