Axel Kochale

Patch-based spatio-temporal super-resolution for video with non-rigid motion

This paper presents a novel approach for spatio-temporal video super-resolution. Whereas the task of synthesizing high-frequency information on the spatial domain can be accomplished without introducing arbitrary priors on the image model (beyond the assumption of local cross-scale self-similarity), the high degree of temporal aliasing in standard-frame-rate video requires applying motion compensation in order to correctly interpolate the video sequence along the temporal axis. As the experimental results show, the proposed technique is capable of augmenting both the frame-rate of a video sequence (by any real up-converting factor) and its effective spatial resolution (by any rational magnification factor). The presented method is suitable for parallelized computing environments, such as GPUs, and provides an output quality comparable to state-of-the-art methods in both temporal interpolation and spatial super-resolution.

Towards a format-agnostic approach for production, delivery and rendering of immersive media

The media industry is currently being pulled in the often-opposing directions of increased realism (high resolution, stereoscopic, large screen) and personalization (selection and control of content, availability on many devices). We investigate the feasibility of an end-to-end format-agnostic approach to support both these trends. In this paper, different aspects of a format-agnostic capture, production, delivery and rendering system are discussed. At the capture stage, the concept of layered scene representation is introduced, including panoramic video and 3D audio capture. At the analysis stage, a virtual director component is discussed that allows for automatic execution of cinematographic principles, using feature tracking and saliency detection. At the delivery stage, resolution-independent audiovisual transport mechanisms for both managed and unmanaged networks are treated. In the rendering stage, a rendering process that includes the manipulation of audiovisual content to match the connected display and loudspeaker properties is introduced. Different parts of the complete system are revisited demonstrating the requirements and the potential of this advanced concept.

Fast single-image super-resolution with filter selection

This paper presents a new method for estimating a super-resolved version of an observed image by exploiting cross-scale self-similarity. We extend prior work on single-image super-resolution by introducing an adaptive selection of the best fitting upscaling and analysis filters for example learning. This selection is based on local error measurements obtained by using each filter with every image patch, and contrasts with the common approach of a constant metric in both dictionary-based and internal learning super-resolution. The proposed method is suitable for interactive applications, offering low computational load and a parallelizable design that allows straight-forward GPU implementations. Experimental results also show how our method generalizes better to different datasets than dictionary-based super-resolution and comparably to internal learning with adaptive post-processing.

Robust single-image super-resolution using cross-scale self-similarity

We present a noise-aware single-image super-resolution (SI-SR) algorithm, which automatically cancels additive noise while adding detail learned from lower-resolution scales. In contrast with most SI-SR techniques, we do not assume the input image to be a clean source of examples. Instead, we adapt the recent and efficient in-place cross-scale self-similarity prior for both learning fine detail examples and reducing image noise. Our experiments show a promising performance, despite the relatively simple algorithm. Both objective evaluations and subjective validations show clear quality improvements when upscaling noisy images.

A Bayesian approach for natural image denoising

This article presents a new method for estimating the latent noiseless version of an observed image corrupted by additive noise. This method stems from classical models in parametric denoising and extends them by modeling the likelihood term, estimating adaptive image priors and automatically choosing an adaptive equivalent to the typically hand-tuned regularization constant. The proposed method introduces a possible path to overcome the limitations of current parametric denoising algorithms and provides a competitive alternative to powerful non-parametric ones. The experimental results show how our method adapts better to different noise types than state-of-the-art parametric and non-parametric algorithms.

Robust super-resolution for interactive video navigation

One of the main technical limitations in interactive systems for next-generation audio-visual experiences is the limited resolution of the captured contents. The presented method tackles the problem of generating super-resolved versions of input video frames, thus allowing the user to visualize the captured visual contents at any desired scale with minimal degradation. First, the low-frequency band of the super-resolved video frame is estimated as an up-scaled interpolation of the low-resolution frame. Then, the high-frequency band is extrapolated from the low-resolution frame by exploiting local cross-scale self-similarity. The introduction of a suitable image prior in both stages allows to robustly enhance the spatial resolution even in video sequences containing aliasing. The most demanding processing stages of the presented algorithm have been implemented on graphics hardware (GPU). The experimental results show a level of quality similar to that of state-of-the-art methods, with the advantages of real-time processing and robustness against spatial aliasing.

Demo: virtual director for live event broadcast

We demonstrate the Production Scripting Engine (PSE), a Virtual Director software automatically selecting, framing and cutting camera shots as part of an interactive live broadcast production system. It reasons for multiple viewers with different devices and preferences in parallel within real-time constraints. The PSE takes individual parameters into account and executes a set of pragmatic and cinematographic principles to decide which of the available content is shown. The decision making process is distributed and takes a rule-based approach with exchangable production logic and domain models. The demo uses recorded content from both a high-resolution 180 degree panoramic camera and broadcast cameras. Extracted metadata from both automatic and manual annotation are indexed in an efficient knowledge base, which streams it to the PSE to simulate a live broadcast. We use a specific renderer that allows to view spherical content. Viewers may experience the output on 2 different devices and tweak preferences to adapt the Virtual Directors behavior.