Pascal Peter

Turning Diffusion-Based Image Colorization Into Efficient Color Compression

The work of Levin et al. (2004) popularized stroke-based methods that add color to gray value images according to a small amount of user-specified color samples. Even though such reconstructions from sparse data suggest a possible use in compression, only few attempts were made so far in this direction. Diffusion-based compression methods pursue a similar idea: they store only few image pixels and inpaint the missing regions. Despite this close relation and a lack of diffusion-based color codecs, colorization ideas were so far only integrated into transform-based approaches such as JPEG. We address this missing link with two contributions. First, we show the relation between the discrete colorization of Levin et al. and continuous diffusion-based inpainting in the YCbCr color space. It decomposes the image into a luma (brightness) channel and two chroma (color) channels. Our luma-guided diffusion framework steers the diffusion inpainting in the chroma channels according to the structure in the luma channel. We show that making the luma-guided colorization anisotropic outperforms the method of Levin et al. significantly. Second, we propose a new luma preference codec that invests a large fraction of the bit budget into an accurate representation of the luma channel. This allows a high-quality reconstruction of color data with our colorization technique. Simultaneously, we exploit the fact that the human visual system is more sensitive to structural than to color information. Our experiments demonstrate that our new codec outperforms the state of the art in diffusion-based image compression and is competitive to transform-based codecs.

Colour image compression with anisotropic diffusion

Schmaltz et al. (2009) have shown that for reasonably high compression rates, diffusion-based codecs can exceed the quality of transformation-based methods such as JPEG 2000. They store only data at a few optimised pixel locations and in-paint missing data with edge-enhancing anisotropic diffusion (EED). However, research on compression with diffusion methods has mainly focussed on grey-value images, and colour images have been compressed in a straightforward way using anisotropic diffusion in RGB space. So far, there is no sophisticated diffusion-based counterpart to the colour mode of JPEG 2000. To address this shortcoming we introduce an advanced colour compression codec that exploits properties of the human visual system in YCbCr space. Since details in the luma channel Y are perceptually relevant, we invest a large fraction of our bit budget in its encoding with high fidelity. For the chroma channels Cb and Cr, the stored information can be very sparse, if we guide the EED-based inpainting with the high quality diffusion tensor from the luma reconstruction. Experiments demonstrate that our novel codec outperforms JPEG 2000 and compression with RGB-diffusion, both visually and quantitatively.

Evaluating the true potential of diffusion-based inpainting in a compression context

Partial differential equations (PDEs) are able to reconstruct images accurately from a small fraction of their image points. The inpainting capabilities of sophisticated anisotropic PDEs allow compression codecs with suboptimal data selection approaches to compete with transform-based methods like JPEG2000. For simple linear PDEs, optimal known data can be found with powerful optimisation strategies. However, the potential of these linear methods for compression has so far not yet been determined.As a remedy, we present a compression framework with homogeneous, biharmonic, and edge-enhancing diffusion (EED) that supports different strategies for data selection and storage: on the one hand, we find exact masks with optimal control or stochastic sparsification and store them with a combination of PAQ and block coding. On the other hand, we propose a new probabilistic strategy for the selection of suboptimal known data that can be efficiently stored with binary trees and entropy coding.This new framework allows us a detailed analysis of the strengths and weaknesses of the three PDEs. Our investigation leads to surprising results: at low compression rates, the simple harmonic diffusion can surpass its more sophisticated PDE-based competitors and even JPEG2000. For high compression rates, we find that EED yields the best result due to its robust inpainting performance under suboptimal conditions. HighlightsPDEs for compression can only be properly evaluated in the context of actual codecs.Simple linear PDEs can be competitive with optimised known data.Sophisticated anisotropic PDEs are well-suited for suboptimal known data.

Compressing Images with Diffusion- and Exemplar-Based Inpainting

Diffusion-based image compression methods can surpass state-of-the-art transform coders like JPEG 2000 for cartoon-like images. However, they are not well-suited for highly textured image content. Recently, advances in exemplar-based inpainting have made it possible to reconstruct images with non-local methods from sparse known data. In our work we compare the performance of such exemplar-based and diffusion-based inpainting algorithms, dependent on the type of image content. We use our insights to construct a hybrid compression codec that combines the strengths of both approaches. Experiments demonstrate that our novel method offers significant advantages over state-of-the-art diffusion-based methods on textured image data and can compete with transform coders.

Three-Dimensional Data Compression with Anisotropic Diffusion

In 2-D image compression, recent approaches based on image inpainting with edge-enhancing anisotropic diffusion (EED) rival the transform-based quasi-standards JPEG and JPEG 2000 and are even able to surpass it. In this paper, we extend successful concepts from these 2-D methods to the 3-D setting, thereby establishing the first PDE-based 3-D image compression algorithm. This codec uses a cuboidal subdivision strategy to select and efficiently store a small set of sparse image data and reconstructs missing image parts with EED-based inpainting. An evaluation on real-world medical data substantiates the superior performance of this new algorithm in comparison to 2-D inpainting methods and the compression standard DICOM for medical data.

From Optimised Inpainting with Linear PDEs Towards Competitive Image Compression Codecs

For inpainting with linear partial differential equations PDEs such as homogeneous or biharmonic diffusion, sophisticated data optimisation strategies have been found recently. These allow high-quality reconstructions from sparse known data. While they have been explicitly developed with compression in mind, they have not entered actual codecs so far: Storing these optimised data efficiently is a nontrivial task. Since this step is essential for any competetive codec, we propose two new compression frameworks for linear PDEs: Efficient storage of pixel locations obtained from an optimal control approach, and a stochastic strategy for a locally adaptive, tree-based grid. Suprisingly, our experiments show that homogeneous diffusion inpainting can surpass its often favoured biharmonic counterpart in compression. Last but not least, we demonstrate that our linear approach is able to beat both JPEG2000 and the nonlinear state-of-the-art in PDE-based image compression.

Justifying Tensor-Driven Diffusion from Structure-Adaptive Statistics of Natural Images

Tensor-driven anisotropic diffusion and regularisation have been successfully applied to a wide range of image processing and computer vision tasks such as denoising, inpainting, and optical flow. Empirically it has been shown that anisotropic models with a diffusion tensor perform better than their isotropic counterparts with a scalar-valued diffusivity function. However, the reason for this superior performance is not well understood so far. Moreover, the specific modelling of the anisotropy has been carried out in a purely heuristic way. The goal of our paper is to address these problems. To this end, we use the statistics of natural images to derive a unifying framework for eight isotropic and anisotropic diffusion filters that have a corresponding variational formulation. In contrast to previous statistical models, we systematically investigate structure-adaptive statistics by analysing the eigenvalues of the structure tensor. With our findings, we justify existing successful models and assess the relationship between accurate statistical modelling and performance in the context of image denoising.

Beyond pure quality

We add new features to the diffusion-based codec R-EED that go beyond pure quality.Our new progressive modes for R-EED can outperform JPEG and JPEG2000.Tree-based subdivision allows efficient region of interest coding.Diffusion-based video decoding is possible in real-time without sacrificing quality.Our newly proposed features are pairwise compatible and can be used simultaneously. Compared to transform-based image compression methods such as JPEG2000, approaches based on partial-differential equations (PDEs) are in a proof-of-concept stage. Nevertheless, R-EED, a codec employing edge-enhancing anisotropic diffusion (EED) and rectangular subdivision, can surpass JPEG2000 quality-wise. However, todays requirements for compression algorithms go beyond pure compression performance. Codecs must also fulfil the feature requirements of specific applications such as online media or medical imaging. We propose three such features for the R-EED codec. By reordering grey values and exploiting the subdivision scheme, we incorporate a progressive mode into R-EED that can outperform JPEG and JPEG2000. Additionally, we show that rectangular subdivision is well-suited for region of interest coding and adapt the quality of image parts according to their importance. Finally, we propose a real-time video player that demonstrates how R-EED-based decoding can be performed efficiently. All of these extensions are compatible with each other and can be used simultaneously.

A proof-of-concept framework for PDE-based video compression

In image compression, codecs that rely on interpolation with partial differential equations (PDEs) are becoming increasingly popular. However, there have not been many attempts to transfer this concept to video compression. Since real-time performance is challenging for PDE-based reconstruction, first efficient approaches work on a frame-by-frame basis and focus on parallel implementations without considering coding quality. So far, there is no fully PDE-based video codec that exploits temporal redundancies. As a remedy, we propose a modular framework that combines PDE-based compression with motion compensation: Intra frames are predicted with PDE-based inpainting and inter frames with dense optic flow fields. We use this framework to develop a proof-of-concept codec that combines homogeneous diffusion inpainting with the variational optic flow model of Brox et al. (2004). Even without sophisticated parallelisation, we are able to perform real-time decompression of colour videos for the first time in PDE-based video compression.

Denoising by Inpainting

The filling-in effect of diffusion processes has been successfully used in many image analysis applications. Examples include image reconstructions in inpainting-based compression or dense optic flow computations. As an interesting side effect of diffusion-based inpainting, the interpolated data are smooth, even if the known image data are noisy: Inpainting averages information from noisy sources. Since this effect has not been investigated for denoising purposes so far, we propose a general framework for denoising by inpainting. It averages multiple inpainting results from different selections of known data. We evaluate two concrete implementations of this framework: The first one specifies known data on a shifted regular grid, while the second one employs probabilistic densification to optimise the known pixel locations w.r.t. the inpainting quality. For homogeneous diffusion inpainting, we demonstrate that our regular grid method approximates the quality of its corresponding diffusion filter. The densification algorithm with homogeneous diffusion inpainting, however, shows edge-preserving behaviour. It resembles space-variant diffusion and offers better reconstructions than homogeneous diffusion filters.