Rafal Mantiuk

Display adaptive tone mapping

We propose a tone mapping operator that can minimize visible contrast distortions for a range of output devices, ranging from e-paper to HDR displays. The operator weights contrast distortions according to their visibility predicted by the model of the human visual system. The distortions are minimized given a display model that enforces constraints on the solution. We show that the problem can be solved very efficiently by employing higher order image statistics and quadratic programming. Our tone mapping technique can adjust image or video content for optimum contrast visibility taking into account ambient illumination and display characteristics. We discuss the differences between our method and previous approaches to the tone mapping problem.

Predicting visible differences in high dynamic range images: model and its calibration

New imaging and rendering systems commonly use physically accurate lighting information in the form of high-dynamic range (HDR) images and video. HDR images contain actual colorimetric or physical values, which can span 14 orders of magnitude, instead of 8-bit renderings, found in standard images. The additional precision and quality retained in HDR visual data is necessary to display images on advanced HDR display devices, capable of showing contrast of 50,000:1, as compared to the contrast of 700:1 for LCD displays. With the development of high-dynamic range visual techniques comes a need for an automatic visual quality assessment of the resulting images. In this paper we propose several modifications to the Visual Difference Predicator (VDP). The modifications improve the prediction of perceivable differences in the full visible range of luminance and under the adaptation conditions corresponding to real scene observation. The proposed metric takes into account the aspects of high contrast vision, like scattering of the light in the optics (OTF), nonlinear response to light for the full range of luminance, and local adaptation. To calibrate our HDR VDP we perform experiments using an advanced HDR display, capable of displaying the range of luminance that is close to that found in real scenes.

Perception-motivated high dynamic range video encoding

Due to rapid technological progress in high dynamic range (HDR) video capture and display, the efficient storage and transmission of such data is crucial for the completeness of any HDR imaging pipeline. We propose a new approach for inter-frame encoding of HDR video, which is embedded in the well-established MPEG-4 video compression standard. The key component of our technique is luminance quantization that is optimized for the contrast threshold perception in the human visual system. The quantization scheme requires only 10--11 bits to encode 12 orders of magnitude of visible luminance range and does not lead to perceivable contouring artifacts. Besides video encoding, the proposed quantization provides perceptually-optimized luminance sampling for fast implementation of any global tone mapping operator using a lookup table. To improve the quality of synthetic video sequences, we introduce a coding scheme for discrete cosine transform (DCT) blocks with high contrast. We demonstrate the capabilities of HDR video in a player, which enables decoding, tone mapping, and applying post-processing effects in real-time. The tone mapping algorithm as well as its parameters can be changed interactively while the video is playing. We can simulate post-processing effects such as glare, night vision, and motion blur, which appear very realistic due to the usage of HDR data.

Backward compatible high dynamic range MPEG video compression

To embrace the imminent transition from traditional low-contrast video (LDR) content to superior high dynamic range (HDR) content, we propose a novel backward compatible HDR video compression (HDR MPEG) method. We introduce a compact reconstruction function that is used to decompose an HDR video stream into a residual stream and a standard LDR stream, which can be played on existing MPEG decoders, such as DVD players. The reconstruction function is finely tuned to the content of each HDR frame to achieve strong decorrelation between the LDR and residual streams, which minimizes the amount of redundant information. The size of the residual stream is further reduced by removing invisible details prior to compression using our HDR-enabled filter, which models luminance adaptation, contrast sensitivity, and visual masking based on the HDR content. Designed especially for DVD movie distribution, our HDR MPEG compression method features low storage requirements for HDR content resulting in a 30% size increase to an LDR video sequence. The proposed compression method does not impose restrictions or modify the appearance of the LDR or HDR video. This is important for backward compatibility of the LDR stream with current DVD appearance, and also enables independent fine tuning, tone mapping, and color grading of both streams.

Dynamic range independent image quality assessment

The diversity of display technologies and introduction of high dynamic range imagery introduces the necessity of comparing images of radically different dynamic ranges. Current quality assessment metrics are not suitable for this task, as they assume that both reference and test images have the same dynamic range. Image fidelity measures employed by a majority of current metrics, based on the difference of pixel intensity or contrast values between test and reference images, result in meaningless predictions if this assumption does not hold. We present a novel image quality metric capable of operating on an image pair where both images have arbitrary dynamic ranges. Our metric utilizes a model of the human visual system, and its central idea is a new definition of visible distortion based on the detection and classification of visible changes in the image structure. Our metric is carefully calibrated and its performance is validated through perceptual experiments. We demonstrate possible applications of our metric to the evaluation of direct and inverse tone mapping operators as well as the analysis of the image appearance on displays with various characteristics.

Optimizing a Tone Curve for Backward-Compatible High Dynamic Range Image and Video Compression

For backward compatible high dynamic range (HDR) video compression, the HDR sequence is reconstructed by inverse tone-mapping a compressed low dynamic range (LDR) version of the original HDR content. In this paper, we show that the appropriate choice of a tone-mapping operator (TMO) can significantly improve the reconstructed HDR quality. We develop a statistical model that approximates the distortion resulting from the combined processes of tone-mapping and compression. Using this model, we formulate a numerical optimization problem to find the tone-curve that minimizes the expected mean square error (MSE) in the reconstructed HDR sequence. We also develop a simplified model that reduces the computational complexity of the optimization problem to a closed-form solution. Performance evaluations show that the proposed methods provide superior performance in terms of HDR MSE and SSIM compared to existing tone-mapping schemes. It is also shown that the LDR image quality resulting from the proposed methods matches that produced by perceptually-based TMOs.

Comparison of Four Subjective Methods for Image Quality Assessment

To provide a convincing proof that a new method is better than the state of the art, computer graphics projects are often accompanied by user studies, in which a group of observers rank or rate results of several algorithms. Such user studies, known as subjective image quality assessment experiments, can be very time‐consuming and do not guarantee to produce conclusive results. This paper is intended to help design efficient and rigorous quality assessment experiments and emphasise the key aspects of the results analysis. To promote good standards of data analysis, we review the major methods for data analysis, such as establishing confidence intervals, statistical testing and retrospective power analysis. Two methods of visualising ranking results together with the meaningful information about the statistical and practical significance are explored. Finally, we compare four most prominent subjective quality assessment methods: single‐stimulus, double‐stimulus, forced‐choice pairwise comparison and similarity judgements. We conclude that the forced‐choice pairwise comparison method results in the smallest measurement variance and thus produces the most accurate results. This method is also the most time‐efficient, assuming a moderate number of compared conditions.

Visible difference predicator for high dynamic range images

Since new imaging and rendering systems commonly use physically accurate lighting information in the form of high-dynamic range data, there is a need for an automatic visual quality assessment of the resulting images. In this work we extend the visual difference predictor (VDP) developed by Daly to handle HDR data. This let us predict if a human observer is able to perceive differences for a pair of HDR images under the adaptation conditions corresponding to the real scene observation.

Extending quality metrics to full luminance range images

Many quality metrics take as input gamma corrected images and assume that pixel code values are scaled perceptually uniform. Although this is a valid assumption for darker displays operating in the luminance range typical for CRT displays (from 0.1 to 80 cd/m2), it is no longer true for much brighter LCD displays (typically up to 500 cd/m2), plasma displays (small regions up to 1000 cd/m2) and HDR displays (up to 3000 cd/m2). The distortions that are barely visible on dark displays become clearly noticeable when shown on much brighter displays. To estimate quality of images shown on bright displays, we propose a straightforward extension to the popular quality metrics, such as PSNR and SSIM, that makes them capable of handling all luminance levels visible to the human eye without altering their results for typical CRT display luminance levels. Such extended quality metrics can be used to estimate quality of high dynamic range (HDR) images as well as account for display brightness.

High dynamic range imaging pipeline: perception-motivated representation of visual content

The advances in high dynamic range (HDR) imaging, especially in the display and camera technology, have a significant impact on the existing imaging systems. The assumptions of the traditional low-dynamic range imaging, designed for paper print as a major output medium, are ill suited for the range of visual material that is shown on modern displays. For example, the common assumption that the brightest color in an image is white can be hardly justified for high contrast LCD displays, not to mention next generation HDR displays, that can easily create bright highlights and the impression of self-luminous colors. We argue that high dynamic range representation can encode images regardless of the technology used to create and display them, with the accuracy that is only constrained by the limitations of the human eye and not a particular output medium. To facilitate the research on high dynamic range imaging, we have created a software package (http://pfstools.sourceforge.net/) capable of handling HDR data on all stages of image and video processing. The software package is available as open source under the General Public License and includes solutions for high quality image acquisition from multiple exposures, a range of tone mapping algorithms and a visual difference predictor for HDR images. Examples of shell scripts demonstrate how the software can be used for processing single images as well as video sequences.