Zicong Mai

Optimizing a Tone Curve for Backward-Compatible High Dynamic Range Image and Video Compression

For backward compatible high dynamic range (HDR) video compression, the HDR sequence is reconstructed by inverse tone-mapping a compressed low dynamic range (LDR) version of the original HDR content. In this paper, we show that the appropriate choice of a tone-mapping operator (TMO) can significantly improve the reconstructed HDR quality. We develop a statistical model that approximates the distortion resulting from the combined processes of tone-mapping and compression. Using this model, we formulate a numerical optimization problem to find the tone-curve that minimizes the expected mean square error (MSE) in the reconstructed HDR sequence. We also develop a simplified model that reduces the computational complexity of the optimization problem to a closed-form solution. Performance evaluations show that the proposed methods provide superior performance in terms of HDR MSE and SSIM compared to existing tone-mapping schemes. It is also shown that the LDR image quality resulting from the proposed methods matches that produced by perceptually-based TMOs.

Subjective evaluation of tone-mapping methods on 3D images

High dynamic range (HDR) imaging provides superior picture quality to traditional 8 bit, low dynamic range (LDR), image representations. Capturing images/videos in HDR format can avoid problems with over and under exposures. Tone-mapping is a process that converts from HDR to LDR, so that HDR content can be shown on existing displays. Tone mapping has been extensively studied in the context of 2D images/video but not for 3D content. This paper addresses the problem of presenting 3D HDR content on stereoscopic LDR displays and presents a subjective psychophysical experiment that evaluates existing tone-mapping operators on 3D HDR images. The results show that 3D content derived using tone-mapping is much preferred to that is captured directly with a pair of LDR cameras. Global tone-mapping methods (which better preserve global contrast) are found to produce images with better 3D effect than local tone-mapping operators (which produce images with high amounts of detail/texture). Also, the brightness of the tone-mapped images is found to be highly collated with perceived 3D quality.

Visually Favorable Tone-Mapping With High Compression Performance in Bit-Depth Scalable Video Coding

In bit-depth scalable video coding, the tone-mapping scheme used to convert high-bit-depth to eight-bit videos is an essential yet very often ignored component. In this paper, we demonstrate that an appropriate choice of a tone-mapping operator can improve the coding efficiency of bit-depth scalable encoders. We present a new tone-mapping scheme that delivers superior compression efficiency while adhering to a predefined base layer perceptual quality. We develop numerical models that estimate the base layer bit-rate (Rb), the enhancement layer bitrate (Re), and the mismatch (QL) between the resulting low dynamic range (LDR) base-layer signal and the predefined base layer representation. Our proposed tone curve is given by the solution of an optimization problem which minimizes a weighted sum of Rb, Re, and QL. The problem formulation also considers the temporal effect of tone-mapping by adding a constraint to the optimization problem that suppresses flickering artifacts. We also propose a technique with which to tone-map a high-bit-depth video directly in a compression-friendly color space (e.g., one luma and two chroma channels) without converting to the RGB domain. Experimental results show that we can save up to 40% of the total bit-rate (or 3.5 dB PSNR improvement for the same bitrate), and, in general, about 20% bit-rate savings can be achieved.

On-the-fly tone mapping for backward-compatible high dynamic range image/video compression

In this paper, we propose a real-time tone-mapping scheme for backward compatible high dynamic range (HDR) video compression. The appropriate choice of a tone-mapping operator (TMO) can significantly improve the HDR quality reconstructed from a low dynamic range (LDR) version. We develop a statistical model that approximates the mean square error (MSE) distortion resulting from the combined processes of tone-mapping and compression. Using this model, we formulate a numerical optimization problem to find the tone-curve that minimizes the expected MSE in the reconstructed HDR sequence. We then simplify the developed model in order to reduce the computational complexity of the optimization problem to a closed-form solution. Performance evaluations show that the proposed methods provide superior performance in terms of HDR MSE and SSIM compared to existing tone-mapping schemes. It is also shown that the LDR image quality resulting from the proposed methods matches that produced by perceptually-based TMOs.

Effect of brightness on the quality of visual 3D perception

Over the years, a consensus has been reached that the introduction of 3D entertainment can only be a lasting success if the perceived image quality and the viewing comfort are better than those of conventional 2D television. There are different factors that affect the perceived quality of 3D content. In this paper, our objective is to obtain a good understanding of the effect that brightness has on the visual quality of 3D videos and compare it to that of the 2D. We capture outdoor and indoor scenes with different exposures and we perform subjective evaluation to investigate how brightness affects the perceived quality of the 3D experience.

Rendering 3-D High Dynamic Range Images: Subjective Evaluation of Tone-Mapping Methods and Preferred 3-D Image Attributes

High dynamic range (HDR) images provide superior picture quality by allowing a larger range of brightness levels to be captured and reproduced than traditional 8-bit low dynamic range (LDR) images. Even with existing 8-bit displays, picture quality can be significantly improved if content is first captured in HDR format, and then is tone-mapped to convert it from HDR to the LDR format. Tone mapping methods have been extensively studied for 2-D images. This paper addresses the problem of presenting stereoscopic tone-mapped HDR images on 3-D LDR displays and how it is different from the 2-D scenario. We first present a subjective psychophysical experiment that evaluates existing tone-mapping operators on 3-D HDR images. The results show that 3-D content derived using tone-mapping is much preferred to that captured directly with a pair of LDR cameras. Global (spatially invariant) and local (spatially variant) tone-mapping methods have similar 3-D effects. The second part of our study focuses on how the preferred level of brightness and the preferred amount of details differ between 3-D and 2-D images by conducting another set of subjective experiments. Our results show that while people selected slightly brighter images in 3-D viewing compared to 2-D, the difference is not statistically significant. However, compared to 2-D images, the subjects consistently preferred having a greater amount of details when watching 3-D. These results suggest that 3-D content should be prepared differently (sharper and possibly slightly brighter) from the same content intended for 2-D displaying, to achieve optimal appearance in each format. The complete database of the original HDR image pairs and their LDR counterparts are available online.

Visually-favorable tone-mapping with high compression performance

In bit-depth scalable video coding, the tone-mapping scheme used to convert high-bit-depth to eight-bit videos is an essential yet very often ignored component. In this paper, we demonstrate that an appropriate choice of a tone-mapping operator can improve the coding efficiency of bit-depth scalable encoders. We present a new tone-mapping scheme that delivers superior compression efficiency while adhering to a predefined base layer perceptual quality. We develop numerical models that estimate the base layer bit-rate (Rb), the enhancement layer bitrate (Re), and the mismatch (QL) between the resulting low dynamic range (LDR) base-layer signal and the predefined base layer representation. Our proposed tone curve is given by the solution of an optimization problem which minimizes a weighted sum of Rb, Re, and QL. The problem formulation also considers the temporal effect of tone-mapping by adding a constraint to the optimization problem that suppresses flickering artifacts. We also propose a technique with which to tone-map a high-bit-depth video directly in a compression-friendly color space (e.g., one luma and two chroma channels) without converting to the RGB domain. Experimental results show that we can save up to 40% of the total bit-rate (or 3.5 dB PSNR improvement for the same bitrate), and, in general, about 20% bit-rate savings can be achieved.

Contrast effect on 3D and 2D video perception

There are many factors and parameters that have an effect on the perceptual quality of 2D and 3D media. This paper investigates how the contrast affects 3D and 2D perceptions. We implemented the double stimulus continuous quality scale method to conduct subjective tests that evaluate the perceptual quality of experience of 3D video and its 2D counterpart at different contrast levels between the object of interest and the background. Our test set-up guarantees that the results are independent of the brightness reduction caused by a specific 3D viewing/displaying technology. Statistical analysis shows that the effect of contrast on the perceptual quality of 2D and 3D videos is similar.

A wavelet-based intra-prediction lossless image compression scheme

We propose a wavelet-based intra-prediction image coding scheme that is very efficient for lossless compression. This scheme takes advantage of JPEG-2000s wavelet decomposition and H.264s intra-prediction capability to compress still images. The compression performance is analyzed, and the results demonstrate that our method outperforms H.264 intra-mode coding by about 7% and JPEG-2000 by around 15% for the lossless case.

Efficient DVB-MHP to Blu-Ray System Information Transcoding

Blu-ray is a new-generation DVD format. Compared to the traditional DVD technology, Blu-ray offers advanced interactive features, high definition video quality and storage capacity that is more than 5 times that of DVD. DVB-MHP, on the other hand, is the latest Digital Video Broadcasting standard which supports a Multimedia Home Platform, specifically designed for offering advanced interactive TV services. Although both systems support Java-based interactivity, there are several differences between the two to make them incompatible. Our ultimate goal is to ensure compatibility between the two interactive services, enabling the video and interactive contents transmitted by the DVB-MHP system to be played in real time on the Blu-ray platform. One of the main challenges in realizing this compatibility is the conversion of DVB-MHP metadata, which are System Information transmitted in MPEG-2 Transport Stream packets, to the format supported by the Blu-ray standard. This study analyzed the differences between the two standards in terms of the System Information and proposed methods that ensure the desired compatibility. An optimized method for generating Program Association, Program Map and Selection Information contents for the Blu-ray system using DVB-MHP information at the elementary stream and transport stream levels is presented. Our approach takes advantage of the existing correlation between the two systems, adapts the existing DVB-MHP metadata, and intelligently searches for the information needed from the DVB-MHP elementary streams to generate in real-time the Blu-ray equivalent information.