Fabrice Urban

Saliency-Guided Consistent Color Harmonization

The focus of this paper is automatic color harmonization, which amounts to re-coloring an image so that the obtained color palette is more harmonious for human observers. The proposed automatic algorithm builds on the pioneering works described in [3,12] where templates of harmonious colors are defined on the hue wheel. We bring three contributions in this paper: first, saliency [9] is used to predict the most attractive visual areas and estimate a consistent harmonious template. Second, an efficient color segmentation algorithm, adapted from [4], is proposed to perform consistent color mapping. Third, a new mapping function substitutes usual color shifting method. Results show that the method limits the visual artifacts of state-of-the-art methods and leads to a visually consistent harmonization.

No-reference Harmony-Guided Quality Assessment

Color harmony of simple color patterns has been widely studied for color design. Rules defined then by psychological experiments have been applied to derive image aesthetic scores, or to re-colorize pictures. But what is harmonious or not in an image? What can the human eye perceive disharmonious? Extensive research has been done in the context of quality assessment to define what is visible or not in images and videos. Techniques based on human visual system models use signal masking to define visibility thresholds. Based on results in both fields, we present a harmony quality assessment method to assess what is harmonious or not in an image. Color rules are used to detect what part of images are disharmonious, and visual masking is applied to estimate to what extent an image area can be perceived disharmonious. The output perceptual harmony quality map and scores can be used in a photo editing framework to guide the user getting the best artistic effects. Results show that the harmony maps reflect what a user perceives and that the score is correlated to the artistic intent.

Image and video saliency models improvement by blur identification

Visual saliency models aim at predicting where people look. In free viewing conditions, people look at relevant objects that are in focus. Assuming blurred or out-of-focus objects do not belong to the region of interest, this paper proposes a significant improvement and the validation of a saliency model by taking blur into account. Blur identification is associated to a spatio-temporal saliency model. Bottom-up models are designed to mimic the low-level processing of the human visual system and can thus detect out-of-focus objects as salient. The blur identification allows decreasing saliency values on blurred areas while increasing values on sharp areas. In order to validate our new saliency model we conducted eye-tracking experiments to record ground truth of observers fixations on images and videos. Blur identification significantly improves fixation prediction for natural images and videos.

Perceptual video coding based on MB classification and rate–distortion optimization

Abstract The efficiency of a video coding scheme should be defined as the video quality achieved for a given bit-rate. The quality refers here to the subjective quality perceived by the final user. This is the only valid criteria, and it involves the user vision properties. Therefore, in order to optimize this perceived video quality, a new perceptual coding scheme is proposed taking into account the Human Visual System (HVS). Perceptual distortion measures are included in the encoding loop to compute an adaptive local quantization step. The idea is that optimizing the choice of macroblock quantization parameters based on a more accurate perceptual distortion measure would result in an improved subjective quality. The resulting macroblock-level rate allocation problem is first modeled as a constrained optimization problem solved with a Lagrangian multiplier-based algorithm. The local adaptation is performed by a classification of each macroblock according to its distortion–quantization properties. A learning strategy applied on a set of sequences provides a set of representative distortion–quantization curves for all macroblock types. Then a rate–quantization model is derived from the ρ - domain linear approximation to reach the target bit-rate. A two-pass only algorithm is required to compute the necessary models and encode each macroblock. The complexity of the proposed solution is thus reduced compared to exhaustive search algorithms. The SSIM (Structural SIMilarity) is used as in-loop distortion measure, and the performances are evaluated with another perceptual measure, more correlated to the user perception, called the WQA (Wavelet Quality Assessment). The coding schemes being based on a biological possible modeling of the Human Visual System, the video includes less visual artifacts and a more uniform subjective quality, compared to traditional rate–distortion coding.

Harmony-guided image editing

While image editing tends to be popularized, it remains a time-consuming task requiring a minimal expertise. This proposal presents an innovative tool for performing guided image editing. Oriented by a quality metric based on color harmony theory, the user picks up the least harmonious color areas and retouches them based on our recommended color palette. This set of proposed colors ensure the maximization of the global color harmony for the considered picture. In few clicks, anyone is able to retouch incongruous colors while respecting his taste or intent.

Creating experimental color harmony map

Starting in the 17th century with Newton, color harmony is a topic that did not reach a consensus on definition, representation or modeling so far. Previous work highlighted specific characteristics for color harmony on com- bination of color doublets or triplets by means of a human rating on a harmony scale. However, there were no investigation involving complex stimuli or pointing out how harmony is spatially located within a picture. The modeling of such concept as well as a reliable ground-truth would be of high value for the community, since the applications are wide and concern several communities: from psychology to computer graphics. We propose a protocol for creating color harmony maps from a controlled experiment. Through an eye-tracking protocol, we focus on the identification of disharmonious colors in pictures. The experiment was composed of a free viewing pass in order to let the observer be familiar with the content before a second pass where we asked ”to search for the most disharmonious areas in the picture”. Twenty-seven observers participated to the experiments that was composed of a total of 30 different stimuli. The high inter-observer agreement as well as a cross-validation confirm the validity of the proposed ground-truth.

Erratum: Saliency-Guided Consistent Color Harmonization

In the original version, reference 16 was wrong. It should read as follows: 16. Skurowski, P., Kozielski, M.: Investigating human color harmony preferences using unsupervised machine learning. In: European Conference on Colour in Graphics, Imaging, and Vision, pp. 59-64 2012