Seyed Ali Amirshahi

PHOG-derived aesthetic measures applied to color photographs of artworks, natural scenes and objects

Previous research in computational aesthetics has led to the identification of multiple image features that, in combination, can be related to the aesthetic quality of images, such as photographs. Moreover, it has been shown that aesthetic artworks possess specific higher-order statistical properties, such as a scale-invariant Fourier spectrum, that can be linked to coding mechanisms in the human visual system. In the present work, we derive novel measures based on a PHOG representation of images for image properties that have been studied in the context of the aesthetic assessment of images previously. We demonstrate that a large dataset of colored aesthetic paintings of Western provenance is characterized by a specific combination of the PHOG-derived aesthetic measures (high self-similarity, moderate complexity and low anisotropy). In this combination, the artworks differ significantly from seven other datasets of photographs that depict various types of natural and man-made scenes, patterns and objects. To the best of our knowledge, this is the first time that these features have been derived and evaluated on a large dataset of different image categories.

Statistical image properties of print advertisements, visual artworks and images of architecture

Most visual advertisements are designed to attract attention, often by inducing a pleasant impression in human observers. Accordingly, results from brain imaging studies show that advertisements can activate the brains reward circuitry, which is also involved in the perception of other visually pleasing images, such as artworks. At the image level, large subsets of artworks are characterized by specific statistical image properties, such as a high self-similarity and intermediate complexity. Moreover, some image properties are distributed uniformly across orientations in the artworks (low anisotropy). In the present study, we asked whether images of advertisements share these properties. To answer this question, subsets of different types of advertisements (single-product print advertisements, supermarket and department store leaflets, magazine covers and show windows) were analyzed using computer vision algorithms and compared to other types of images (photographs of simple objects, faces, large-vista natural scenes and branches). We show that, on average, images of advertisements and artworks share a similar degree of complexity (fractal dimension) and self-similarity, as well as similarities in the Fourier spectrum. However, images of advertisements are more anisotropic than artworks. Values for single-product advertisements resemble each other, independent of the type of product promoted (cars, cosmetics, fashion or other products). For comparison, we studied images of architecture as another type of visually pleasing stimuli and obtained comparable results. These findings support the general idea that, on average, man-made visually pleasing images are characterized by specific patterns of higher-order (global) image properties that distinguish them from other types of images. Whether these properties are necessary or sufficient to induce aesthetic perception and how they correlate with brain activation upon viewing advertisements remains to be investigated.

PHOG analysis of self-similarity in aesthetic images

In recent years, there have been efforts in defining the statistical properties of aesthetic photographs and artworks using computer vision techniques. However, it is still an open question how to distinguish aesthetic from non-aesthetic images with a high recognition rate. This is possibly because aesthetic perception is influenced also by a large number of cultural variables. Nevertheless, the search for statistical properties of aesthetic images has not been futile. For example, we have shown that the radially averaged power spectrum of monochrome artworks of Western and Eastern provenance falls off according to a power law with increasing spatial frequency (1/f2 characteristics). This finding implies that this particular subset of artworks possesses a Fourier power spectrum that is self-similar across different scales of spatial resolution. Other types of aesthetic images, such as cartoons, comics and mangas also display this type of self-similarity, as do photographs of complex natural scenes. Since the human visual system is adapted to encode images of natural scenes in a particular efficient way, we have argued that artists imitate these statistics in their artworks. In support of this notion, we presented results that artists portrait human faces with the self-similar Fourier statistics of complex natural scenes although real-world photographs of faces are not self-similar. In view of these previous findings, we investigated other statistical measures of self-similarity to characterize aesthetic and non-aesthetic images. In the present work, we propose a novel measure of self-similarity that is based on the Pyramid Histogram of Oriented Gradients (PHOG). For every image, we first calculate PHOG up to pyramid level 3. The similarity between the histograms of each section at a particular level is then calculated to the parent section at the previous level (or to the histogram at the ground level). The proposed approach is tested on datasets of aesthetic and non-aesthetic categories of monochrome images. The aesthetic image datasets comprise a large variety of artworks of Western provenance. Other man-made aesthetically pleasing images, such as comics, cartoons and mangas, were also studied. For comparison, a database of natural scene photographs is used, as well as datasets of photographs of plants, simple objects and faces that are in general of low aesthetic value. As expected, natural scenes exhibit the highest degree of PHOG self-similarity. Images of artworks also show high selfsimilarity values, followed by cartoons, comics and mangas. On average, other (non-aesthetic) image categories are less self-similar in the PHOG analysis. A measure of scale-invariant self-similarity (PHOG) allows a good separation of the different aesthetic and non-aesthetic image categories. Our results provide further support for the notion that, like complex natural scenes, images of artworks display a higher degree of self-similarity across different scales of resolution than other image categories. Whether the high degree of self-similarity is the basis for the perception of beauty in both complex natural scenery and artworks remains to be investigated.

Spatial-temporal Video Quality Metric based on an estimation of QoE

In this work a new Reduced Reference (RR) Video Quality Metric (VQM) is proposed. The method takes advantage of the Human Visual System (HVS) sensitivity to sharp changes in the video. The proposed method has a spatial-temporal approach and because of that it is named as STAQ (Spatial-Temporal Assessment of Quality). In the first step of STAQ we take a temporal approach and find the matching regions in consecutive frames. In the next step, a spatial approach is taken in the way of calculating the quality of the matching regions in the temporal approach. In the last step, the quality of the video is calculated based on the parameters gathered in the spatial and temporal domain and using the motion activity density of the video as a controlling factor. An important improvement lies in taking into account the Quality of Experience (QoE) represented as the motion activity density of the reference video. The results show a great improvement in the case of H.264 and MPEG-2 compressed and IP distorted videos even when compared to state of the art Full Reference (FR) metrics.

From regular text to artistic writing and artworks: Fourier statistics of images with low and high aesthetic appeal

The spatial characteristics of letters and their influence on readability and letter identification have been intensely studied during the last decades. There have been few studies, however, on statistical image properties that reflect more global aspects of text, for example properties that may relate to its aesthetic appeal. It has been shown that natural scenes and a large variety of visual artworks possess a scale-invariant Fourier power spectrum that falls off linearly with increasing frequency in log-log plots. We asked whether images of text share this property. As expected, the Fourier spectrum of images of regular typed or handwritten text is highly anisotropic, i.e., the spectral image properties in vertical, horizontal, and oblique orientations differ. Moreover, the spatial frequency spectra of text images are not scale-invariant in any direction. The decline is shallower in the low-frequency part of the spectrum for text than for aesthetic artworks, whereas, in the high-frequency part, it is steeper. These results indicate that, in general, images of regular text contain less global structure (low spatial frequencies) relative to fine detail (high spatial frequencies) than images of aesthetics artworks. Moreover, we studied images of text with artistic claim (ornate print and calligraphy) and ornamental art. For some measures, these images assume average values intermediate between regular text and aesthetic artworks. Finally, to answer the question of whether the statistical properties measured by us are universal amongst humans or are subject to intercultural differences, we compared images from three different cultural backgrounds (Western, East Asian, and Arabic). Results for different categories (regular text, aesthetic writing, ornamental art, and fine art) were similar across cultures.

JenAesthetics Subjective Dataset: Analyzing Paintings by Subjective Scores

Over the last few years, researchers from the computer vision and image processing community have joined other research groups in searching for the bases of aesthetic judgment of paintings and photographs. One of the most important issues, which has hampered research in the case of paintings compared to photographs, is the lack of subjective datasets available for public use. This issue has not only been mentioned in different publications, but was also widely discussed at different conferences and workshops. In the current work, we perform a subjective test on a recently released dataset of aesthetic paintings. The subjective test not only collects scores based on the subjective aesthetic quality, but also on other properties that have been linked to aesthetic judgment.

How self-similar are artworks at different levels of spatial resolution?

Recent research has shown that a large variety of aesthetic paintings are highly self-similar. The degree of self-similarity seen in artworks is close to that observed for complex natural scenes, to which low-level visual coding in the human visual system is adapted. In this paper, we introduce a new measure of self-similarity, which we will refer to as the Weighted Self-Similarity (WSS). Using PHOG, which is a state-of-the-art technique from computer vision, WSS is derived from a measure that has been previously linked to aesthetic paintings and represents self-similarity on a single level of spatial resolution. In contrast, WSS takes into account the similarity values at multiple levels of spatial resolution. The values are linked to each other by using a weighting factor so that the overall self-similarity of an image reflects how self-similarity changes at different spatial levels. Compared to the previously proposed metric, WSS has the advantage that it also takes into account differences between self-similarity at different levels of spatial resolution with respect to one another. An analysis of a large image dataset of aesthetic artworks (the JenAesthetics dataset) and other categories of images reveals that artworks, on average, show a relatively high WSS. Similarly, high values for WSS were obtained for images of natural patterns that can be described as being fractal (for example, images of clouds, branches or lichen growth patterns). The analysis of the JenAesthetics dataset, which consists of paintings of Western provenance, yielded similar values of WSS for different art styles. In conclusion, self-similarity is uniformly high across different levels of spatial resolution in the artworks analyzed in the present study.

Evaluating the Rule of Thirds in Photographs and Paintings

The rule of thirds (ROT) is one of the best-known composition rules used in painting and photography. According to this rule, the focus point of an image should be placed along one of the third lines or on one of the four intersections of the third lines, to give aesthetically pleasing results. Recently, calculated saliency maps have been used in an attempt to predict whether or not images obey the rule of thirds. In the present study, we challenged this computer-based approach by comparing calculated ROT values with behavioral (subjective) ROT scores obtained from 30 participants in a psychological experiment. For photographs that did not follow the rule of thirds, subjective ROT scores matched calculated ROT values reasonably well. For photographs that followed the rule of thirds, we found a moderate correlation between subjective scores and calculated values. However, aesthetic rating scores correlated only weakly with subjective ROT scores and not at all with calculated ROT values. Moreover, for photographs that were rated as highly aesthetic and for a large set of paintings, calculated ROT values were about as low as in photographs that did not follow the rule of thirds. In conclusion, the computer-based ROT metrics can predict the behavioral data, but not completely. Despite its proclaimed importance in artistic composition, the rule of thirds seems to play only a minor, if any, role in large sets of high-quality photographs and paintings.

A new approach for image quality assessment using SVD

The traditional mean square error (MSE) and signal to noise ratio (SNR) measurements are based on the pixel-by-pixel differences between the original and the distorted image. These quality values are not adapted with the human visual system (HVS). Also, they can not distinguish some different corruptions. In this paper, we use the singular value decomposition algorithm of each 8times8 blocks of image to calculate the ratio of the first and second largest eigenvalues to measure the distribution of brightness. So, we could measure the quality of an image and hence to determine the type of the relating distortion. The method is so simple that the simulation results easily arrange and show the best images in order.

Preference for Well-Balanced Saliency in Details Cropped from Photographs

Photographic cropping is the act of selecting part of a photograph to enhance its aesthetic appearance or visual impact. It is common practice with both professional (expert) and amateur (non-expert) photographers. In a psychometric study, McManus et al. (2011b) showed that participants cropped photographs confidently and reliably. Experts tended to select details from a wider range of positions than non-experts, but other croppers did not generally prefer details that were selected by experts. It remained unclear, however, on what grounds participants selected particular details from a photograph while avoiding other details. One of the factors contributing to cropping decision may be visual saliency. Indeed, various saliency-based computer algorithms are available for the automatic cropping of photographs. However, careful experimental studies on the relation between saliency and cropping are lacking to date. In the present study, we re-analyzed the data from the studies by McManus et al. (2011a,b), focusing on statistical image properties. We calculated saliency-based measures for details selected and details avoided during cropping. As expected, we found that selected details contain regions of higher saliency than avoided details on average. Moreover, the saliency center-of-mass was closer to the geometrical center in selected details than in avoided details. Results were confirmed in an eye tracking study with the same dataset of images. Interestingly, the observed regularities in cropping behavior were less pronounced for experts than for non-experts. In summary, our results suggest that, during cropping, participants tend to select salient regions and place them in an image composition that is well-balanced with respect to the distribution of saliency. Our study contributes to the knowledge of perceptual bottom-up features that are germane to aesthetic decisions in photography and their variability in non-experts and experts.