Pere Obrador

Towards computational models of the visual aesthetic appeal of consumer videos

In this paper, we tackle the problem of characterizing the aesthetic appeal of consumer videos and automatically classifying them into high or low aesthetic appeal. First, we conduct a controlled user study to collect ratings on the aesthetic value of 160 consumer videos. Next, we propose and evaluate a set of low level features that are combined in a hierarchical way in order to model the aesthetic appeal of consumer videos. After selecting the 7 most discriminative features, we successfully classify aesthetically appealing vs. aesthetically unappealing videos with a 73% classification accuracy using a support vector machine.

The role of image composition in image aesthetics

In this paper, we explore the role that image composition plays in image aesthetic appeal classification. We propose low-level image composition features that approximate traditional photography composition guidelines, such as simplicity and visual balance (e.g. golden mean, golden triangles). We then use these features to build an image aesthetics classifier that we test with a baseline dataset. Interestingly, our approach, that only takes into account image composition features, yields close to state-of-the-art image aesthetic-based classification accuracies, which highlights the importance of image composition in image aesthetic appeal assessment.

The role of tags and image aesthetics in social image search

In recent years, there has been a proliferation of consumer digital photographs taken and stored in both personal and online repositories. As the amount of user-generated digital photos increases, there is a growing need for efficient ways to search for relevant images to be shared with friends and family. Text-query based search approaches rely heavily on the similarity between the input textual query and the tags added by users to the digital content. Unfortunately, text-query based search results might include a large number of relevant photos, all of them containing very similar tags, but with varying levels of image quality and aesthetic appeal. In this paper we introduce an image re-ranking algorithm that takes into account the aesthetic appeal of the images retrieved by a consumer image sharing site search engine (Googles Picasa Web Album). In order to do so, we extend a state-of-the-art image aesthetic appeal algorithm by incorporating a set of features aimed at consumer photographs. The results of a controlled user study with 37 participants reveal that image aesthetics play a varying role on the selected images depending on the query type and on the user preferences.

Region based image appeal metric for consumer photos

Image appeal may be defined as the interest that a photograph generates when viewed by human observers, incorporating subjective factors on top of the traditional objective quality measures. User studies were conducted in order to identify the right features to use in an image appeal measure; these studies also revealed that a photograph may be appealing even if only a region/area of the photograph is actually appealing. Extensive experimentation helped identify a good set of low level features, which were combined into a set of image appeal metrics. The appealing region detection and the image appeal metrics were validated with experiments conducted on 2000 ground truth images, each graded by three human observers. Two main image appeal metrics are presented: one that ranks the images based on image appeal with a high correlation with the human observations; and a second one which performs better at retrieving highly appealing images from an image collection.

Towards category-based aesthetic models of photographs

We present a novel data-driven category-based approach to automatically assess the aesthetic appeal of photographs. In order to tackle this problem, a novel set of image segmentation methods based on feature contrast are introduced, such that luminance , sharpness , saliency , color chroma , and a measure of region relevance are computed to generate different image partitions. Image aesthetic features are computed on these regions (e.g. sharpness , colorfulness , and a novel set of light exposure features). In addition, color harmony , image simplicity , and a novel set of image composition features are measured on the overall image. Support Vector Regression models are generated for each of 7 popular image categories: animals , architecture , cityscape , floral , landscape , portraiture and seascapes . These models are analyzed to understand which features have greater influence in each of those categories, and how they perform with respect to a generic state of the art model.

Low level features for image appeal measurement

Image appeal may be defined as the interest that a photograph generates when viewed by human observers, incorporating subjective factors on top of the traditional objective quality measures. User studies were conducted in order to identify the right features to use in an image appeal measure; these studies also revealed that a photograph may be appealing even if only a region/area of the photograph is actually appealing. Due to the importance of faces regarding image appeal, a detailed study of a set of face features is also presented, including face size, color and smile detection. Extensive experimentation helped identify a good set of low level features, which are described in depth. These features were optimized using extensive ground truth generated from sets of consumer photos covering all possible appeal levels, by observers with a range of expertise in photography.

Multimodal video copy detection applied to social media

Reliable content-based copy detection algorithms (CBCD) are at the core of effective multimedia data management and copyright enforcement systems. CBCD techniques focus on detecting videos that are identical to or transformed versions of an original video. The fast growth of online video sharing services challenges state-of-the-art copy detection algorithms as they need to be: able to deal with vast amounts of data, computationally efficient and robust to a wide range of image and audio transformations. In this paper, we present two related multimodal CBCD algorithms that effectively fuse audio and video information by means of a compact multimodal signature based on audio and video global descriptors. We validate our algorithms with a benchmark database (MUSCLE-VCD) and obtain over a 14% relative improvement with respect to state-of-the-art systems. In addition, we illustrate the performance of our approach in a video view-count re-ranking task with YouTube data.

Multiresolution color patch extraction

Certain applications require the extraction of patches of color from an image, their size and location. These applications may be: color harmonization algorithms, non-photorealistic rendering, etc. These applications use not too big a palette of colors, and in both cases large areas of homogeneous color are favored along with high detail preserved in the smaller areas with a lot of color activity. The main problem this paper will tackle is to identify the underlying color in an image region, which will be referred to as its underlying color patch, and also try to protect as much as possible the high color activity detail areas. No perfect scene object segmentation is intended in this process, since different objects may be quantized to the same color, the result may be a merged color patch.

Automatic image selection by means of a hierarchical scalable collection representation

This paper presents a system for automatic image selection for storytelling applications, like slideshows and photobooks, where targeting a specific image count is usually of high importance. A versatile image collection representation is introduced, which allows for automatic scalable selection in order to target a specific final image count, while preserving a good coverage of the event in order to maintain the storytelling potential of the selection. A hierarchical time clustering is presented, which is traversed at a specific hierarchy level in order to select images by alternating among all time clusters, and selecting the most relevant images in that cluster. The relevance ordering we use is based on a combination of features, namely, important people, smile detection, image appeal measures, and whether a nearduplicate of the image has already been selected. Once this Hierarchical Scalable Representation has been created, it can be reused to generate any target size selection. Two automatic image selection algorithms have been implemented, one that selects images from clusters with high average image relevance more frequently, and another one that selects images from larger clusters more frequently. The overall system has been used over the last year on several large image collections; the resulting selection was presented to their owners in the form of photo-books in order to get feedback, validating the presented approach.

Image collection taxonomies for photo-book auto-population with intuitive interaction

We demonstrate a system for automatic image selection for photobook creation, along with an intuitive user interface for fine tuning of the selection results. A versatile image collection representation is introduced, which allows for automatic scalable selection in order to target a specific image count for a predetermined size photobook. The images are selected based on their relevance, while preserving a good coverage of the event (time plus people) in order to maintain the storytelling potential of the selection. The selected images are laid out and presented to the user through an Adobe Flex user interface, which allows them to select images and swap them by semantically related ones, in an intuitive manner. The final result is output to a PDF file.