Stina Westman

Information interaction in 140 characters or less: genres on twitter

In this paper, we describe a genre analysis of Twitter updates, commonly called tweets. The aim was to understand and characterize the communication supported by Twitter in a structured manner enabled by the genre concept. We analyzed six facets of Twitter genres: who, what, where, when, why, and how, and identified a set of five common Twitter genres.

Search strategies in multimodal image retrieval

This paper reports on a study on search strategies in multimodal image retrieval. We analyzed the queries and search tactics employed by image journalism professionals and non-professionals in a user test. Transaction log data show that searchers are able to combine up to four query modes into a query. Most queries combined at least two of the modes (text, color, sketch, quality, and category). Task type was shown to affect the choice of which modes to employ. Known item and data search tasks led to queries combining text, color and category modes. Visually cued tasks resulted in searches combining several content-based and textual modes. Conceptual tasks led to a large number of queries by text and category only. User background also significantly affected the types of queries constructed. Professionals used the color mode more whereas non-professionals drew more sketches. Non-professionals were more likely to switch query modes whereas professionals edited the content of their queries. We described the search processes with Markov models and maximal repeating patterns. Common patterns and probable transitions dealt with querying, inspecting result images and saving them into the workspace or iterating queries of the same type. The results indicate a need to support multimodal image query formulation.

Image semantics in the description and categorization of journalistic photographs

This paper reports a study on the description and categorization of images. The aim of the study was to evaluate existing indexing frameworks in the context of reportage photographs and to find out how the use of this particular image genre influences the results. The effect of different tasks on image description and categorization was also studied. Subjects performed keywording and free description tasks and the elicited terms were classified using the most extensive one of the reviewed frameworks. Differences were found in the terms used in constrained and unconstrained descriptions. Summarizing terms such as abstract concepts, themes, settings and emotions were used more frequently in keywording than in free description. Free descriptions included more terms referring to locations within the images, people and descriptive terms due to the narrative form the subjects used without prompting. The evaluated framework was found to lack some syntactic and semantic classes present in the data and modifications were suggested. According to the results of this study image categorization is based on high-level interpretive concepts, including affective and abstract themes. The results indicate that image genre influences categorization and keywording modifies and truncates natural image description.

Naturalness and interestingness of test images for visual quality evaluation

Balanced and representative test images are needed to study perceived visual quality in various application domains. This study investigates naturalness and interestingness as image quality attributes in the context of test images. Taking a top-down approach we aim to find the dimensions which constitute naturalness and interestingness in test images and the relationship between these high-level quality attributes. We compare existing collections of test images (e.g. Sony sRGB images, ISO 12640 images, Kodak images, Nokia images and test images developed within our group) in an experiment combining quality sorting and structured interviews. Based on the data gathered we analyze the viewer-supplied criteria for naturalness and interestingness across image types, quality levels and judges. This study advances our understanding of subjective image quality criteria and enables the validation of current test images, furthering their development.

Content-Based Prediction of Movie Style, Aesthetics, and Affect: Data Set and Baseline Experiments

The affective content of a movie is often considered to be largely determined by its style and aesthetics. Recently, studies have attempted to estimate affective movie content with computational features, but results have been mixed, one of the main reasons being a lack of data on perceptual stylistic and aesthetic attributes of film, which would provide a ground truth for the features. The distinctions between energetic and tense arousal as well as perceived and felt affect are also often neglected. In this study, we present a data set of ratings by 73 viewers of 83 stylistic, aesthetic, and affective attributes for a selection of movie clips containing complete scenes taken from mainstream movies. The affective attributes include the temporal progression of perceived and felt valence and arousal within the clips. The data set is aimed to be used to train algorithms that predict viewer assessments based on low-level computational features. With this data set, we performed a baseline study modeling the relation between a large selection of low-level computational features (i.e., visual, auditory, and temporal) and perceptual stylistic, aesthetic, and affective attributes of movie clips. Two algorithms were compared in a realistic prediction scenario: linear regression and the neural-network-based Extreme Learning Machine (ELM). Felt and perceived affect as well as stylistic attributes were shown to be equally easy to predict, whereas the prediction of aesthetic attributes failed. The performance of the ELM predictor was overall found to be slightly better than the linear regression. A feature selection experiment illustrated that features from all low-level computational modalities, visual, auditory and temporal, contribute to the prediction of the affect assessments. We have made our assessment data and extracted computational features publicly available.

Multifaceted image similarity criteria as revealed by sorting tasks

This paper reports a study on the types of image categories constructed from magazine photographs. A novel sorting procedure was tested with the aim of providing more data on image similarity and possible category overlap. Expert and non-expert participants were compared in their categorizations. The new similarity sorting procedure resulted in an average of 67%–111% increase in similarity data gathered compared to basic free sorting. Categories were constructed on various levels of similarity: image Function, main visual content (People, Objects and Scene), conceptual content (Theme) and descriptors (Story, Affective, Description, Photography and Visual). Most categories were based on the theme and people portrayed in the photograph, and in the case of the expert subjects, image function. Also abstract and syntactic similarity criteria were employed by the subjects. The categories created by each subject showed on average a 35%–53% overlap. Participants also demonstrated a tendency to use multiple similarity criteria simultaneously and to combine terms from different levels in a single category name. These results indicate a need for a multifaceted approach in image categorization.

Development and evaluation of a multifaceted magazine image categorization model

The development of visual retrieval methods requires information about user interaction with images, including their description and categorization. This article presents the development of a categorization model for magazine images based on two user studies. In Study 1, we elicited 10 main classes of magazine image categorization criteria through sorting tasks with nonexpert and expert users (N=30). Multivariate methods, namely, multidimensional scaling and hierarchical clustering, were used to analyze similarity data. Content analysis of category names gave rise to classes that were synthesized into a categorization framework. The framework was evaluated in Study 2 by experts (N=24) who categorized another set of images consistent with the framework and found it to be useful in the task. Based on the evaluation study the framework was solidified into a model for categorizing magazine imagery. Connections between classes were analyzed both from the original sorting data and from the evaluation study and included into the final model. The model is a practical categorization tool that may be used in workplaces, such as magazine editorial offices. It may also serve to guide the development of computational methods for image understanding, selection of concepts for automatic detection, and approaches to support browsing and exploratory image search.

The effect of page context on magazine image categorization

Image categorization research has created knowledge on the types of attributes humans use in interpreting image similarity. Little effort has gone into studying the effect of any wider contextual factors on the image groupings people create. This paper reports the results of an investigation on the effect of associated text on magazine image categorization. Image journalism professionals performed a two-phase free sorting of 100 test images and the resulting data were analyzed both qualitatively and quantitatively; using grounded theory methods and hierarchical clustering. The categorization behavior of the two groups was rather similar but there was a statistically significant difference in the types of names given to the categories constructed. Results indicate that having page context available results more likely in descriptions based on overall theme or story of the image. When the context is withheld, people are more prone to describe the people, objects and scenes portrayed in the images, and to combine various categorization criteria. This has implications for the design of interfaces for image archival and retrieval.

Stylistic Features for Affect-Based Movie Recommendations

In recent years, studies have estimated affective movie content computationally with stylistic features, yet knowledge of the perceptual relation between style and affect remains scarce. Such knowledge would be useful in affect-based movie recommendation systems. To this end, a user study was conducted in which seventy-three participants with varying levels of film expertise rated movie clips according to 13 stylistic features in three modalities (visual, aural and temporal) as well as perceived and felt affect in three dimensions (hedonic tone, energetic arousal and tense arousal). Style-based linear regression models were then constructed for each affect dimension. Visual features contributed the most to hedonic tone and tense arousal, and temporal features to energetic arousal. Also, perceived affect showed greater inter-rater agreement and better modeling performance than felt affect. The results indicate that the influence of specific stylistic features on affect varies by dimension and by whether the affect is perceived or felt.

Comparison of categorization criteria across image genres

This paper describes a comparison of categorization criteria for three image genres. Two experiments were conducted, where naive participants freely sorted stock photographs and abstract/surreal graphics. The results were compared to a previous study on magazine image categorization. The study also aimed to validate and generalize an existing framework for image categorization. Stock photographs were categorized mostly based on the presence of people, and whether they depicted objects or scenes. For abstract images, visual attributes were used the most. The lightness/darkness of images and their user-evaluated abstractness/representativeness also emerged as important criteria for categorization. We found that image categorization criteria for magazine and stock photographs are fairly similar, while the bases for categorizing abstract images differ more from the former two, most notably in the use of visual sorting criteria. However, according to the results of this study, people tend to use descriptors related to both image content and image production technique and style, as well as to interpret the affective impression of the images in a way that remains constant across image genres. These facets are present in the evaluated categorization framework which was deemed valid for these genres.