Mari Laine-Hernandez

Image semantics in the description and categorization of journalistic photographs

This paper reports a study on the description and categorization of images. The aim of the study was to evaluate existing indexing frameworks in the context of reportage photographs and to find out how the use of this particular image genre influences the results. The effect of different tasks on image description and categorization was also studied. Subjects performed keywording and free description tasks and the elicited terms were classified using the most extensive one of the reviewed frameworks. Differences were found in the terms used in constrained and unconstrained descriptions. Summarizing terms such as abstract concepts, themes, settings and emotions were used more frequently in keywording than in free description. Free descriptions included more terms referring to locations within the images, people and descriptive terms due to the narrative form the subjects used without prompting. The evaluated framework was found to lack some syntactic and semantic classes present in the data and modifications were suggested. According to the results of this study image categorization is based on high-level interpretive concepts, including affective and abstract themes. The results indicate that image genre influences categorization and keywording modifies and truncates natural image description.

Audiovisual Speech Perception and Eye Gaze Behavior of Adults with Asperger Syndrome

Audiovisual speech perception was studied in adults with Asperger syndrome (AS), by utilizing the McGurk effect, in which conflicting visual articulation alters the perception of heard speech. The AS group perceived the audiovisual stimuli differently from age, sex and IQ matched controls. When a voice saying /p/ was presented with a face articulating /k/, the controls predominantly heard /k/. Instead, the AS group heard /k/ and /t/ with almost equal frequency, but with large differences between individuals. There were no differences in gaze direction or unisensory perception between the AS and control participants that could have contributed to the audiovisual differences. We suggest an explanation in terms of weak support from the motor system for audiovisual speech perception in AS.

Multifaceted image similarity criteria as revealed by sorting tasks

This paper reports a study on the types of image categories constructed from magazine photographs. A novel sorting procedure was tested with the aim of providing more data on image similarity and possible category overlap. Expert and non-expert participants were compared in their categorizations. The new similarity sorting procedure resulted in an average of 67%–111% increase in similarity data gathered compared to basic free sorting. Categories were constructed on various levels of similarity: image Function, main visual content (People, Objects and Scene), conceptual content (Theme) and descriptors (Story, Affective, Description, Photography and Visual). Most categories were based on the theme and people portrayed in the photograph, and in the case of the expert subjects, image function. Also abstract and syntactic similarity criteria were employed by the subjects. The categories created by each subject showed on average a 35%–53% overlap. Participants also demonstrated a tendency to use multiple similarity criteria simultaneously and to combine terms from different levels in a single category name. These results indicate a need for a multifaceted approach in image categorization.

Development and evaluation of a multifaceted magazine image categorization model

The development of visual retrieval methods requires information about user interaction with images, including their description and categorization. This article presents the development of a categorization model for magazine images based on two user studies. In Study 1, we elicited 10 main classes of magazine image categorization criteria through sorting tasks with nonexpert and expert users (N=30). Multivariate methods, namely, multidimensional scaling and hierarchical clustering, were used to analyze similarity data. Content analysis of category names gave rise to classes that were synthesized into a categorization framework. The framework was evaluated in Study 2 by experts (N=24) who categorized another set of images consistent with the framework and found it to be useful in the task. Based on the evaluation study the framework was solidified into a model for categorizing magazine imagery. Connections between classes were analyzed both from the original sorting data and from the evaluation study and included into the final model. The model is a practical categorization tool that may be used in workplaces, such as magazine editorial offices. It may also serve to guide the development of computational methods for image understanding, selection of concepts for automatic detection, and approaches to support browsing and exploratory image search.

The effect of page context on magazine image categorization

Image categorization research has created knowledge on the types of attributes humans use in interpreting image similarity. Little effort has gone into studying the effect of any wider contextual factors on the image groupings people create. This paper reports the results of an investigation on the effect of associated text on magazine image categorization. Image journalism professionals performed a two-phase free sorting of 100 test images and the resulting data were analyzed both qualitatively and quantitatively; using grounded theory methods and hierarchical clustering. The categorization behavior of the two groups was rather similar but there was a statistically significant difference in the types of names given to the categories constructed. Results indicate that having page context available results more likely in descriptions based on overall theme or story of the image. When the context is withheld, people are more prone to describe the people, objects and scenes portrayed in the images, and to combine various categorization criteria. This has implications for the design of interfaces for image archival and retrieval.

Comparison of categorization criteria across image genres

This paper describes a comparison of categorization criteria for three image genres. Two experiments were conducted, where naive participants freely sorted stock photographs and abstract/surreal graphics. The results were compared to a previous study on magazine image categorization. The study also aimed to validate and generalize an existing framework for image categorization. Stock photographs were categorized mostly based on the presence of people, and whether they depicted objects or scenes. For abstract images, visual attributes were used the most. The lightness/darkness of images and their user-evaluated abstractness/representativeness also emerged as important criteria for categorization. We found that image categorization criteria for magazine and stock photographs are fairly similar, while the bases for categorizing abstract images differ more from the former two, most notably in the use of visual sorting criteria. However, according to the results of this study, people tend to use descriptors related to both image content and image production technique and style, as well as to interpret the affective impression of the images in a way that remains constant across image genres. These facets are present in the evaluated categorization framework which was deemed valid for these genres.

Evaluating Local Feature Detectors in Salient Region Detection

In this work, we study local feature extraction methods and evaluate their performance in detecting local features from the salient regions of images. In order to measure the detectors’ performance, we compared the detected regions to gaze fixations obtained from the eye movement recordings of human participants viewing two types of images: natural images (photographs) and abstract/surreal images. The results indicate that all of the six evaluated local feature detectors perform clearly above chance level. The Hessian-Affine detector performs the best and almost reaches the performance level of state-of-the-art saliency detection methods.

What's in a magazine image: Evaluation of a multifaceted categorization model

This poster reports a practitioner evaluation of a multifaceted image categorization model designed for journalistic imagery. We show that the model led to consistent categorizations and was evaluated as useful by the participants in the user test.