J. M. Corridoni

Image retrieval by color semantics

Abstract. The development of a system supporting querying of image databases by color content tackles a major design choice about properties of colors which are referenced within user queries. On the one hand, low-level properties directly reflect numerical features and concepts tied to the machine representation of color information. On the other hand, high-level properties address concepts such as the perceptual quality of colors and the sensations that they convey. Color-induced sensations include warmth, accordance or contrast, harmony, excitement, depression, anguish, etc. In other words, they refer to the semantics of color usage. In particular, paintings are an example where the message is contained more in the high-level color qualities and spatial arrangements than in the physical properties of colors. Starting from this observation, Johannes Itten introduced a formalism to analyze the use of color in art and the effects that this induces on the users psyche. In this paper, we present a system which translates the Itten theory into a formal language that expresses the semantics associated with the combination of chromatic properties of color images. The system exploits a competitive learning technique to segment images into regions with homogeneous colors. Fuzzy sets are used to represent low-level region properties such as hue, saturation, luminance, warmth, size and position. A formal language and a set of model-checking rules are implemented to define semantic clauses and verify the degree of truth by which they hold over an image.

Image retrieval by color semantics with incomplete knowledge

Retrieval by content from image databases faces the distance between low-level syntactic features that can be automatically detected by conventional image processing tools and high level semantics which captures users filtering intentions. A system is presented which bridges this gap by resorting to a theory formulated by Johannes Itten in 1960, and widely accepted in the community of fine arts, to support objective interpretation of color arrangements over paintings. The system relies upon a schema distinguishing archiving, querying, and retrieval stages. In the archiving stage, images are associated with a description capturing the spatial arrangement of regions with homogeneous chromatic attributes, as detected by the use of an automatic image processing tool. Imprecise descriptions are supported through the adoption of a hierarchical index providing a multi-resolution representation of image contents. In the querying stage, a visual iconic language allows the expression of sentences about chromatic contents in accordance with a high-level semantic model of colors combinations. By permitting flexible expression of abstract, non-literal, properties, the model supports intentional vagueness and incompleteness in the specification of searching queries. In the retrieval stage, a similarity score is introduced, which accounts for the degree with which a query assertion applies to a given image. The measure of similarity drives the traversal of the hierarchical index up to find the minimum level of description precision, permitting a definite decision about the satisfaction of the query on each stored image.

Retrieval of paintings using effects induced by color features

Image databases are now a subject of increasing attention in multimedia, for archiving and retrieval of images in the fields of art, history, medicine and industry, among others. From the psychological point of view, color perception is related to several factors including color features (brightness, chromaticity and saturation), surrounding colors, color spatial organization, observer memory/knowledge/experience etc. Paintings are an example where the message is contained more in the high-level color qualities and spatial arrangements than in the physical properties of colors. Starting from this observation, Johannes Itten (1961) introduced a formalism to analyze the use of color in art and the effects that this induces on the users psyche. We present a system which translates the Itten theory into a formal language that allows to express the semantics associated with the combination of chromatic properties of color images. Fuzzy sets are used to represent low-level region properties. A formal language and a set of model checking rules are implemented that allow to define semantic clauses and verify the degree of truth by which they hold over an image.

Structured digital video indexing

This paper addresses the problems of film segmentation into its syntactic elements (the shots) and of their aggregation into semantic sets under constrained conditions: the shot/reverse-shot scenes (SRS). It then addresses the problem of describing the dynamic content of each shot through the analysis of camera motion. New techniques for edit effect detection and classification are herein proposed. A new algorithm is thereby introduced for automatic inferring the belonging of a set of shots to the same SRS scene. Finally, an algorithm is presented, which segments a sequence into regions having different motion types and extracts the camera motion. Extended performance analysis of the algorithms has been made on about 20 hours movies.

Multi-perspective Navigation of Movies

Abstract Digital libraries of video are rapidly growing in size and availability through digital networks of computers. The explosion of digital unstructure information, such as video, traveling through the network raises the need for tools that allow video information structuring and “intelligent” access to sequences. In this paper we present a novel video database system, which has been expressly designed to support structured storage of movies. A semiotic formalization for movie logical organization is here introduced and exploited to derive the conceptual schema of an object-oriented movie database. This structure is expressly tailored to enable queries on film technical features and on semantic contents as well, thus allowing feature fusion and access from multiple perspectives. A novel visual interaction scheme is implemented that allows users to formulate queries as filtering operations, that enable focussing on the interesting part of information, while skipping the rest. Formalization of the operators of “perspective” and “filtering” is thereby supplied. Retrieval of stored sequences by iconic motion example is also implemented for accessing sequences by contents. Examples are supplied of interaction with the database and of expressive power of visual query language.

A visual language for color-based painting retrieval

The availability of large image databases is emphasizing the relevance of filters, which permit one to focus on a small subset of data. Visual specification of such filters provides a natural way to express content-oriented queries. To support a high power of expression, an original visual language is proposed for the symbolic representation of the semantics induced by the color quality and arrangement over a painting. The proposed language is based on the theory of color semantics introduced by artists in the 20th Century, and is developed to support a visual querying paradigm. This paper formalizes the grammar of the language in its visual form and describes it implementation in a prototype system of painting retrieval by color content.

3D object classification using multi-object kohonen networks

The problem of three-dimensional planar-faced opaque object recognition from one single perspective view, is addressed in this paper. A view parameterization, based on Hough Transform, is described. Classification is accomplished by multiple multi-object Kohonen networks. A new object grouping criterion has been investigated to assign disjoint subsets of objects to each Kohonen network of the system, on the basis of input space topology. Recognition tests are presented on both synthetic and real-world 3D objects, even in the presence of partial occlusion.

Film Editing Reconstruction and Semantic Analysis

In the present paper the problem of video segmentation and semantic analysis for cinematographic language classification is addressed. Video analysis is here focused on the automatic detection of editing procedures, through which the shots are fused into the video stream. This work proposes new data-driven algorithms to detect cuts between two shots, and special editing artifacts, as well. The second part of this work is devoted to a higher level analysis, which involves the semantics of a film. On the basis of the common film editing rules, a method is presented to detect sets of shots having unitary content. Experimental validation of the techniques proposed is thereby presented.

Film semantic analysis

In the paper the problem of video segmentation and semantic analysis for video indexing and cinematographic language classification is addressed. Large archives of video sequences will remain a largely unexploitable resource of information, unless a straightforward way to describe their content is designed. The first step to be considered in this direction is the highlighting of the syntactic units of the visual speech. Therefore, video segmentation must focus on the editing procedures, through which the shots are fused into the video stream. This work proposes a statistically based new data-driven algorithm to detect cuts between two shots in a video sequence. On the contrary, the analysis of how the other editing techniques are performed, supplies a mathematical model, upon which the algorithms for the detection of fades, dissolves and mattes are based. A second part of this work is devoted to a higher level analysis, which involves the semantics of a film. On the basis of the common film editing rules, a method is presented to detect when a set of shots has unitary meaning. Experimental validation of the techniques proposed is thereby presented.

Image query by semantical color content

The availability of large image databases is emphasizing the relevance of filters, which permit to focus the interest on a small subset of data. Taking advantage of the pictorial features of images, visual specification of such filters provides a powerful and natural way to express content-oriented queries. Albeit direct, the by example paradigm, does not allow to express high-level assertions on the pictorial content of images and specifically, paintings. To support the visuality, without losing power of expression, an original visual language is herein proposed, for the symbolic representation of the semantics induced by the colour quality and arrangement over a painting. The proposed language is based on the concepts of color semantics introduced by artists in the 900 and is developed to support a visual query paradigm. The present paper formalizes the grammar of the language and describes its implementation into a prototype system of painting retrieval by colour content.