Simone Santini

Content-based image retrieval at the end of the early years

Presents a review of 200 references in content-based image retrieval. The paper starts with discussing the working conditions of content-based retrieval: patterns of use, types of pictures, the role of semantics, and the sensory gap. Subsequent sections discuss computational steps for image retrieval systems. Step one of the review is image processing for retrieval sorted by color, texture, and local geometry. Features for retrieval are discussed next, sorted by: accumulative and global features, salient points, object and shape features, signs, and structural combinations thereof. Similarity of pictures and objects in pictures is reviewed for each of the feature types, in close connection to the types and means of feedback the user of the systems is capable of giving by interaction. We briefly discuss aspects of system engineering: databases, system architecture, and evaluation. In the concluding section, we present our view on: the driving force of the field, the heritage from computer vision, the influence on computer vision, the role of similarity and of interaction, the need for databases, the problem of evaluation, and the role of the semantic gap.

Similarity measures

With complex multimedia data, we see the emergence of database systems in which the fundamental operation is similarity assessment. Before database issues can be addressed, it is necessary to give a definition of similarity as an operation. We develop a similarity measure, based on fuzzy logic, that exhibits several features that match experimental findings in humans. The model is dubbed fuzzy feature contrast (FFC) and is an extension to a more general domain of the feature contrast model due to Tversky (1977). We show how the FFC model can be used to model similarity assessment from fuzzy judgment of properties, and we address the use of fuzzy measures to deal with dependencies among the properties.

Integrated browsing and querying for image databases

The image database system El Nino uses a new interaction model that aims to overcome the problem of the semantic gap where the meaning that the user has in mind for an image is at a higher semantic level than the features on which the database operates. To solve this problem, we replaced the usual query paradigm with a more active exploration process and developed an interface based on these premises.

The cell-centered database: a database for multiscale structural and protein localization data from light and electron microscopy.

The creation of structured shared data repositories for molecular data in the form of web-accessible databases like GenBank has been a driving force behind the genomic revolution. These resources serve not only to organize and manage molecular data being created by researchers around the globe, but also provide the starting point for data mining operations to uncover interesting information present in the large amount of sequence and structural data. To realize the full impact of the genomic and proteomic efforts of the last decade, similar resources are needed for structural and biochemical complexity in biological systems beyond the molecular level, where proteins and macromolecular complexes are situated within their cellular and tissue environments. In this review, we discuss our efforts in the development of neuroinformatics resources for managing and mining cell level imaging data derived from light and electron microscopy. We describe the main features of our web-accessible database, the Cell Centered Database (CCDB; http://ncmir.ucsd.edu/CCDB/), designed for structural and protein localization information at scales ranging from large expanses of tissue to cellular microdomains with their associated macromolecular constituents. The CCDB was created to make 3D microscopic imaging data available to the scientific community and to serve as a resource for investigating structural and macromolecular complexity of cells and tissues, particularly in the rodent nervous system.

Beyond query by example

This paper considers some of the problems we found trying to extract meaning from images in database applications, and proposes some ways to solve them. We argue that the meaning of an image is an ill-defined entity, and it is not in general possible to derive from an image the meaning that the user of the database wants. Rather, we should be content with a correlation between the intended meaning and simple perceptual clues that databases can extract. Rather than working on the impossible task of extracting unambiguous meaning from images, we should provide the user with the tools he needs to drive the database in the areas of the feature space where “interesting” images are. 1 Meaningless Responses Try to remember the last time you used a web demo to experiment with an image database. Most—if not all— database demos have very similar interfaces. On one side you form a query, either drawing a sketch or selecting one of a set of random images that the database gives you. While doing this, you also decide one or more similarity criteria on which you want to base you query. Typical choices are color, structure, or texture [2]. Sometimes you can use several criteria together [3], or you can draw areas and decide that the spatial relations between these areas is relevant to your query. On the other (metaphorical) half of your interface you have a browser. You hit the “go” button and—after a certain time—the browser will display the first n (typically 9 to 15) images in the database in order of similarity with the query. If you are like us, you will look at the results with mixed feelings. One or two of them will make perfect sense; some others will not. Consider Fig. 1. This is the result of a query done with one of the standard image database engines currently available. (The image on the top left corner is the query.) Some of the images returned are somewhat ∗This work was partially supported by the National Science Foundation under grant NSF-IRI-9610518 Figure 1: disappointing. Yet, looking at them closely, it is possible to understand in most cases why the database returned them. The head of the woman in the third image resembles superficially the shape of the arch on top of the door in the query. The woman in the fourth image wears a vest similar in color to the door, and so on. We can explain in this way most of the images returned. Still, you are not happy. The fact is that you asked for an image similar to a door, and received images that semantically were not doors at all. In this paper, we propose and support the following explaination: determining the meaning of an image is an inherently ill posed problem, since it depends on the “situatedness” of the observer as well as on the image data; we can however find a correlation between reasonable interpretations of an image and the simple perceptual clues that a database can use. With the use of the right interface, the user can interact with the database to “drive it” in interesting regions of the feature space. There were problems in the example, essentially, because images can be similar at many different levels. Two images can be similar because they have the same dominant colors, because they are both paintings by Brügel the elder, because they both convey a sense of calm, because they both represent an old man and a dog, and so on. The database can interpret and use only some of these possible similarities, based on very primitive semantics. This results in breakdown [11]. The computer system operated in accordance to its own semantic categories. The breakdown occurs because at first the system gave the illusion to operate according to the same categories as the user (i.e. some of the image returned are actually doors), and when these are violated, the user experiences frustration. If we assume that we have no annotations, and that the domain of the database is not overly restricted, it is virtually impossible to work with models with high semantic content. In our work we intentionally reject any form of object identification or region segmentation, and decide to rely on simple perceptual clues. Users, on the other hand, reason on a different semantic level—one in which objects, and not perceptual clues are the main concern. A flexible, perceptual approach can be useful only if there is a correlation between the two levels that is, if the perceptual level can provide information about the semantic categories of interest to the user. We emphasized the word “correlation” to stress the fact that we don’t look for an exact correspondence. That would be tantamount to doing unconstrained object recognition: an effort bound to incur in the frame problem [6, 1]. We argue that for the retrieval problem a correlation is enough. The user will provide the missing information and the situatedness to drive the system towards the right images. 2 Where did semantics go? What the user wants from a database is a semantically meaningful answer to a query. If we refuse any symbolic representation of semantic categories in favor of a more perceptual approach, we should ask whether this approach can be useful for the queries we have in mind. In absolute terms, the answer is no, as testified by the terrible difficulty of building an autonomous agent driven by vision. Our goal, however, is not to build a machine that sees, but to build a machine that will assist a user in activity that require perception. Our refusal of the anthropomorphic paradigm has shifted the problem significantly. The relevant question that we need to answer now is no longer “can simple perceptual clues identify semantically meaningful objects,” but: is there enough correlation between simple perceptual clues and semantically meaningful objects so that the interaction between the user and the system will be meaningful? The answer to that question is not unique. It not only depends on the similarity measurement, but on the whole system. In particular, in a social system, as opposed to an anthropomorphic one, we cannot ignore the role of the interface. Consider two queries, which are presented to the database as “query-by-example.” The first “Apple” query uses the image of Fig. 2.a, the second “Cat” query uses the image of Fig. 2.b. We submitted the queries to the similarity measurement system in [9]. How well does our simple perceptual engine capture some of the possible meanings associated to the images in Fig. 2? Let us start with the apple query: some possible interpretations for the image of Fig. 2.a are that it represents a fruit, a red object, a round object, and an apple. Fig. 3 shows the percentage of the first k images returned by the database that have these four semantics. These four semantic interpretations of the image are usually considered at very different “levels.” Color is considered a very “low level,” or perceptual attribute, while being a fruit or an apple is a cognitive attribute. The fact that a significant percentage of results returned by the database are in effect fruits or apples, indicates a correlation between perceptual and cognitive semantics that our system can exploit. Figure 2: The referent of the apple query (a) and the cat query (b).

Similarity queries in image databases

Query-by-content image database will be based on similarity, rather than on matching, where similarity is a measure that is defined and meaningful for every pair of images in the image space. Since it is the human user that, in the end, has to be satisfied with the results of the query, it is natural to base the similarity measure that we will use on the characteristics of human similarity assessment. In the first part of this paper, we review some of these characteristics and define a similarity measure based on them. Another problem that similarity-based databases will have to face is how to combine different queries into a single complex query. We present a solution based on three operators that are the analogous of the and, or, and not operators one uses in traditional databases. These operators are powerful enough to express queries of unlimited complexity, yet have a very intuitive behavior, making easy for the user to specify a query tailored to a particular need.

Similarity is a Geometer

Multimedia databases (in particular image databases) are different from traditional system since they cannot ignore the perceptual substratum on which the data come. There are several consequences of this fact. The most relevant for our purposes is that it is no longer possible to identify a well defined meaning of an image and, therefore, matching based on meaning is impossible. Matching should be replaced by similarity assessment and, in particular, by something close to human preattentive similarity.In this paper we propose a geometric model of similarity measurement that subsumes most of the models proposed for psychological similarity.

Image retrieval by shape and texture

Abstract Effective image retrieval by content from database requires that visual image properties are used instead of textual labels to recover pictorial data. Retrieval by image similarity given a template image is particularly challenging. The difficulty is to derive a similarity measure that combines shape, grey level patterns and texture in a way that closely conforms to human perception. In this paper a system is presented which supports retrieval by image similarity based on elastic template matching. The template can be both a 1D template modeling the contour of an object, and a 2D template modeling a part of an image with a significant grey level pattern. The retrieval process is obtained as a continuous interaction by which the original query of the user can be refined or changed on the basis of the results provided by the system.

The Graphical Specification of Similarity Queries

Abstract Image databases will require a completely new organization due to the unstructured and ‘perceptual’ structure of the data they contain. We argue that similarity measures, rather than matching, will be the organizing principle of image databases. Similarity is a very elusive and complex judgment, and typical databases will have to rely on a number of different metrics to satisfy the different needs of their users. This poses the problem of how to combine different similarity measures in a coherent and intuitive way. In this paper we propose our solution, which is loosely based on ideas derived from fuzzy logic in that it uses the equivalent in the similarity domain of the and, or and not operations. The approach is much more general than that, however, and can be adapted to work with any operation that combines together similarity judgment. With this approach, a query can be described as a Directional Acyclic graph with certain properties. We analyse briefly the properties of this graph, and we present the interface we are developing to specify these queries.

Analysis of traffic flow in urban areas using web cameras

With the development of the Internet and the omnipresence of video cameras, the amount of visual information available to individuals with access to the world wide web has increased by orders of magnitude in the last few years. Due to cost and communication constraints, this information is usually of rather low quality. The cooccurrence of a large number of sensors and the low quality of every single sensor has the potential to create a new episteme in computer vision. This paper presents an example of how certain limitations of a single sensor can be overcome by using the sensor multiplicity. Using a number of web cameras available in the Seattle area, the paper performs first a simple qualitative traffic analysis from each single camera. Since the cameras provide images at a very low rate (1 image every 2 minutes), standard techniques based on motion detection are unusable, and the paper proposes a simple approach based on image variance. Then, the data are integrated using the structure of the Seattle highway system and network tomography to determine the major flows of traffic at different times of the day.