Mathias Eitz

Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors

We introduce a benchmark for evaluating the performance of large-scale sketch-based image retrieval systems. The necessary data are acquired in a controlled user study where subjects rate how well given sketch/image pairs match. We suggest how to use the data for evaluating the performance of sketch-based image retrieval systems. The benchmark data as well as the large image database are made publicly available for further studies of this type. Furthermore, we develop new descriptors based on the bag-of-features approach and use the benchmark to demonstrate that they significantly outperform other descriptors in the literature.

Sketch-based shape retrieval

We develop a system for 3D object retrieval based on sketched feature lines as input. For objective evaluation, we collect a large number of query sketches from human users that are related to an existing data base of objects. The sketches turn out to be generally quite abstract with large local and global deviations from the original shape. Based on this observation, we decide to use a bag-of-features approach over computer generated line drawings of the objects. We develop a targeted feature transform based on Gabor filters for this system. We can show objectively that this transform is better suited than other approaches from the literature developed for similar tasks. Moreover, we demonstrate how to optimize the parameters of our, as well as other approaches, based on the gathered sketches. In the resulting comparison, our approach is significantly better than any other system described so far.

Technical Section: An evaluation of descriptors for large-scale image retrieval from sketched feature lines

We address the problem of fast, large scale sketch-based image retrieval, searching in a database of over one million images. We show that current retrieval methods do not scale well towards large databases in the context of interactively supervised search and propose two different approaches for which we objectively evaluate that they significantly outperform existing approaches. The proposed descriptors are constructed such that both the full color image and the sketch undergo exactly the same preprocessing steps. We first search for an image with similar structure, analyzing gradient orientations. Then, best matching images are clustered based on dominant color distributions, to offset the lack of color-based decision during the initial search. Overall, the query results demonstrate that the system offers intuitive access to large image databases using a user-friendly sketch-and-browse interface.

A descriptor for large scale image retrieval based on sketched feature lines

We address the problem of large scale sketch based image retrieval, searching in a database of over a million images. The search is based on a descriptor that elegantly addresses the asymmetry between the binary user sketch on the one hand and the full color image on the other hand. The proposed descriptor is constructed such that both the full color image and the sketch undergo exactly the same preprocessing steps. We also design an adapted version of the descriptor proposed for MPEG-7 and compare their performance on a database of 1.5 million images. Best matching images are clustered based on color histograms, to offset the lacking color in the query. Overall, the query results demonstrate that the system allows users an intuitive access to large image databases.

PhotoSketch: a sketch based image query and compositing system

We introduce a system for progressively creating images through a simple sketching and compositing interface. A large database of over 1.5 million images is searched for matches to a users binary outline sketch; the results of this search can be combined interactively to synthesize the desired image. We introduce image descriptors for the task of estimating the difference between images and binary outline sketches. The compositing part is based on graph cut and Poisson blending. We demonstrate that the resulting system allows generating complex images in an intuitive way.

SHREC'12 track: sketch-based 3D shape retrieval

Sketch-based 3D shape retrieval has become an important research topic in content-based 3D object retrieval. The aim of this track is to measure and compare the performance of sketch-based 3D shape retrieval methods implemented by different participants over the world. The track is based on a new sketch-based 3D shape benchmark, which contains two types of sketch queries and two versions of target 3D models. In this track, 7 runs have been submitted by 5 groups and their retrieval accuracies were evaluated using 7 commonly used retrieval performance metrics. We hope that the benchmark, its corresponding evaluation code, and the comparative evaluation results of the state-of-the-art sketch-based 3D model retrieval algorithms will contribute to the progress of this research direction for the 3D model retrieval community.

Hierarchical Spatial Hashing for Real-time Collision Detection

We present a new, efficient and easy to use collision detection scheme for real-time collision detection between highly deformable tetrahedral models. Tetrahedral models are a common representation of volumetric meshes which are often used in physically based simulations, e.g. in Virtual surgery. In a deformable models environment collision detection usually is a performance bottleneck since the data structures used for efficient intersection tests need to be rebuilt or modified frequently. Our approach minimizes the time needed for building a collision detection data structure. We employ an infinite hierarchical spatial grid in which for each single tetrahedron in the scene a well fitting grid cell size is computed. A hash function is used to project occupied grid cells into a finite ID hash table. Only primitives mapped to the same hash index indicate a possible collision and need to be checked for intersections. This results in a high performance collision detection algorithm which does not depend on user defined parameters and thus flexibly adapts to any scene setup.

Sketch-based 3D shape retrieval

As large collections of 3D models are starting to become as common as public image collections, the need arises to quickly locate models in such collections. Models are often insufficiently annotated such that a keyword based search is not promising. Our approach for content based searching of 3D models relies entirely on visual analysis and is based on the observation that a large part of our perception of shapes stems from their salient features, usually captured by dominant lines in their display. Recent research on such feature lines has shown that 1) people mostly draw the same lines when asked to depict a certain model and 2) the shape of an object is well represented by the set of feature lines generated by recent NPR line drawing algorithms [Cole et al. 2009]. Consequently, we suggest an image based approach for 3D shape retrieval, exploiting the similarity of human sketches and the results of current line drawing algorithms. Our search engine takes a sketch of the desired model drawn by a user as the input and compares this sketch to a set of line drawings automatically generated for each of the models in the collection.

Interactively browsing large image collections

Manually locating an image in a large collection has become infeasible with the recent rapid growth in size of such collections. Nowadays, even private collections easily contain tens of thousands of images; public collections have long passed the billion images mark. Current approaches for finding images in large collections, therefore, try to confine the set of images by returning only those images that correspond to certain properties defined by a query. Such properties can include: keywords, semantic information associated with the images, similarity to an example image, a rough sketch of the desired outlines, or any combination thereof.