M. Fatih Demirci

Object Recognition as Many-to-Many Feature Matching

Object recognition can be formulated as matching image features to model features. When recognition is exemplar-based, feature correspondence is one-to-one. However, segmentation errors, articulation, scale difference, and within-class deformation can yield image and model features which don’t match one-to-one but rather many-to-many. Adopting a graph-based representation of a set of features, we present a matching algorithm that establishes many-to-many correspondences between the nodes of two noisy, vertex-labeled weighted graphs. Our approach reduces the problem of many-to-many matching of weighted graphs to that of many-to-many matching of weighted point sets in a normed vector space. This is accomplished by embedding the initial weighted graphs into a normed vector space with low distortion using a novel embedding technique based on a spherical encoding of graph structure. Many-to-many vector correspondences established by the Earth Mover’s Distance framework are mapped back into many-to-many correspondences between graph nodes. Empirical evaluation of the algorithm on an extensive set of recognition trials, including a comparison with two competing graph matching approaches, demonstrates both the robustness and efficacy of the overall approach.

Many-to-many feature matching using spherical coding of directed graphs

In recent work, we presented a framework for many-to-many matching of multi-scale feature hierarchies, in which features and their relations were captured in a vertex-labeled, edge-weighted directed graph. The algorithm was based on a metric-tree representation of labeled graphs and their metric embedding into normed vector spaces, using the embedding algorithm of Matousek [13]. However, the method was limited by the fact that two graphs to be matched were typically embedded into vector spaces with different dimensionality. Before the embeddings could be matched, a dimensionality reduction technique (PCA) was required, which was both costly and prone to error. In this paper, we introduce a more efficient embedding procedure based on a spherical coding of directed graphs. The advantage of this novel embedding technique is that it prescribes a single vector space into which both graphs are embedded. This reduces the problem of directed graph matching to the problem of geometric point matching, for which efficient many-to-many matching algorithms exist, such as the Earth Mover’s Distance. We apply the approach to the problem of multi-scale, view-based object recognition, in which an image is decomposed into a set of blobs and ridges with automatic scale selection.

The representation and matching of categorical shape

We present a framework for categorical shape recognition. The coarse shape of an object is captured by a multiscale blob decomposition, representing the compact and elongated parts of an object at appropriate scales. These parts, in turn, map to nodes in a directed acyclic graph, in which edges encode both semantic relations (parent/child) as well as geometric relations. Given two image descriptions, each represented as a directed acyclic graph, we draw on spectral graph theory to derive a new algorithm for computing node correspondence in the presence of noise and occlusion. In computing correspondence, the similarity of two nodes is a function of their topological (graph) contexts, their geometric (relational) contexts, and their node contents. We demonstrate the approach on the domain of view-based 3-D object recognition.

Indexing through laplacian spectra

With ever growing databases containing multimedia data, indexing has become a necessity to avoid a linear search. We propose a novel technique for indexing multimedia databases in which entries can be represented as graph structures. In our method, the topological structure of a graph as well as that of its subgraphs are represented as vectors whose components correspond to the sorted laplacian eigenvalues of the graph or subgraphs. Given the laplacian spectrum of graph G, we draw from recently developed techniques in the field of spectral integral variation to generate the laplacian spectrum of graph G+e without computing its eigendecomposition, where G+e is a graph obtained by adding edge e to graph G. This process improves the performance of the system for generating the subgraph signatures for 1.8% and 6.5% in datasets of size 420 and 1400, respectively. By doing a nearest neighbor search around the query spectra, similar but not necessarily isomorphic graphs are retrieved. Given a query graph, a voting schema ranks database graphs into an indexing hypothesis to which a final matching process can be applied. The novelties of the proposed method come from the powerful representation of the graph topology and successfully adopting the concept of spectral integral variation in an indexing algorithm. To examine the fitness of the new indexing framework, we have performed a number of experiments using an extensive set of recognition trials in the domain of 2D and 3D object recognition. The experiments, including a comparison with a competing indexing method using two different graph-based object representations, demonstrate both the robustness and efficacy of the overall approach.

The Representation and Matching of Images Using Top Points

In previous work, singular points (or top points) in the scale space representation of generic images have proven valuable for image matching. In this paper, we propose a novel construction that encodes the scale space description of top points in the form of a directed acyclic graph. This representation allows us to utilize coarse-to-fine graph matching algorithms for comparing images represented in terms of top point configurations instead of using solely the top points and their features in a point matching algorithm, as was done previously. The nodes of the graph represent the critical paths together with their top points. The edge set captures the neighborhood distribution of vertices in scale space, and is constructed through a hierarchical tessellation of scale space using a Delaunay triangulation of the top points. We present a coarse-to-fine many-to-many matching algorithm for comparing such graph-based representations. The algorithm is based on a metric-tree representation of labeled graphs and their low-distortion embeddings into normed vector spaces via spherical encoding. This is a two-step transformation that reduces the matching problem to that of computing a distribution-based distance measure between two such embeddings. To evaluate the quality of our representation, four sets of experiments are performed. First, the stability of this representation under Gaussian noise of increasing magnitude is examined. Second, a series of recognition experiments is run on a face database. Third, a set of clutter and occlusion experiments is performed to measure the robustness of the algorithm. Fourth, the algorithm is compared to a leading interest point-based framework in an object recognition experiment.

Many-to-many matching of scale-space feature hierarchies using metric embedding

Scale-space feature hierarchies can be conveniently represented as graphs, in which edges are directed from coarser features to finer features. Consequently, feature matching (or view-based object matching) can be formulated as graph matching. Most approaches to graph matching assume a one-to-one correspondence between nodes (features) which, due to noise, scale discretization, and feature extraction errors, is overly restrictive. In general, a subset of features in one hierarchy, representing an abstraction of those features, may best match a subset of features in another. We present a framework for the many-to-many matching of multi-scale feature hierarchies, in which features and their relations are captured in a vertex-labeled, edge-weighted graph. The matching algorithm is based on a metric-tree representation of labeled graphs and their low-distortion metric embedding into normed vector spaces. This two-step transformation reduces the many-to-many graph matching problem to that of computing a distribution-based distance measure between two such embeddings. To compute the distance between two sets of embedded, weighted vectors, we use the Earth Movers Distance under transformation. To demonstrate the approach, we target the domain of multi-scale, qualitative shape description, in which an image is decomposed into a set of blobs and ridges with automatic scale selection. We conduct an extensive set of view-based matching trials, and compare the results favorably to matching under a one-to-one assumption.

Pipelining Harris corner detection with a tiny FPGA for a mobile robot

With their parallelizable inner structures, field programmable gate array (FPGA) are increasing their popularity in todays embedded systems. In this paper, we present an implemented, unique and pipelined FPGA architecture designed with Verilog HDL to be used on a mobile robot for detecting corners in colored stereo images using Harris corner detection (HCD) algorithm in real time. The architecture consists of 3 pipelined modules and processes RGB555 formatted images in 640×480 resolution. The design is implemented on Xilinxs ML501 board having a XC5VLX50 FPGA, one of the smallest FPGAs of Virtex-5 series. Raw and processed data are stored into a single DDR2 memory of Micron, MT4HTF3264HY on the board, allowing only a single read or write operation at a time. By using less than 75% of FPGA resources and a 100MHz system clock, we achieved a corner detection rate of 0.33 pixels per clock cycle (ppcc) corresponding to a corner detection frequency of 54Hz for the stereo images.

Layout indexing of trademark images

Ensuring the uniqueness of trademark images and protecting their identities are the most important objectives for the trademark registration process. To prevent trademark infringement, each new trademark must be compared to a database of existing trademarks. Given a newly designed trademark image, trademark retrieval systems are not only concerned with finding images with similar shapes but also locating images with similar layouts. Performing a linear-search, i.e., computing the similarity between the query and each database entry and selecting the closest one, is inefficient for large database systems. An effective and efficient indexing mechanism is, therefore, essential to select a small collection of candidates. This paper proposes a framework in which a graph-based indexing schema will be applied to facilitate efficient trademark retrieval based on spatial relations between image components, regardless of mutual shape similarity. Our framework starts by segmenting trademark images into distinct shapes using a shape identification algorithm. Identified shapes are then encoded automatically into an attributed graph whose vertices represent shapes and whose edges show spatial relations (both directional and topological) between the shapes. Using a graph-based indexing schema, the topological structure of the graph as well as that of its subgraphs are represented as vectors in which the components correspond to the sorted Laplacian eigenvalues of the graph or subgraphs. Having established the signatures, the indexing amounts to a nearest neighbour search in a model database. For a query graph and a large graph data set, the indexing problem is reformulated as that of fast selection of candidate graphs whose signatures are close to the query signature in the vector space. An extensive set of recognition trials, including a comparison with manually constructed graphs, show the efficacy of both the automatic graph construction process and the indexing schema.

Many-to-Many Feature Matching in Object Recognition

One of the bottlenecks of current recognition (and graph matching) systems is their assumption of one-to-one feature (node) correspondence. This assumption breaks down in the generic object recognition task where, for example, a collection of features at one scale (in one image) may correspond to a single feature at a coarser scale (in the second image). Generic object recognition therefore requires the ability to match features many-to-many. In this paper, we will review our progress on three independent object recognition problems, each formulated as a graph matching problem and each solving the many-to-many matching problem in a different way. First, we explore the problem of learning a 2-D shape class prototype (represented as a graph) from a set of object exemplars (also represented as graphs) belonging to the class, in which there may be no one-to-one correspondence among extracted features. Next, we define a low-dimensional, spectral encoding of graph structure and use it to match entire subgraphs whose size can be different. Finally, in very recent work, we embed graphs into geometric spaces, reducing the many-to-many graph matching problem to a weighted point matching problem, for which efficient many-to-many matching algorithms exist.

Discrete representation of top points via scale space tessellation

In previous work, singular points (or top points) in the scale space representation of generic images have proven valuable for image matching. In this paper, we propose a construction that encodes the scale space description of top points in the form of a directed acyclic graph. This representation allows us to utilize graph matching algorithms for comparing images represented in terms of top point configurations instead of using solely the top points and their features in a point matching algorithm, as was done previously. The nodes of the graph represent the critical paths together with their top points. The edge set will capture the neighborhood distribution of vertices in scale space, and is constructed through a Delaunay triangulation scheme. We also will present a many-to-many matching algorithm for comparing such graph-based representations. This algorithm is based on a metric-tree representation of labelled graphs and their low-distortion embeddings into normed vector spaces via spherical encoding. This is a two-step transformation that reduces the matching problem to that of computing a distribution-based distance measure between two such embeddings. To evaluate the quality of our representation, two sets of experiments are considered. First, the stability of this representation under Gaussian noise of increasing magnitude is examined. In the second set of experiments, a series of recognition experiments is run on a small face database.