Graph Matching with Anchor Nodes: A Learning Approach
GGraph Matching with Anchor Nodes: A Learning Approach
Nan Hu Raif M. Rustamov Leonidas GuibasStanford UniversityStanford, CA, USA [email protected], [email protected], [email protected]
Abstract
In this paper, we consider the weighted graph matchingproblem with partially disclosed correspondences betweena number of anchor nodes. Our construction exploits re-cently introduced node signatures based on graph Lapla-cians, namely the Laplacian family signature (LFS) on thenodes, and the pairwise heat kernel map on the edges. Inthis paper, without assuming an explicit form of parametricdependence nor a distance metric between node signatures,we formulate an optimization problem which incorporatesthe knowledge of anchor nodes. Solving this problem givesus an optimized proximity measure specific to the graphsunder consideration. Using this as a first order compat-ibility term, we then set up an integer quadratic program(IQP) to solve for a near optimal graph matching. Our ex-periments demonstrate the superior performance of our ap-proach on randomly generated graphs and on two widely-used image sequences, when compared with other exist-ing signature and adjacency matrix based graph matchingmethods.
1. Introduction
The exact and approximate graph matching problem isof great interest in computer vision due to its numerous ap-plications in areas such as 2D and 3D image registration,object recognition and biomedical identification, and objecttracking in video sequences. An important variant of theproblem is the semi-supervised setting where a small pro-portion of correct node correspondences between the graphsare known. Such correspondences can be based on addi-tional information provided for only a few nodes, humanjudgement or prior knowledge, etc. Algorithms that cantake advantage of this information to infer the correspon-dences for the rest of the graph nodes are highly desirable.While there has been a recent effort in applying machinelearning concepts to the graph matching problem in com-puter vision [3, 15], these works are based on the assump-tion that a training set consisting of pairs of graphs with fully correct correspondences given, and that the trainingset is representative enough of testing graphs, so that learn-ing done with training graphs can be usefully transferredto testing graphs. This problem setup, however, is differ-ent from our setting where we only have two graphs withpartially known correspondences; in a sense, these knowncorrespondences constitute our “training data”. In addition,this amounts to a much smaller and more restricted amountof training data, making the problem challenging.In this work we provide a method for effectively incor-porating known correspondences into the commonly usedinteger quadratic program formulation of graph match-ing. Specifically, our main contribution is to devise a newfirst order compatibility term between two nodes of differ-ent graphs. Our method uses the recently proposed one-parameter family of node signatures called Laplacian Fam-ily Signatures (LFS) [11], which provide a feature vector(signature) for each node based solely on the node’s struc-tural position within the graph. In contrast to [11], we donot assume an explicit form of parametric dependence forgenerating these signatures, but leave it in an unspecifiedgeneric form. Since graph matching is performed usingthe dissimilarity/distance between these signatures, we de-rive the distance between the generic signatures. As a re-sult of this manipulation, we find that the entire processof computing and comparing these generic signatures canbe encoded into a single proximity matrix. We then intro-duce an algorithm to learn this proximity matrix from theknowledge of provided correct correspondences. This isdone by requiring anchor nodes in one graph to correspondnear to their known partners in the other and to be far fromnon-correspondences, which can be set up as a max-marginproblem [28].This method was chosen due to a number of benefits.First, our max-margin formulation makes an effective use ofthe scarce training data: even a small number of known cor-respondences (two anchor correspondences are used in allof our experiments) leads to a large number of constraintson the proximity matrix. Second, our formulation resultsin learning a proximity matrix that is relatively small (tens1 a r X i v : . [ c s . C V ] A p r y tens in the examples shown) which allows us to reliablylearn it without over-fitting. Third, our max-margin prob-lem, can be solved using column generation [16], whichresults in an efficient algorithm that scales well with the in-creasing size of the graphs and number of constraints. Notation and Paper Organization
Let G = ( V, E ) and G (cid:48) = ( V (cid:48) , E (cid:48) ) be two undirected weighted graphs. Ourgoal is to find an approximate matching between these twographs based solely on their structural properties (e.g. no ex-ternally provided attributes are available for nodes). We as-sume that a partial correspondence between graphs is given.Namely, let U ⊂ V and U (cid:48) ⊂ V (cid:48) be the subsets of nodesthat are known to be in correspondence; we will refer tothese as anchor node sets for G and G (cid:48) respectively.We follow the commonly used integer quadratic program(IQP) formulation for the graph matching problem based ontwo kinds of compatibility terms. The first order compati-bility d ( i, a ) encodes similarity of a node i ∈ V in graph G to a node a ∈ V (cid:48) in graph G (cid:48) . For pairs of nodes i, j ∈ V and a, b ∈ V (cid:48) , the second order compatibility d ( i, j, a, b ) measures the compatibility of matching the node pair ( i, j ) to node pair ( a, b ) .Our main goal in this paper is to incorporate the knowl-edge of the anchor correspondences into the first order com-patibility term which will be expressed as follows: d ( i, a ) = c B d B ( i, a ) + c ap d ap ( i, a ) , where c B and c ap are some weights. The first term d B ( i, a ) is based on our formulation of LFS proximity matrix andis presented in Section 3.1. The construction of the secondterm d ap ( i, a ) which involves the heat kernel is explainedin Section 3.2. The matching scheme incorporating boththe first order and second order compatibility functions ispresented in Section 4. In Section 5, we present experimentsusing our algorithm on three common datasets.
2. Related Work
Our family of signatures are closely related to node-based signatures on graphs, different forms of which hasalready been considered, e.g. [7, 23, 9, 27, 12]. Recently,Sun et. al [24] proposed the heat kernel signature (HKS)for application of shape matching in geometry processing.Their signature is based on the simulated heat diffusion pro-cess on manifold. Aubry et. al. [1] later proposed a sig-nature of similar structure based on quantum processes ongraph. Both have been shown in [11] to be special instancesof LFS.Different forms of spectral matrices have already beenconsidered in matching. Among the pioneering work isUmeyama’s [25] weighted graph matching algorithm from adecomposition of adjacency matrices. His method was later generalized to graphs of different sizes [17, 31]. Robles-Kelly et. al. [20] used the steady state of the Markov tran-sition matrix to order the nodes and match using edit dis-tances.Later in [21], the same authors proposed to use theleading eigenvector of the adjacency matrix to serialize thegraph nodes for matching. Qiu et. al [19] considered us-ing the Fiedler vector together with the proximity to theperimeter of the graph to partition the graph into discon-nected components for a hierarchical matching. Cho et. al.[4] constructed a reweighed random walk similar to person-alized PageRank on the association graph with the additionof an absorbing node, and used the quasi-stationary distri-bution to find a matching. Emms et. al. [6] simulated aquantum walk on the auxiliary graph and used the particleprobability of each auxiliary node as the cost of assignmentfor a bipartite matching.In a broader sense, other relaxation-based matching al-gorithms are also related to our work. Gold and Rangarajan[8] proposed the well-known
Graduated Assignment Algo-rithm . van Wyk et. al. [26] designed a projection ontoconvex set (POCS) based algorithm to solve IQP. Schelle-wald et. al. [22] constructed a semi-definite programmingrelaxation of the IQP. Leordeanu et. al. [13] proposed aspectral method to solve a relaxed IQP by only consideringlinear inequality constraints at discretization. The idea wasfurther extended by Cour et. al. [5], where they added anaffine constraint during relaxation. Zaslavskiy et. al. [29]approached the IQP from the point of a relaxation of theoriginal least-square problem to a convex and concave opti-mization problem on the set of doubly stochastic matrices.Leordeanu et. al. [14] proposed an integer projected fixedpoint (IPFP) algorithm to iteratively search for a fixed pointsolution and then discretize it into the matching domain.
3. Anchor Based Compatibility
In this section we discuss the construction and compu-tation of two kinds of first order compatibility terms thattake advantage of known correspondences between anchornodes.
We start by reviewing the concept of Laplacian FamilySignatures introduced in [11]. Consider one of the graphsto be matched, say G = ( V, E ) . Let w be the weights onedges, i.e. w : E (cid:55)→ R + . The graph Laplacian is defined as L = D − A , where A is the graph adjacency matrix with A ij = (cid:40) w ( i, j ) if ( i, j ) ∈ E otherwiseand D is a diagonal matrix of total incident weights, i.e. D ii = (cid:80) j A ij . L has numerous nice properties [2], ofhich most relevant to us is the symmetry and positivesemi-definiteness. This makes it possible to consider theeigen-decomposition of L ; we denote by { λ k , φ k } | V | k =1 theeigenpairs of graph Laplacian matrix L (eigenvalue and as-sociated eigenvector). We use the same notation with theprime symbol added for the corresponding constructs of oursecond graph G (cid:48) .The Laplacian Family Signatures (LFS) for a node u ∈ V is a one-parameter family of structural node descriptorsthat is defined by s u ( t ) = (cid:88) k h ( t ; λ k ) φ k ( u ) (1)where h ( t ; λ k ) is a real valued function. Special h ( t ; λ k ) of different forms will result in the heat kernel signature(HKS) [24] when h ( t ; λ k ) = exp( − tλ k ) , the wave kernelsignature (WKS) [1] when h ( t ; λ k ) = exp( − ( t − log λ k ) σ ) ,or the wavelet signature if h ( t ; λ k ) admits some special be-havior as described in [10].These signatures describe a given node’s structural rela-tionship to its neighborhood. For example, HKS has an in-terpretation in terms of a simulated heat diffusion process:for each node, this signature captures the amount of heatleft at the node at various times (here t ) assuming that a unitamount is put on the node initially ( t = 0 ). These signaturesare naturally intrinsic, namely if two graphs are isomorphic,then the signatures of corresponding nodes are the same; thesignatures are also stable under small perturbations [11].The above discussion suggests using these signatures asnode attributes to design first order compatibility terms —for if the signatures of two nodes from the two graphs arevery different, then these vertices are less likely to be incorrespondence. However, such an approach does not takeinto account the given anchor correspondences, because theform of the function h ( t ; λ k ) is explicitly provided before-hand.To overcome this difficulty, in this paper, in contrast to[11], we will not assume an explicit form for the function h ( t ; λ k ) , nor will we assume a specific form of dissimilaritymeasure when comparing the LFS of two nodes. Instead,we assume that h ( t ; λ k ) is a generic linear combination ofsome real-valued functions { b i ( t ) } N b i =1 , given as h ( t ; λ k ) = N b (cid:88) i =1 a ki b i ( t ) , (2)where { a ki } are some real coefficients. We assume a similarexpression for the second graph G (cid:48) with possibly a differ-ent set of coefficients { a (cid:48) ki } . Let (cid:104)· , ·(cid:105) be an arbitrary innerproduct of real-valued functions. Assuming that LFS com-parison employs this dot product, the dissimilarity betweentwo nodes u ∈ V, v ∈ V (cid:48) can be expressed as d ( s u ( t ) , s (cid:48) v ( t )) = (cid:104) s u ( t ) − s (cid:48) v ( t ) , s u ( t ) − s (cid:48) v ( t ) (cid:105) Substituting (2) to (1), we have s u ( t ) = K (cid:88) k =1 φ k ( u ) N b (cid:88) i =1 a ki b i ( t ) ,s (cid:48) v ( t ) = K (cid:48) (cid:88) k =1 φ (cid:48) k ( v ) N b (cid:88) i =1 a (cid:48) ki b i ( t ) . Denote A = [ a ki ] ∈ R K × N b , θ u = φ ( u ) ... φ K ( u ) ∈ R K , A (cid:48) = [ a (cid:48) ki ] ∈ R K (cid:48) × N b , θ (cid:48) v = φ (cid:48) ( v ) ... φ (cid:48) K (cid:48) ( v ) ∈ R K (cid:48) . Let b ( t ) = b ( t ) ... b N b ( t ) ∈ R N b and C ij = (cid:104) b i ( t ) , b j ( t ) (cid:105) . Nowafter denoting C = [ C ij ] ∈ R N b × N b , we obtain d ( s u ( t ) , s (cid:48) v ( t ))= (cid:104) θ (cid:62) u Ab ( t ) − θ (cid:48)(cid:62) v A (cid:48) b ( t ) , θ (cid:62) u Ab ( t ) − θ (cid:48)(cid:62) v A (cid:48) b ( t ) (cid:105) = (cid:20) θ u θ (cid:48) v (cid:21) (cid:62) (cid:20) A − A (cid:48) (cid:21) C (cid:20) A − A (cid:48) (cid:21) (cid:62) (cid:124) (cid:123)(cid:122) (cid:125) B ∈ R ( K + K (cid:48) ) × ( K + K (cid:48) ) (cid:20) θ u θ (cid:48) v (cid:21)(cid:124) (cid:123)(cid:122) (cid:125) w uv ∈ R ( K + K (cid:48) ) = w (cid:62) uv Bw uv This formulation holds for any inner-product based dis-similarity metric. The number of basis functions, althoughassumed finite above can be easily extended to infinite, i.e.the formulation is still valid as N b → ∞ and the basis perse is also arbitrary. The only restriction, as a result of thepositive semi-definiteness of C , is B (cid:23) .The above discussion gives the general expression thatwe will use as a part of our first-order compatibility mea-sure. Namely, we set d B ( i, a ) = (cid:112) w (cid:62) ia Bw ia for any twonodes i ∈ V, a ∈ V (cid:48) . This representation is especially use-ful since it avoids determining the intermediate matrices C , A and A (cid:48) explicitly, but allows us to learn directly the prox-imity matrix B which is a small matrix (tens by tens in ourexperiments). Here we explain how to learn the proximity matrix B fromthe knowledge of anchor nodes. A good proximity ma-trix B should move closer node pairs that correspond, andmove away nodes that are non-matches. If we let U ⊂ V be the set of anchor nodes in graph G , and U (cid:48) ⊂ V (cid:48) betheir known correspondences in graph G (cid:48) , then we want B ( i, a ) = w (cid:62) ia Bw ia to be small if i and a are in corre-spondence, and to be large otherwise. One way of achievingthis is to formulate the problem as a max-margin problemsimilar to SVM. max γ s . t . w (cid:62) ib Bw ib − w (cid:62) ia Bw ia ≥ γ, ∀ i, ∀ b (cid:54) = aw (cid:62) ja Bw ja − w (cid:62) ia Bw ia ≥ γ, ∀ a, ∀ j (cid:54) = iB (cid:23) (cid:107) B (cid:107) F ≤ . This can be easily verified to be equivalent to min (cid:107) B (cid:107) F s . t . tr(( w ib w (cid:62) ib − w ia w (cid:62) ia ) B ) ≥ , ∀ i, ∀ b (cid:54) = a tr(( w ja w (cid:62) ja − w ia w (cid:62) ia ) B ) ≥ , ∀ a, ∀ j (cid:54) = iB (cid:23) . For large graphs, however, the problem could be verypossibly infeasible. Therefore, we allow some violation inthe training set and introduce slack variables. min (cid:107) B (cid:107) F + C n (cid:80) ni =1 ξ i s . t . tr(( w ik w (cid:62) ik − w ij w (cid:62) ij ) B ) ≥ − ξ i , ∀ i, ∀ k (cid:54) = j tr(( w lj w (cid:62) lj − w ij w (cid:62) ij ) B ) ≥ − ξ i , ∀ j, ∀ l (cid:54) = iB (cid:23) ξ i ≥ , where n = | U | is the number of anchor nodes.One drawback of this formulation is that we put uniformweights on slack variables. However, intuitively for a vi-olation of the margin constraint, we would rather to havethe violated nodes to be near to the correct matches withinthe graph, namely we want to put a non-uniform scale onthe slack variables to penalize more severely for nodes thatare farther from the correct matches. Therefore, we intro-duce a loss-function Ω( k, j ) to re-scale the slack variables.In our graph matching setting, Ω( k, j ) could be the shortestdistance over the graph, or the heat kernel as described inSection 3.2 (we used heat kernel in our experiments as ithas been shown to be more robust than the adjacency ma-trix [11] and, hence, shortest distance). Now the problembecomes min (cid:107) B (cid:107) F + C n (cid:80) ni =1 ξ i s . t . tr(( w ib w (cid:62) ib − w ia w (cid:62) ia ) B ) ≥ − ξ i Ω (cid:48) ( b,a ) , ∀ i, ∀ b (cid:54) = a tr(( w ja w (cid:62) ja − w ia w (cid:62) ia ) B ) ≥ − ξ i Ω( j,i ) , ∀ a, ∀ j (cid:54) = iB (cid:23) ξ i ≥ . Let ( · ) vec be the vector form of a matrix, and b = B vec and ψ ik = ( w ik w (cid:62) ik − w ij w (cid:62) ij ) vec . The above problem could be solved by first relaxing the semi-definite constraint andthen projecting the solution to the semi-definite cone. Therelaxed problem is a quadratic programming problem min (cid:107) b (cid:107) + C n (cid:80) ni =1 ξ i s . t . ψ (cid:62) ib b ≥ − ξ i Ω (cid:48) ( b,a ) , ∀ i, ∀ b (cid:54) = aψ (cid:62) ja b ≥ − ξ i Ω( j,i ) , ∀ a, ∀ j (cid:54) = iξ i ≥ . The dual of it is max − (cid:80) ib (cid:80) ja α ib α ja ψ (cid:62) ib ψ ja + (cid:80) α ib s . t . α ib ≥ (cid:80) i (cid:16) α ib Ω (cid:48) ( b,a ) + α ja Ω( j,i ) (cid:17) ≤ Cn . As the number of constraints in this problem is of O ( | U | ( | V | + | V (cid:48) | )) , it becomes impossible to solve whenthe size of the graphs is very large. One technique that couldbe used to lower the computational cost is column genera-tion [16]. The key idea of this iterative algorithm is thatalthough the number of constraints is large, only a smallportion of them will be nonzero at the solution. Therefore,only this small subset of constraints are necessary to the so-lution. To find this subset, the algorithm iteratively adds oneconstraint per training sample that violated the constraintthe most until all constraints are satisfied. In addition, aftereach iteration, we need to project B back to semi-definitecone to restrain B (cid:23) . The pseudocode of the algorithm isomitted here for the sake of saving space. In this subsection we introduce the second term appear-ing in our first order compatibility measure. This term isbased on the heat diffusion process on graph G . Specifi-cally, consider the graph heat kernel k t ( u, v ) , which mea-sures the amount of heat transferred from node u to node v after time t , assuming a unit amount was placed at u in thebeginning ( t = 0 ). The heat kernel has the following repre-sentation in terms of the eigen-decomposition of the graphLaplacian: k t ( u, v ) = (cid:88) k exp( − tλ k ) φ k ( u ) φ k ( v ) . We use the heat kernel value of anchor nodes at a given nodeas another first order constraint. Namely, for any node v ofgraph G define the heat kernel distance to anchor nodes as d H ap ( v ) = (cid:88) u ∈ U k t ( u, v ) , where U is the set of anchor nodes of G . The same con-struction using the anchor nodes U (cid:48) of our second graph G (cid:48) provides the quantity d (cid:48) H ap ( · ) for each node of G (cid:48) .or two nodes i and a in graphs G and G (cid:48) , we define thesecond portion of our first order compatibility measure by d ap ( i, a ) = (cid:12)(cid:12)(cid:12) d H ap ( i ) − d (cid:48) H ap ( a ) (cid:12)(cid:12)(cid:12) . This quantity is a plausible measure of dissimilarity be-tween nodes because the anchor nodes U and U (cid:48) are knownto be in correspondence. Moreover, the heat kernel is nat-urally intrinsic (if two graphs are isomorphic, their corre-sponding heat kernels are the same) and it is stable undersmall perturbations [11].
4. Matching Scheme
Here we formulate the graph matching as an integerquadratic program (IQP). Let G = ( V, E ) and G (cid:48) =( V (cid:48) , E (cid:48) ) be the two graphs, U and U (cid:48) be the anchor nodeset for G and G (cid:48) respectively. For any two nodes i ∈ V \ U and a ∈ V (cid:48) \ U (cid:48) , let d B ( i, a ) = w (cid:62) ia Bw ia be the learnedproximity, and let d ap ( i, a ) = (cid:12)(cid:12)(cid:12) d H ap ( i ) − d (cid:48) H ap ( a ) (cid:12)(cid:12)(cid:12) . Our firstorder compatibility term is d ( i, a ) = c B d B ( i, a ) + c ap d ap ( i, a ) . Now we need a second order compatibility term, forwhich we will use the heat kernel as done in [11]. For anytwo nodes i, j ∈ V and a, b ∈ V (cid:48) , the pairwise heat kerneldistance is defined as d k ( i, j, a, b ) = | k t ( i, j ) − k (cid:48) t ( a, b ) | , and this gives a measure of how compatible matching nodes i and a is with matching j and b . As has been discussed in[11], the heat kernel can be thought of as a noise tolerantapproximation of the adjacency matrix, and is stable undersmall perturbations. Thus, our second order proximity termcan be thought as a generalization of the commonly usedadjacency-based second order term.Combining all this information, weconstruct a compatibility matrix W ∈ R ( | V |−| U | )( | V (cid:48) |−| U (cid:48) | ) × ( | V |−| U | )( | V (cid:48) |−| U (cid:48) | ) , W ia,jb = (cid:40) d k ( i, j, a, b ) , i (cid:54) = j, a (cid:54) = bd ( i, a ) , i = j, a = b Let X ∈ { , } ( | V |−| U | )( | V (cid:48) |−| U (cid:48) | ) be the one-to-one map-ping matrix, and x ∈ { , } ( | V |−| U | )( | V (cid:48) |−| U (cid:48) | ) be the vectorform of X . The IQP can be written as x ∗ = arg max( x (cid:62) W x ) s.t. x ∈ { , } ( | V |−| U | )( | V (cid:48) |−| U (cid:48) | ) ∀ i (cid:88) a ∈ V (cid:48) \ U (cid:48) x ia ≤ , ∀ a (cid:88) i ∈ V \ U x ia ≤ . As is well-known this problem is NP-complete and thereis rich literature on approximation algorithms for this prob-lem. Comparison of the performance of different IQP ap-proximation solvers is outside the scope of our paper. Inour experiments, we selected a recently proposed algorithm,reweighed random walk matching (RRWM) [4]. The mainreason we chose this algorithm is its superior performancewhen compared with other state-of-the-art approximationsolvers, including SM [13], SMAC [5], HGM [30], IPFP[14], GAGM [8], SPGM [26]. In consideration of space,we omit the introduction of their algorithm here and leavethe details to the original paper [4].
5. Experiments
We tested our approach on three different datasets: 1)synthetically generated random graphs; 2) CMU Hotel se-quence for large baseline matching; and 3) pose house se-quence from [18] for large rotation angle matching.
In this section, following the experimental protocol of[4], we synthetically generated random graphs and per-formed a comparative study. For a pair of graph G and G , they share n in common nodes and n (1) out and n (2) out outliernodes. Edges are constructed with a density ρ and weightsare randomly distributed in [0 , . Perturbation is done withadded random Gaussian noise N (0 , σ ) .In this experiment, we test the matching performancefor W constructed using i) only the adjacency matrix [4],ii) only pairwise heat kernel distance d k ( i, j, a, b ) , iii) d k ( i, j, a, b ) with WKS [11], iv) d k ( i, j, a, b ) with d ap ( i, a ) ( c B = 0 , c ap = 1 ), v) d k ( i, j, a, b ) with d B ( i, a ) ( c B =1 , c ap = 0 ), and vi) d k ( i, j, a, b ) with d ap ( i, a ) and d B ( i, a ) ( c B = 8 , c ap = 3 ), on three different settings: 1) differentlevels of deformation noise σ ; 2) different numbers of out-liers; 3) different edge densities ρ . Fig. 1 shows the averagematching accuracy. In the experiment, the number of an-chor nodes | U | = 2 . The Red solid curve for RRWM isthe baseline approach using the adjacency matrices. FromFig. 1 (a), it can be seen that with the help of learned prox-imity matrix B and the term d ap ( i, a ) , the matching resultsare more robust to noise. In Fig. 1 (c), the matching ac-curacy is much improved at different edge densities for arelatively large deformation noise ( σ = 0 . ). Not only thematching accuracy is improved, their corresponding com-putational time is also decreased as shown in Fig. 1 (f). In this experiment, we test our descriptors on the CMUHotel sequence, which is widely used in performance eval-uation of graph matching algorithms as a wide baselinedataset. It consists of 101 frames, and there are 30 fea-ture points labeled consistently across all frames. We build
Deformation noise σ A cc u r ac y n in = 20 n out = 0Edge density ρ = 1 RRWMd k RRWMWKS+d k RRWMd ap +d k RRWMd B +d k RRWMd B +d ap +d k RRWM (a) n out A cc u r ac y n in = 20Deformation noise σ = 0.25Edge density ρ = 1 RRWMd k RRWMWKS+d k RRWMd ap +d k RRWMd B +d k RRWMd B +d ap +d k RRWM (b)
Edge density ρ A cc u r ac y n in = 20 n out = 0Deformation noise σ = 0.5 RRWMd k RRWMWKS+d k RRWMd ap +d k RRWMd B +d k RRWMd B +d ap +d k RRWM (c)
Deformation noise σ T i m e n in = 20 n out = 0Edge density ρ = 1 RRWMd k RRWMWKS+d k RRWMd ap +d k RRWMd B +d k RRWMd B +d ap +d k RRWM (d) n out T i m e n in = 20Deformation noise σ = 0.25Edge density ρ = 1 RRWMd k RRWMWKS+d k RRWMd ap +d k RRWMd B +d k RRWMd B +d ap +d k RRWM (e)
Edge density ρ T i m e n in = 20 n out = 0Deformation noise σ = 0.5 RRWMd k RRWMWKS+d k RRWMd ap +d k RRWMd B +d k RRWMd B +d ap +d k RRWM (f)
Figure 1: Matching accuracy and computation time of IQP with different compatibility functions.fully connected graphs purely based on the geometry of thefeature points, taking the Euclidean distance as the weightsbetween pair of feature points. Affinity matrix W were setup similarly as in Section 5.1. | U | = 2 nodes were ran-domly selected as the anchor nodes. We compute the aver-age matching accuracy of each frame to the rest of framesin the sequence. Fig. 2 (a) showed the performance of thematching. As can be seen the matching performance wasimproved when heat kernel is used in lieu of the adjacencymatrix, because the noise tolerance property of the formersmoothes out the effect of deformation noise. With the add-on effect of proximity matrix B and d ap ( i, a ) , furthermore,the matching performance was much improved. Fig. 2 (b,c)showes an example of the matching between the 20th andthe 90th frame of the sequence (yellow lines are correctmatches and red lines are wrong matches). Matching givesall correct matches, hence is only shown once.In the second part of this experiment, we select to testthe influence of the number of anchor nodes on the averagematching rate. We intentionally drop off the term d ap to re-duce the side effect, i.e. the matching scheme will be basedon d B + d k . We increase the number of anchor nodes andcompare the matching performance in otherwise the samesetting.As shown in Fig. 3, with the increase of the number ofanchor nodes, the overall matching performance increased.However, the marginal performance gain seems to have adrop-off with the increase of the number of anchor nodes, since the matching rate gap between | U | = 10 and | U | = 5 is much smaller than that between | U | = 5 and | U | = 2 . In this experiment, we test our descriptor on the posehouse sequence used in [18]. The dataset consists of 70frames with 51 labeled feature points across the sequence.The house undergoes large pan and tilt angle change ( − ◦ for pan angle and − ◦ for tilt). The compatibilitymatrix W is built the same way as in previous experiments. | U | = 2 nodes were randomly selected as the anchor nodes.Fig. 4 showed the matching results. In Fig. 4 (a), eachlump from left to right represent a tilt angle from 0 to ◦ with a ◦ step and within a lump is the pan angle changefrom left to right for 0 to ◦ with a ◦ step. It can beenseen that extreme pan angles gave poor results while midrange pan angles yield matches with much higher accuracy— while matching accuracy was not much influenced by thedifference in tilt angles. With the addition of our learnedproximity matrix B , the gap between different pan anglesdecreases and the overall matching accuracy is superior toothers. Fig. 4 (b,c) shows the matching results and the firstand last frame of the sequence, which represent the largestpan and tilt angles. Even in this extreme case, it can be seenthat our d B + d ap + d k matching gives useful results, and ismuch better than the adjacency matrix based matching. Image Index A v e r a g e A cc u r ac y Average accuracy of an image to the rest of the sequence
RRWMd k RRWMWKS+d k RRWMd ap +d k RRWMd B +d k RRWMd B +d ap +d k RRWM (a) Matching performance(b) RRWM(c) d B + d ap + d k RRWM
Figure 2: Matching on Hotel sequence. Yellow lines de-pict the correct matches, while red lines show the wrongmatches.
6. Conclusion
In this paper, we have considered the problem of graphmatching where some correspondences are known. We havedesigned a learning algorithm which uses the anchor cor-respondences as training samples. The matching problemis set up as an IQP, where we use the learned proximityand heat kernel distance to anchor nodes as first order com-patibility, and pairwise heat kernel distance difference as asecond order compatibility. With a very small number ofanchor nodes ( | U | = 2 in all of the experiments) we haveobtained superior performance as compared to the state-of-the-art techniques based on adjacency matrices. Acknowledgment : The authors would like to ac-knowledge NSF grants IIS 1016324, CCF 1161480, DMS1228304, AFOSR FA9550-12-1-0372, the Max Planck
10 20 30 40 50 60 70 80 90 1006065707580859095100
Image Index A v e r a g e A cc u r ac y Average accuracy of an image to the rest of the sequence |U|=2|U|=5|U|=10
Figure 3: Matching on Hotel sequence with different num-ber of anchor nodes. (Matching scheme is d B + d k .)Center for Visual Computing and Communications, and aGoogle research award. References [1] M. Aubry, U. Schlickewei, and D. Cremers. The wave kernelsignature: A quantum mechanical approach to shape analy-sis. In
ICCV Workshop 4DMOD , 2011. 2, 3.1[2] T. Biyikoglu, J. Leydold, and P. F. Stadler.
Laplacian eigen-vectors of graphs: Perron-Frobenius and Faber-Krahn TypeTheorems . Springer, 2007. 3.1[3] T. Caetano, J. McAuley, C. L., Q. Le, and A. Smola. Learn-ing graph matching.
IEEE Trans. PAMI , 31(6):1048–1058,2009. 1[4] M. Cho, J. Lee, and K. M. Lee. Reweighted random walksfor graph matching. In
Proceedings of the 11th Europeanconference on Computer vision: Part V , ECCV’10, pages492–505, Berlin, Heidelberg, 2010. Springer-Verlag. 2, 4,5.1, 5.1[5] T. Cour, P. Srinivasan, and J. Shi. Balanced graph matching.In
NIPS’06 , pages 313–320, 2006. 2, 4[6] D. Emms, R. C. Wilson, and E. R. Hancock. Graph matchingusing the interference of continuous-time quantum walks.
Pattern Recogn. , 42(5):985–1002, May 2009. 2[7] M. Eshera and K. Fu. A graph distance measure for imageanalysis.
IEEE Trans. Syst. Man Cybern. , page 398Ð408,1984. 2[8] S. Gold and A. Rangarajan. A graduated assignment algo-rithm for graph matching.
IEEE Trans. Patt. Anal. Mach.Intell. , 18, 1996. 2, 4[9] M. Gori, M. Maggini, and L. Sarti. Exact and approximategraph matching using random walks.
IEEE Trans. PatternAnal. Mach. Intell. , 27(7):1100–1111, July 2005. 2[10] D. K. Hammond, P. Vandergheynst, and R. Gribonval.Wavelets on graphs via spectral graph theory.
Applied andComputational Harmonic Analysis , 30(2):129–150, Mar.2011. 3.1
Image Index A v e r a g e A cc u r ac y Average accuracy of an image to the rest of the sequence
RRWMd k RRWMWKS+d k RRWMd ap +d k RRWMd B +d k RRWMd B +d ap +d k RRWM (a) Matching performance(b) RRWM(c) d ap + d k RRWM(d) d B + d ap + d k RRWM
Figure 4: Matching on pose house sequence. Yellow linesdepict the correct matches, while red lines show the wrongmatches. [11] N. Hu and L. Guibas. Spectral Descriptors for Graph Match-ing.
ArXiv e-prints , Apr. 2013. arXiv:1304.1572[cs.CV]. 1,2, 3.1, 3.1, 3.1.1, 3.2, 4, 5.1[12] S. Jouili and S. Tabbone. Graph matching based on nodesignatures. In
Proceedings of GbRPR ’09 , pages 154–163,Berlin, Heidelberg, 2009. Springer-Verlag. 2[13] M. Leordeanu and M. Hebert. A spectral technique for cor-respondence problems using pairwise constraints. In
ICCV’05 , pages 1482–1489, Washington, DC, USA, 2005. IEEEComputer Society. 2, 4[14] M. Leordeanu, M. Hebert, and R. Sukthankar. An integerprojected fixed point method for graph matching and map inference. In
NIPS . Springer, December 2009. 2, 4[15] M. Leordeanu, R. Sukthankar, and M. Hebert. Unsuper-vised learning for graph matching.
Int. J. Comput. Vision ,96(1):28–45, Jan. 2012. 1[16] M. E. Lübbecke and J. Desrosiers. Selected topics in col-umn generation.
Oper. Res. , 53(6):1007–1023, Nov. 2005.1, 3.1.1[17] B. Luo and E. R. Hancock. Structural graph matching usingthe em algorithm and singular value decomposition.
IEEETrans. Pattern Anal. Mach. Intell. , 23(10):1120–1136, Oct.2001. 2[18] J. J. McAuley, T. de Campos, and T. S. Caetano. Unifiedgraph matching in euclidean spaces.
IEEE Conf. CVPR ,pages 1871–1878, 2010. 5, 5.3[19] H. Qiu and E. R. Hancock. Graph matching and clusteringusing spectral partitions.
Pattern Recogn. , 39(1):22–34, Jan.2006. 2[20] A. Robles-Kelly and E. R. Hancock. String edit distance,random walks and graph matching. In
Proceedings of theJoint IAPR International Workshop on Structural, Syntactic,and Statistical Pattern Recognition , pages 104–112, London,UK, UK, 2002. Springer-Verlag. 2[21] A. Robles-Kelly and E. R. Hancock. Graph edit distancefrom spectral seriation.
IEEE Trans. Pattern Anal. Mach.Intell. , 27(3):365–378, Mar. 2005. 2[22] C. Schellewald and C. Schnörr. Probabilistic subgraphmatching based on convex relaxation. In
EMMCVPR’05 ,pages 171–186, Berlin, Heidelberg, 2005. Springer-Verlag.2[23] A. Shokoufandeh and S. J. Dickinson. A unified frameworkfor indexing and matching hierarchical shape structures. In
IWVF-4 , pages 67–84, London, UK, UK, 2001. Springer-Verlag. 2[24] J. Sun, M. Ovsjanikov, and L. Guibas. A concise and prov-ably informative multi-scale signature based on heat diffu-sion. In
Eurographics Symposium on Geometry Processing(SGP) , 2009. 2, 3.1[25] S. Umeyama. An eigendecomposition approach to weightedgraph matching problems.
IEEE Trans. Pattern Anal. Mach.Intell. , 10(5):695–703, 1988. 2[26] B. J. van Wyk and M. A. van Wyk. A pocs-based graphmatching algorithm.
IEEE Trans. Pattern Anal. Mach. Intell. ,26(11):1526–1530, Nov. 2004. 2, 4[27] P. C. Wong, H. Foote, G. Chin, P. Mackey, and K. Perrine.Graph signatures for visual analytics.
IEEE trans. on visual-ization and computer graphics , 12(6):1399–413, 2006. 2[28] E. P. Xing, A. Y. Ng, M. I. Jordan, and S. J. Russell. Dis-tance metric learning with application to clustering withside-information. In
NIPS , pages 505–512, 2002. 1[29] M. Zaslavskiy, F. Bach, and J.-P. Vert. A path following al-gorithm for the graph matching problem.
IEEE Transactionson PAMI , 31(12):2227–2242, 2009. 2[30] R. Zass and A. Shashua. Probabilistic graph and hypergraphmatching.
CVPR , 2008. 4[31] G. Zhao, B. Luo, J. Tang, and J. Ma. Usingeigen-decomposition method for weighted graph matching.