Enumerating Unique Computational Graphs via an Iterative Graph Invariant
aa r X i v : . [ c s . D M ] F e b Enumerating Unique Computational Graphs viaan Iterative Graph Invariant
Chris [email protected] Brain
As part of a research project studying neural architectures, we needed an algo-rithm that could identify isomorphic computational graphs , the building blocksof neural networks. In our context, computational graphs represent operationsperformed on arbitrary tensors where the vertices are operations and the edgesare tensors. To generalize this problem, we represent a computational graph asa colored directed acyclic graph where the colors represent unique operations.Depending on the graph representation (e.g., adjacency matrix plus a color pervertex), multiple representations may encode the same computational graph(Figure 1).In this report, we describe a novel graph invariant for computational graphsand how we used it to generate all distinct computational graphs up to isomor-phism for small graphs. While this invariant cannot perfectly distinguish allpairs of non-isomorphic computational graphs, we suggest that it may be usefulas a heuristic for comparing graphs.
Definition 1. A computational graph on n vertices and k colors is a directedacyclic graph where each vertex is assigned an arbitrary color, and every vertexlies on a path between two designated vertices. Formally, it is a tuple ( n, k, E, c ),where:1. n ∈ N is the number of vertices;2. k ∈ N is the number of colors;3. E ⊆ [ n ] × [ n ] is a set of directed edges ( i, j ) such that i < j ;4. c : [ n ] → [ k ] is a function assigning a color to each vertex; and12 3 45 12 435 13 42 5Figure 1: Three computational graphs with different adjacencies and coloringswhen ordered that are isomorphic in the sense of Definition 2 (best viewed incolor).5. for each vertex i ∈ [ n ], there is a directed path from vertex 1 to vertex n that passes through vertex i .Note that there is no restriction that adjacent vertices have different colors.The goal is to count the number of distinct computational graphs up toisomorphism, where isomorphism is defined as follows. Definition 2.
Computational graphs G = ( n, k, E , c ) and G = ( n, k, E , c )on n vertices and k colors are isomorphic if there exists a bijection P : [ n ] → [ n ]such that:1. adjacency is preserved: ∀ i, j ∈ [ n ] . ( i, j ) ∈ E iff ( P ( i ) , P ( j )) ∈ E ; and2. coloring is preserved: ∀ i ∈ [ n ] . c ( i ) = c ( P ( i )). In the context of general graph isomorphism, a graph invariant is a propertyof a graph such that if two graphs are isomorphic, they have the same valueof that property (the converse is not necessarily true). The Weisfeiler–Lehmanalgorithm [5] uses an iterative coloring approach to come up with a canonicalcoloring on a graph, though subsequent work [1] shows that the algorithm canfail for some graphs. It is unknown whether graph isomorphism can be solvedin polynomial time [2].This paper deals with a more constrained problem than general graph iso-morphism, where the graphs are directed acyclic graphs and there are coloringsassigned to the vertices. Nonetheless, the algorithm is partially inspired by theiterative nature of the Weisfeiler–Lehman algorithm. This condition is not required for the graph invariant algorithm 1 but is used by theenumeration algorithm 2 to heavily reduce the number of possible graphs. • A000088: Number of graphs on n unlabeled nodes. This series does notconsider coloring and considers all graphs rather than just directed acyclicgraphs. • A003024: Number of acyclic digraphs with n labeled nodes. This seriestreats all vertices as uniquely labeled rather than individual colored andalso counts disconnected graphs. • A057500: Number of connected labeled graphs with n edges and n nodes.This series does not use directed edges and does not consider coloring. • A240955: Number of k -colored labeled digraphs with n vertices. In thisseries, colored refers to vertices which cannot be colored the same color asits neighbors, which is different than the notion of coloring used here.P´olya–Redfield counting can be used to count the number of colorings onundirected graphs [3] but it does not provide a way to quickly identify if twodirected colored graphs are isomorphic. In this section, we describe an algorithm for generating a novel graph invariantfor computational graphs. At a high-level, the key idea is to iteratively applyisomorphism-invariant operations to the graph in a way takes into account thegraph structure as well as the coloring. Algorithm 1 provides the pseudo-codefor the graph hashing algorithm: 3 lgorithm 1:
Iterative graph hashing algorithm for colored DAGs
Input:
Computational graph G = ( n, k, E, c ) Output:
Fixed-length hash of the graph and coloring let H = [ ] ; ⊲ List such that H i is hash for vertex i forall vertices i ∈ [ n ] do let | e − i | = in-degree ( i ) ; let | e + i | = out-degree ( i ) ; H i ← hash ( | e + i | , | e − i | , c ( i )) ; ⊲ Initialize hashes end for iteration ← to n do let ˆ H = [ ] ; ⊲ Next iteration of hashes forall vertices i ∈ [ n ] do let H − i = [ H o : ( o, i ) ∈ E ] ; ⊲ List of in-neighbor hashes let H + i = [ H o : ( i, o ) ∈ E ] ; ⊲ List of out-neighbor hashes ˆ H i ← hash ( sort ( H + i ) , sort ( H − i ) , H i ) ; end H ← ˆ H ; ⊲ Update hashes end return hash ( sort ( H ))Specifically, hash returns a fixed-length hash. Our implementation usesthe 128-bit MD5 hash algorithm which we found sufficient for our use-case.The sort function performs a lexicographical sort of the hash outputs. Werepeat the algorithm up to the number of vertices iterations (line 7) but wesuspect that it may be sufficient to iterate up to the diameter of the graph.In our implementation, we represent G as an adjacency matrix along with alist of colors of length equal to the number of vertices. The in-degree and out-degree functions are implemented as summations across the columns orrows of the matrix. To show that this algorithm computes a graph invariant, we must show thatany two isomorphic any computational graphs output the same hash. Considertwo graphs G = ( n, k, E, c ) and G ′ = ( n, k, E ′ , c ′ ) that are isomorphic withisomorphism P . Suppose that we run Algorithm 1 on G and G ′ . We will use H and H ′ to refer to the values of the hashes in the executions of the algorithmon G and G ′ , respectively. We will say that the hashes are consistent if theyrespect the isomorphism P : that is, if H i = H ′ P ( i ) for each vertex i .Choose any vertex i , and let j = P ( i ). Then the initial hash H i of i in G is equal to the initial hash H ′ j of j in G ′ (after line 6), because the two verticeshave the same adjacency and coloring, by Definition 2. Thus: Lemma 4.1.
The initial hashes H and H ′ are consistent (after line 6).4n each iteration of the outer for-loop (line 7), the hashes are updated.Suppose, at the start of an iteration, that the hashes are consistent. Again,choose any i , and let j = P ( i ). If ( o, i ) ∈ E , then ( P ( o ) , P ( i )) ∈ E ′ bythe isomorphism condition. By the consistency assumption, H o = H ′ P ( o ) .Since H − i = [ H o : ( o, i ) ∈ E ] and H ′− j = [ H ′ P ( o ) : ( P ( o ) , i ) ∈ E ′ ] (line 10),then the multisets (ignoring order) represented by H − i and H ′− j are the same.Likewise, H + i is the same as H ′ + j (line 11) ignoring order. Because sort-ing ensures that the list orderings are identical, sort ( H − i ) = sort ( H ′− j ) and sort ( H + i ) = sort ( H ′ + j ). We also have H i = H ′ j (because H and H ′ are consis-tent), so it follows that ˆ H i = ˆ H ′ j (line 14) since the hash function is operatingon the identical triplets. Thus: Lemma 4.2.
If the hashes are consistent at the start of an iteration (line 7),then they are also consistent at the end of that iteration (after line 14).By induction, Lemma 4.1 and Lemma 4.2 show that the hashes are consistentthroughout the full loop. Because the final hashes H and H ′ are consistent, theyare permutations of each other, so their sorted forms are the same, and thusthe final hashed results are identical. Therefore, Algorithm 1 computes a graphinvariant. Given the graph invariant, we can proceed to generate all computational graphs,up to isomorphism. Using the canonical ordering, we treat the first vertex (noin-neighbors) as the “input” vertex and the last vertex (no out-neighbors) asthe “output” vertex.We observe that vertices not on a directed path from v to v n in a coloreddirected acyclic graph can be pruned to yield a valid computational graph. If wegenerate the directed acyclic graphs in increasing number of vertices order, thenany graph that needs to be pruned has been already generated at a previousiteration and can be immediately skipped.5 lgorithm 2: Enumerating computational graphs
Input:
Maximum vertices n max , maximum edges e max , and colors k Output:
Yields all unique computational graphs up to constraints for numbers of vertices n ← to n max do forall bit vectors of length n ( n − / do convert bit vector to n × n upper-triangular adjacency matrix ; if number of edges > e max or contains vertex not on directed path from input to output then discard and continue to next matrix ; else forall potential colorings c : [ n ] → [ k ] do hash (matrix , coloring) using Algorithm 1 ; if hash has not been observed before then yield (matrix , coloring) ; end end end end end This algorithm also provides a canonical computational graph (i.e., the firstone that is observed) for each unique hash which represents the equivalence classof computational graphs induced by the Algorithm 1.
For our neural network use-case, we needed to generate all graphs up to 7 ver-tices, 9 edges, and 3 colors. Furthermore, the first vertex and the last vertexare specially colored and distinct from each other and the other 3 colors (theyrepresent the input and output tensors of the network).We verified that all graphs with the same hash generated in Algorithm 2 wereunique up to isomorphism by running an expensive procedure which enumer-ates all possible permutations to confirm that any graph with a duplicate hash(line 11) is isomorphic to the canonical computational graph. The definition ofgraph invariant implies that graphs with different hashes are non-isomorphic.Thus for our constrained use-case, Algorithm 1 can exactly identify if twocomputational graphs are isomorphic or not.
An adversarial example to the identifiability of the algorithm consists of twonon-isomorphic computational graphs which hash to the same value. One suchexample can be seen in Figure 2 using 10 vertices and 16 edges. The coun-62 3 4 56 7 8 910 12 3 4 56 7 8 910Figure 2: A counterexample using 10 vertices and 16 edges. Vertices 2, 3, 4, 5must be the same color and likewise for 6, 7, 8, 9 (the two sets of vertices canbe the same color). Vertices 1 and 10 can be colored with any color.terexample holds also long as vertices 2, 3, 4, 5 are the same color and likewisefor vertices 6, 7, 8, 9. The two graphs are non-isomorphic by inspection butAlgorithm 1 fails to distinguish between the two. This is because vertices 2, 3,4, 5 all start with the same initial hash due to having the same degree and eachiteration maintains this equivalence because the in and out neighbors share thesame colors (and likewise for vertices 6, 7, 8, 9).An infinite number of similar adversarial graphs can be constructed frompairs of directed non-isomorphic bipartite graphs where all edges point fromone partition to the other and the degree of all vertices within each partition isthe same.
Modifying the algorithm to deal with cases like the counterexample above is thefirst direction for future work.In addition to the counter example discussed above, another possible prob-lem is hash collision. A possible solution is to replace the hash function withstring concatenation, which would cause the iterative “hashes” to grow expo-nentially in length at each iteration. This eliminates the possibility of hashcollision, and the proof of graph invariance still holds.
Acknowledgements
We would like to thank Chris Jones for suggesting Weisfeiler–Lehman color re-finement and finding a counterexample, Esteban Real for reviewing the codeimplementation of the algorithm, and William Chargin for reviewing the nota-tion and proof. 7 eferences [1] L. Babai and L. Kucera. Canonical labelling of graphs in linear averagetime. In , pages 39–46, Oct 1979.[2] L´aszl´o Babai. Graph isomorphism in quasipolynomial time.
CoRR ,abs/1512.03547, 2015.[3] Robert W Robinson. Enumeration of colored graphs.
Journal of Combina-torial Theory , 4(2):181–190, 1968.[4] N. J. A. Sloane. The on-line encyclopedia of integer sequences. https://oeis.org .[5] Boris Weisfeiler and Andrei A. Lehman. A reduction of a graph to a canonicalform and an algebra arising during this reduction.