XGNN: Towards Model-Level Explanations of Graph Neural Networks
XXGNN: Towards Model-Level Explanations of GraphNeural Networks
Hao Yuan [email protected] A&M UniversityCollege Station, Texas, United States
Jiliang Tang [email protected] State UniversityEast Lansing, Michigan, United States
Xia Hu [email protected] A&M UniversityCollege Station, Texas, United States
Shuiwang ji [email protected] A&M UniversityCollege Station, Texas, United States
ABSTRACT
Graphs neural networks (GNNs) learn node features by aggregatingand combining neighbor information, which have achieved promis-ing performance on many graph tasks. However, GNNs are mostlytreated as black-boxes and lack human intelligible explanations.Thus, they cannot be fully trusted and used in certain applicationdomains if GNN models cannot be explained. In this work, wepropose a novel approach, known as XGNN, to interpret GNNsat the model-level. Our approach can provide high-level insightsand generic understanding of how GNNs work. In particular, wepropose to explain GNNs by training a graph generator so thatthe generated graph patterns maximize a certain prediction of themodel. We formulate the graph generation as a reinforcement learn-ing task, where for each step, the graph generator predicts how toadd an edge into the current graph. The graph generator is trainedvia a policy gradient method based on information from the trainedGNNs. In addition, we incorporate several graph rules to encour-age the generated graphs to be valid. Experimental results on bothsynthetic and real-world datasets show that our proposed methodshelp understand and verify the trained GNNs. Furthermore, ourexperimental results indicate that the generated graphs can provideguidance on how to improve the trained GNNs.
CCS CONCEPTS • Computing methodologies → Neural networks ; Artificial in-telligence ; •
Mathematics of computing → Graph algorithms . KEYWORDS
Deep learning, Interpretability, Graph Neural Networks
ACM Reference Format:
Hao Yuan, Jiliang Tang, Xia Hu, and Shuiwang ji. 2020. XGNN: TowardsModel-Level Explanations of Graph Neural Networks. In
Proceedings of the26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected].
KDD ’20, August 23–27, 2020, Virtual Event, CA, USA © 2020 Association for Computing Machinery.ACM ISBN 978-1-4503-7998-4/20/08...$15.00https://doi.org/10.1145/3394486.3403085 (KDD ’20), August 23–27, 2020, Virtual Event, CA, USA.
ACM, New York, NY,USA, 9 pages. https://doi.org/10.1145/3394486.3403085
Graph Neural Networks (GNNs) have shown their effectiveness andobtained the state-of-the-art performance on different graph tasks,such as node classification [11, 37], graph classification [39, 47],and link prediction [46]. In addition, extensive efforts have beenmade towards different graph operations, such as graph convolu-tion [13, 16, 19], graph pooling [20, 44], and graph attention [10,36, 37]. Since graph data widely exist in different real-world appli-cations, such as social networks, chemistry, and biology, GNNs arebecoming increasingly important and useful. Despite their greatperformance, GNNs share the same drawback as other deep learn-ing models; that is, they are usually treated as black-boxes andlack human-intelligible explanations. Without understanding andverifying the inner working mechanisms, GNNs cannot be fullytrusted, which prevents their use in critical applications pertainingto fairness, privacy, and safety [7, 40]. For example, we can train aGNN model to predict the effects of drugs where we treat each drugas a molecular graph. Without exploring the working mechanisms,we do not know what chemical groups in a molecular graph leadto the predictions. Then we cannot verify whether the rules ofthe GNN model are consistent with real-world chemical rules, andhence we cannot fully trust the GNN model. This raises the needof developing interpretation techniques for GNNs.Recently, several interpretations techniques have been proposedto explain deep learning models on image and text data. Dependingon what kind of interpretations are provided, existing techniquescan be categorized into example-level [5, 9, 29, 31, 32, 43, 45, 48]or model-level [8, 24, 25] methods. Example-level interpretationsexplain the prediction for a given input example, by determin-ing important features in the input or the decision procedure forthis input through the model. Common techniques in this cate-gory include gradient-based methods [31, 32, 43], visualizationsof intermediate feature maps [29, 48], and occlusion-based meth-ods [5, 9, 45]. Instead of providing input-dependent explanations,model-level interpretations aim to explain the general behavior ofthe model by investigating what input patterns can lead to a certainprediction, without respect to any specific input example. Inputoptimization [8, 24–26] is the most popular model-level interpreta-tion method. These two categories of interpretation methods aim at a r X i v : . [ c s . L G ] J un xplaining deep models in different views. Since the ultimate goalof interpretations is to verify and understand deep models, we needto manually check the interpretation results and conclude if thedeep models work in our expected way. For example-level meth-ods, we may need to explore the explanations for a large numberof examples before we can trust the models. However, it is time-consuming and requires extensive expert efforts. For model-levelmethods, the explanations are more general and high-level, andhence need less human supervision. However, the explanationsof model-level methods are less precise compared with example-level interpretations. Overall, both model-level and example-levelmethods are important for interpreting and understanding deepmodels.Interpreting deep learning models on graph data become in-creasingly important but is still less explored. To the best of ourknowledge, there is no existing study on interpreting GNNs at themodel-level. The existing study [4, 40] only provides example-levelexplanations for graph models. As a radical departure from exist-ing work, we propose a novel interpretation technique, known asXGNN, for explaining deep graph models at the model-level. Wepropose to investigate what graph patterns can maximize a cer-tain prediction. Specifically, we propose to train a graph generatorsuch that the generated graph patterns can be used to explain deepgraph models. We formulate it as a reinforcement learning prob-lem that at each step, the graph generator predicts how to add anedge to a given graph and form a new graph. Then the generatoris trained based on the feedback from the trained graph modelsusing policy gradient [35]. We also incorporate several graph rulesto encourage the generated graphs to be valid. Note that the graphgeneration part in our XGNN framework can be generalized toany suitable graph generation method, determined by the datasetat hand and the GNNs to be interpreted. Finally, we trained GNNmodels on both real-world and synthetic datasets which can yieldgood performance. Then we employ our proposed XGNN to explainthese trained models. Experimental results show that our proposedXGNN can find the desired graph patterns and explains these mod-els. With our generated graph patterns, we can verify, understand,and even improve the trained GNN models. Graphs are wildly employed to represent data in different real-world domains and graph neural networks have shown promisingperformance on these data. Different from image and text data, agraph is represented by a feature matrix and an adjacency matrix.Formally, a graph G with n nodes is represented by its feature ma-trix X ∈ R n × d and its adjacency matrix A ∈ { , } n × n . Note thatwe assume each node has a d -dimension vector to represent itsfeatures. Graph neural networks learn node features based on thesematrices. Even though there are several variants of GNNs, suchas graph convolution networks (GCNs) [19], graph attention net-works (GATs) [37], and graph isomorphism networks (GINs) [39],they share a similar feature learning strategy. For each node, GNNsupdate its node features by aggregating the features from its neigh-bors and combining them with its own features. We take GCNs asan example to illustrate the neighborhood information aggregation scheme. The operation of GCNs is defined as X i + = f ( D − ˆ AD − X i W i ) , (1)where X i ∈ R n × d i and X i + ∈ R n × d i + are the input and outputfeature matrices of the i th graph convolution layer. In addition,ˆ A = A + I is used to add self-loops to the adjacency matrix, D de-notes the diagonal node degree matrix to normalize ˆ A . The matrix W i ∈ R d i × d i + is a trainable matrix for layer i and is used to performlinear feature transformation and f (·) denotes a non-linear acti-vation function. By stacking j graph convolution layers, the j -hopneighborhood information can be aggregated. Due to its superiorperformance, we incorporate the graph convolution in Equation (1)as our graph neural network operator. Next, we briefly discuss popular model-level interpretation tech-niques for deep learning models on image data, known as inputoptimization methods [8, 24–26]. These methods generally gener-ate optimized input that can maximize a certain behavior of deepmodels. They randomly initialize the input and iteratively updatethe input towards an objective, such as maximizing a class score.Then such optimized input can be regarded as the explanations forthe target behavior. Such a procedure is known as optimization andis similar to training deep neural networks. The main difference isthat in such input optimization techniques, all network parametersare fixed while the input is treated as trainable variables. Whilesuch methods can provide meaningful model-level explanations fordeep models on images, they cannot be directly applied to interpretGNNs due to three challenges. First, the structural information of agraph is represented by a discrete adjacency matrix, which cannotbe directly optimized via back-propagation. Second, for images, theoptimized input is an abstract image and the visualization showshigh-level patterns and meanings. In the case of graphs, the ab-stract graph is not meaningful and hard to visualize. Third, theobtained graphs may not be valid for chemical or biological rulessince non-differentiable graph rules cannot be directly incorporatedinto optimization. For example, the node degree of an atom shouldnot exceed its maximum chemical valency.
To the best of our knowledge, there are only a few existing studiesfocusing on the interpretability of deep graph models [4, 40]. Therecent GNN interpretation tool GNN Explainer [40] proposes toexplain deep graph models at the example-level by learning softmasks. For a given example, it applies soft masks to graph edgesand node features and updates the masks such that the predictionremains the same as the original one. Then some graph edges andnode features are selected by thresholding the masks, and theyare treated as important edges and features for making the pre-diction for the given example. The other work [4] also focuses onthe example-level interpretations of deep graph models. It appliesseveral well-known image interpretation methods to graph mod-els, such as sensitivity analysis (SA) [12], guided backpropagation(GBP) [33], and layer-wise relevance propagation (LRP) [3]. TheSA and GBP methods are based on the gradients while the LRPmethod computes the saliency maps by decomposing the output raph set
Human observation
GNNs
Prediction Maximize
Graphrules
ValidGraph generatorStep 0 Step 1 Step 2 Step 3Rewards Generated graph
Figure 1: Illustrations of our proposed XGNN for graph interpretation via graph generation. The GNNs represent a trainedgraph classification model that we try to explain. All graph examples in the graph set are classified to the third class. Theleft part shows that we can manually conclude the key graph patterns for the third class but it is challenging. The right partshows that we propose to train a graph generator to generate graphs that can maximize the class score and be valid accordingto graph rules. prediction into a combination of its inputs. In addition, both ofthese studies generate input-dependent explanations for individualexamples. To verify and understand a deep model, humans need tocheck explanations for all examples, which is time-consuming oreven not feasible.While input-dependent explanations are important for under-standing deep models, model-level interpretations should not beignored. However, none of the existing work investigates the model-level interpretations of deep graph models. In this work, we arguethat model-level interpretations can provide higher-level insightsand a more general understanding in how a deep learning modelworks. Therefore, we aim at providing model-level interpretationsfor GNNs. We propose a novel method, known as XGNN, to explainGNNs by graph generation such that the generated graphs canmaximize a certain behavior.
Intuitively, given a trained GNN model, the model-level interpre-tations for it should explain what graph patterns or sub-graphpatterns lead to a certain prediction. For example, one possibletype of patterns is known as network motifs that represent simplebuilding blocks of complex networks (graphs), which widely existin graphs from biochemistry, neurobiology, ecology, and engineer-ing [1, 2, 23, 30]. Different motif sets can be found in graphs withdifferent functions [1, 23], which means different motifs may di-rectly relate to the functions of graphs. However, it is still unknownwhether GNNs make predictions based on such motifs or othergraph information. By identifying the relationships between graphpatterns and the predictions of GNNs, we can better understand the models and verify whether a model works as expected. Therefore,we propose our XGNN, which explains GNNs using such graphpatterns. Specifically, in this work, we investigate the model-levelinterpretations of GNNs for graph classification tasks and the graphpatterns are obtained by graph generations.Formally, let f (·) denote a trained GNN classification model, and y ∈ { c , · · · , c ℓ } denote the classification prediction. Given f (·) anda chosen class c i , i ∈ { , · · · , ℓ } , our goal is to investigate what inputgraph patterns maximize the predicted probability for this class.The obtained patterns can be treated as model-level interpretationswith respect to c i . Formally, the task can be defined as G ∗ = argmax G P ( f ( G ) = c i ) , (2)where G ∗ is the optimized input graph we need. A popular way toobtain such optimized input for interpreting image and text modelsis known as input optimization [8, 24–26, 43]. However, as dis-cussed in Section 2.2, such optimization method cannot be appliedto interpret graph models because of the special representationsof graph data. Instead, we propose to obtain the optimized graph G ∗ via graph generation. The general illustration of our proposedmethod is shown in Figure 1. Given a pre-trained graph classifica-tion model, we interpret it by providing explanations for its thirdclass. We may manually conclude the graph patterns from the graphdataset. By evaluating all graph examples in the dataset, we canobtain the graphs that are predicted to be the third class. Then wecan manually check what are the common graph patterns amongthese graphs. For example, the left part of Figure 1 shows that a setof four graphs are classified into the third class. Based on humanobservations, we know that the important graph pattern leadingto the prediction is the triangle pattern consisting of a red node,a yellow node, and a blue node. However, such manual analysiss time-consuming and not applicable for large-scale and complexgraph datasets. As shown in the right part, we propose to train agraph generator to generate graph patterns that can maximize theprediction score of the third class. In addition, we incorporate graphrules, such as the chemical valency check, to encourage valid andhuman-intelligible explanations. Finally, we can analyze the gener-ated graphs to obtain model-level explanations for the third class.Compared with directly manual analysis on the original dataset, ourproposed method generates small-scale and less complex graphs,which can significantly reduce the cost for further manual analysis. Recent advances in graph generation lead to many successful graphgeneration models, such as GraphGAN [38], ORGAN [14], JunctionTree VAE [17], DGMG [22], and Graph Convolutional Policy Net-work (GCPN) [41]. Inspired by these methods, we propose to traina graph generator which generates G ∗ step by step. For each step,the graph generator generates a new graph based on the currentgraph. Formally, we define the partially generated graph at step t as G t , which contains n t nodes. It is represented as a feature matrix X t ∈ R n t × d and an adjacency matrix A t ∈ { , } n t × n t , assumingeach node has a d -dimensional feature vector. Then we define a θ -parameterized graph generator as д θ (·) , which takes G t as input,and outputs a new graph G t + that X t + , A t + = д θ ( X t , A t ) . (3)Then the generator is trained with the guidance from the pre-trained GNNs f (·) . Since generating the new graph G t + from G t is non-differentiable, we formulate the generation procedure asa reinforcement learning problem. Specifically, assuming thereare k types of nodes in the dataset, we define a candidate set C = { s , s , · · · , s k } denoting these possible node types. For ex-ample, in a chemical molecular dataset, the candidate set can be C = { Carbon , Nitroдen , · · · , Oxyдen , Fluorine } . In a social net-work dataset where nodes are not labeled, the candidate set onlycontains a single node type. Then at each step t , based on the par-tially generated graph G t , the generator д (·) generates G t + bypredicting how to add an edge to the current graph G t . Note thatthe generator may add an edge between two nodes in the currentgraph G t or add a node from the candidate set C to the currentgraph G t and connect it with an existing node in G t . Formally, weformulate it as a reinforcement learning problem, which consistsof four elements: state, action, policy, and reward. State : The state of the reinforcement learning environment atstep t is the partially generated graph G t . The initial graph at thefirst step can be either a random node from the candidate set C ormanually designed based on prior domain knowledge. For example,for the dataset describing organic molecules, we can set the initialgraph as a single node labeled with carbon atom since any organiccompound contains carbon generally [28]. Action : The action at step t , denoted as a t , is to generate the newgraph G t + based on the current graph G t . Specifically, given thecurrent state G t , the action a t is to add an edge to G t by determiningthe starting node and the ending node of the edge. Note that thestarting node a t , start can be any node from the current graph G t while the ending node a t , end is selected from the union of thecurrent graph G t and the candidate set C excluding the selected starting node a t , start , denoted as ( G t (cid:208) C )\ a t , start . Note that withthe predefined maximum action step and maximum node number,we can control the termination of graph generation. Policy : We employ graph neural networks to serve as the policy.The policy determines the action a t based on the state G t . Specifi-cally, the policy is the graph generator д θ (·) , which takes G t and C as the input and outputs the probabilities of possible actions. Withthe reward function, the generator д θ (·) can be trained via policygradient [35]. Reward : The reward for step t , denoted as R t , is employed toevaluate the action at step t , which consists of two parts. The firstpart is the guidance from the trained GNNs f (·) , which encouragesthe generated graph to maximize the class score of class c i . Byfeeding the generated graphs to f (·) , we can obtain the predictedprobabilities for class c i and use them as the feedback to update д θ (·) . The second part encourages the generated graphs to be validin terms of certain graph rules. For example, for social networkdatasets, it is may not allowed to add multiple edges between twonodes. In addition, for chemical molecular datasets, the degree ofan atom cannot exceed its chemical valency. Note that for eachstep, we include both intermediate rewards and overall rewards toevaluate the action.While we formulate the graph generation as a reinforcementlearning problem, it is noteworthy that our proposed XGNN is anovel and general framework for interpreting GNNs at the model-level. The graph generation part in this framework can be general-ized to any suitable graph generation method, determined by thedataset at hand and the GNNs to be interpreted. For step t , the graph generator д θ (·) incorporates the partially gener-ated graph G t and the candidate set C to predict the probabilities ofdifferent actions, denoted as p t , start and p t , end . Assume there are n t nodes in G t and k nodes in C , then both p t , start and p t , end arewith n t + k dimensionality. Then the action a t = ( a t , start , a t , end ) is sampled from the probabilities p t = ( p t , start , p t , end ) . Next, wecan obtain the new graph G t + based on the action a t . Specifically,in our generator, we first employ several graph convolutional layersto aggregate neighborhood information and learn node features.Mathematically, it can be written as (cid:98) X = GCNs ( G t , C ) , (4)where (cid:98) X denotes the learnt node features. Note that the graph G t and the candidate set C are combined as the input of GCNs.We merge all nodes in C to G t without adding any edge and thenobtain the new node feature matrix and adjacency matrix. ThenMultilayer Perceptrons (MLPs) are used to predict the probabilitiesof the starting node, p t , start and the action a t , start is sampledfrom this probabilty distribution. Mathematically, it can be writtenas p t , start = Softmax ( MLPs ( (cid:98) X )) , (5) a t , start ∼ p t , start · m t , start , (6)where · means element-wise product and m t , start is to mask outall candidate nodes since the starting node can be only selectedfrom the current graph G t . Let (cid:98) x start denote the features of thenode selected by the start action a t , start . Then conditioned on ode Features Current Graph
Candidates Combine
GCNs MLPs MLPs New Graph
Starting Node Ending Node
Figure 2: An Illustration of our graph generator for processing a single step. Different colors denote different types of node.Given a graph with nodes and a candidate set with 3 nodes, we first combine them together to obtain the feature matrixand the adjacency matrix. Then we employ several GCN layers to aggregate and learn node features. Next, the first MLPspredict a probability distribution from which we sample the starting node. Finally, the second MLPs predict the ending nodeconditioned on the starting node. Note that the black crosses indicates masking out nodes. the selected node, we employ the second MLPs to compute theprobability distribution of the ending node p t , end from which wesample the ending node action a t , end . Note that since the startingnode and the ending node cannot be the same, we apply a mask m t , end to mask out the node selected by a t , start . Mathematically,it can be written as p t , end = Softmax ( MLPs ([ (cid:98) X , (cid:98) x start ])) , (7) a t , end ∼ p t , end · m t , end , (8)where [· , ·] denotes broadcasting and concatenation. In addition, m t , end is the mask consisting of all 1s except the position indicating a t , start . Note that the same graph generator д θ (·) is shared bydifferent time steps, and our generator is capable to incorporategraphs with variable sizes.We illustrate our graph generator in Figure 2 where we showthe graph generation procedure for one step. The current graph G t consists of 4 nodes and the candidate set has 3 available nodes. Theyare combined together to serve as the input of the graph generator.The embeddings of candidate nodes are concatenated to the featurematrix of G t while the adjacency matrix of G t is expanded accord-ingly. Then multiple graph convolutional layers are employed tolearn features for all nodes. With the first MLPs, we obtain the prob-abilities of selecting different nodes as the starting node, and fromwhich we sample the node 1 as the starting node. Then based on thefeatures of node 1 and all node features, the second MLPs predictthe ending node. We sample from the probabilities and select thenode 7 as the ending node, which corresponds to the red node inthe candidate set. Finally, a new graph is obtained by including ared node and connecting it with node 1. The graph generator is trained to generate specific graphs that canmaximize the class score of class c i and be valid to graph rules. Since such guidance is not differentiable, we employ policy gradient [35]to train the generator. According to [21, 42], the loss function forthe action a t at step t can be mathematically written as L д = − R t (L CE ( p t , start , a t , start ) + L CE ( p t , end , a t , end )) , (9)where L CE (· , ·) denotes the cross entropy loss and R t means thereward function for step t . Intuitively, the reward R t indicateswhether a t has a large chance to generate graph with high classscore of class c i and being valid. Hence, the reward R t consists oftwo parts. The first part R t , f is the feedback from the trained model f (·) and the second part R t , r is from the graph rules. Specifically,for step t , the reward R t , f contains both an intermediate rewardand a final graph reward for graph G t + that R t , f = R t , f ( G t + ) + λ (cid:205) mi = R t , f ( Rollout ( G t + )) m , (10)where λ is a hyper-parameter, and the first term is the intermediatereward which can be obtained by feeding G t + to the trained GNNs f (·) and checking the predicted probability for class c i . Mathemati-cally, it can be computed as R t , f ( G t + ) = p ( f ( G t + ) = c i ) − / ℓ, (11)where ℓ denotes the number of possible classes for f (·) . In addition,the second term in Equation (10) is the final graph reward for G t + which can be obtained by performing Rollout [42] m times on theintermediate graph G t + . Each time, a final graph is generatedbased on G t + until termination and then evaluated by f (·) usingEquation (11). Then the evaluations for m final graphs are averagedto serve as the final graph reward. Overall, R t , f is positive whenthe obtained graph tends to yield high score for class c i , and viceversa.In addition, the reward R t , r is obtained from graphs rules andis employed to encourage the generated graphs to be valid andhuman-intelligible. The first rule we employ is that only one edge is lgorithm 1 The algorithm of our proposed XGNN. Given the trained GNNs for graph classification, denoted as f (·) , we try to interpret it and set the target class as c i . Let C define the candidate node set and д (·) mean our graphgenerator. We predefine the maximum generation step as S max and the number of Rollout as m . Define the initial graph as G . for step t in S max do Merge the current graph G t and the candidate set C . Obtain the action a t from the generator д (·) that a t = ( a t , start , a t , end ) with Equation (4-8). Obtain the new graph G t + based on a t . Evaluate G t + with Equation (10-12) and obtain R t . Update the generator д (·) with Equation (9). if R t < then roll back and set G t + = G t . end if end for allowed to be added between any two nodes. Second, the generatedgraph cannot contain more nodes than the predefined maximumnode number. In addition, we incorporate dataset-specific rules toguide the graph generation. For example, in a chemical dataset,each node represents an atom so that its degree cannot exceedthe valency of the corresponding atom. When any of these rulesis violated, a negative reward will be applied for R t , r . Finally, bycombining the R t , f and R t , r , we can obtain the reward for step t that R t = R t , f ( G t + ) + λ (cid:205) mi = R t , f ( Rollout ( G t + )) m + λ R t , r , (12)where λ and λ are hyper-parameters. We illustrate the trainingprocedure in Algorithm 1. Note that we roll back the graph G t + to G t when the action a t is evaluated as not promising that R t < We evaluate our proposed XGNN on both synthetic and real-worlddatasets. We report the summary statistics of these datasets in Ta-ble 1. Since there is no existing work investigating model-levelinterpretations of GNNs, we have no baseline to compare with.Note that existing studies [4, 40] only focus on interpreting GNNsat example-level while ignoring the model-level explanations. Com-paring with them is not expected since these example-level andmodel-level are two totally different interpretation directions.
Synthetic dataset : Since our XGNN generates model-level expla-nations for Deep GNNs, we build a synthetic dataset, known asIs_Acyclic, where the ground truth explanations are available. Thegraphs are labeled based on if there is any cycle existing in thegraph. The graphs are obtained using Networkx software pack-age [15]. The first class refers to cyclic graphs, including grid-likegraphs, cycle graphs, wheel graphs, and circular ladder graphs. Thesecond class denotes acyclic graphs, containing star-like graphs,binary tree graphs, path graphs and full rary tree graphs [34]. Notethat all nodes in this dataset are unlabeled and we focus on investi-gating the ability of GNNs to capture graph structures.
Real-world dataset : We conduct experiments on the real-world
Table 1: Statistics and properties of datasets. Note that theedge number and node number are averaged numbers.
Dataset Classes
Graph classification models : We train graph classification mod-els using these datasets and then try to explain these models. Thesemodels share a similar pipeline that first learns node features usingmultiple layers of GCNs, then obtain graph level embeddings byaveraging all node features, and finally employs fully-connectedlayers to perform graph classification. For the synthetic datasetIs_Acyclic, we use the node degrees as the initial features for allnodes. Then we apply two layers of GCNs with output dimensionsequal to 8, 16 respectively and perform global averaging to obtainthe graph representations. Finally, we employ one fully-connectedlayer as the classifier. Meanwhile, for the real-world dataset MU-TAG, since all nodes are labeled, we employ the correspondingone-hot representations as the initial node features. Then we em-ploy three layers of GCNs with output dimensions equal to 32, 48,64 respectively and average all node features. The final classifiercontains two fully-connected layers in which the hidden dimensionis set to 32. Note that for all GCN layers, we apply the GCN versionshown in Equation (1). In addition, we employ Sigmoid as the non-linear function in GCNs for dataset Is_Acyclic while we use Relu fordataset MUTAG. These models are implemented using Pytorch [27]and trained using Adam optimizer [18]. The training accuracies ofthese models are reported in Table 1, which show that the modelswe try to interpret are models with reasonable performance.
Graph generators : For both datasets, our graph generators sharethe same structure. Our generator first employs a fully-connectedlayer to map node features to the dimension of 8. Then three layersof GCNs are employed with output dimensions equal to 16, 24, 32respectively. The first MLPs consist of two fully-connected layerswith the hidden dimension equal to 16 and a ReLU6 non-linearfunction. The second MLPs also have two fully-connected layersthat the hidden dimension is set to 24 and ReLU6 is applied. Theinitial features for input graphs are the same as mentioned above.For dataset Is_Acyclic, we set λ = λ =
1, and R t , r = − λ = λ =
2, and the total reward R t = − m =
10 times each step to obtain final graph rewards. The modelsare implemented using Pytorch [27] and trained using Adam op-timizer [18] with β = . β = . =0.7544 p=0.9993 p=0.9992 p=0.9999p=0.9998 p=0.9634p=0.9760p=0.9965p=0.9985p=0.9999 Max node: 3
Max node: 4 Max node: 5
Max node: 6 Max node: 7
Figure 3: Experimental results for the synthetic dataset Is_Acyclic. Each row shows our explanations for a certain class thatthe first row corresponds to the class cyclic while the second row explains the class acyclic. In each row, from left to right, wereport the generated graphs with increasing maximum node number limits. In addition, we feed each generated graph to thepre-trained GCNs and report the predicted probability for the corresponding class.
We first conduct experiments on the synthetic dataset Is_Acyclicwhere the ground truth is available. As shown in Table 1, the trainedGNN classifier can reach a promising performance. Since the datasetis manually and synthetically built based on if the graph containsany circle, we can check if the trained GNN classifier makes pre-dictions in such a way. We explain the model with our proposedXGNN and report the generated interpretations in Figure 3. Weshow the explanations for the class “cyclic” in the first row andthe results for the class “acyclic” in the second row. In addition,we also report different generated explanations by setting differentmaximum graph node limits.First, by comparing the graphs generated for different classes,we can easily conclude the difference that the explanations forthe class “cyclic” always contain circles while the results for theclass “acyclic” have no circle at all. Second, to verify whether ourexplanations can maximize the class probability for a certain class,as shown in Equation (2), we feed each generated graph to thetrained GNN classifier and report the predicted probability for thecorresponding class. The results show that our generated graphpatterns can consistently yield high predicted probabilities. Notethat even though the graph obtained for the class “cyclic” withmaximum node number equal to 3 only leads to p = . We also evaluate our proposed XGNN using real-world data. Fordataset MUTAG, there is no ground truth for the interpretations.Since all nodes are labeled as different types of atoms, we investi-gate whether the trained GNN classifier can capture both graphstructures and node labels. We interpret the trained GNN withour proposed method and report selected results in Figure 4 andFigure 5. Note that the generated graphs may not represent realchemical compounds because, for simplicity, we only incorporate asimple chemical rule that the degree of an atom cannot exceed itsmaximum chemical valency. In addition, since nodes are labeled,we can set the initial graphs as different types of atoms.We first set the initial graph as a single carbon atom and reportthe results in Figure 4, since generally, any organic compound con-tains carbon [28]. The first row reports explanations for the class“non-mutagenic” while the second row shows the results for theclass “mutagenic”. We report the generated graphs with differentnode limits and the GNN predicted probabilities. For the class “mu-tagenic”, we can observe that carbon circles and NO are somecommon patterns, and this is consistent with the chemical fact thatcarbon rings and NO chemical groups are mutagenic [6]. Suchobservations indicate that the trained GNN classifier may capturethese key graph patterns to make predictions. In addition, for theclass “non-mutagenic”, we observe the atom Chlorine is widelyexisting in the generated graphs and the combination of Chlorine,Bromine, and Fluorine always leads to “non-mutagenic” predic-tions. By analyzing such explanations, we can better understandthe trained GNN model.We also explore different initial graphs and report the resultsin Figure 5. We fix the maximum node limit as 5 and generateexplanations for the class “mutagenic”. First, no matter how weset the initial graph, our proposed method can always find graph =1.0000 p=0.9999 p=1.0000 p=0.9999p=1.0000 p=1.0000p=0.9999p=1.0000p=1.0000p=0.9999Max node: 3 Max node: 4 Max node: 5 Max node: 6 Max node: 7 Iodine Carbon Bromine Fluorine Oxygen Nitrogen Chlorine
Figure 4: Experimental results for the MUTAG dataset. The first row reports the explanations for the class non-mutagenicwhile the second row shows results for the class mutagenic. Note that different node colors denote different types of atomsand the legend is shown at the bottom of the figure. All graphs are generated with the initial graph as a single Carbon atom. p=1.0000p=0.9999p=1.0000p=1.0000p=1.0000 IodineCarbon BromineFluorineOxygenNitrogen Chlorinep=1.0000 p=1.0000
Figure 5: Experimental results for the MUTAG dataset. We fix the maximum node number limit as 5 and explore differentinitial graphs. Note that all graphs are generated for explaining the mutagenic class. For each generated graph, we show itspredicted probability and corresponding initial graph at the bottom. patterns maximizing the predicted probability of class “mutagenic”.For the first 5 graphs, which means the initial graph is set to asingle node of Carbon, Nitrogen, Oxygen, Iodine, or Fluorine, somegenerated graphs still have common patterns like carbon circle and NO chemical groups. Our observations further confirm that thesekey patterns are captured by the trained GNNs. In addition, wenotice that the generator can still produce graphs with Chlorinewhich are predicted as “mutagenic”, which is contrary to our con-clusion above. If all graphs with Chlorine should be identified as“non-mutagenic”, such explanations show the limitations of trainedGNNs. Then these generated explanations can provide guidancefor improving the trained GNNs, for example, we may place moreemphasis on the graphs Chlorine when training the GNNs. Further-more, the generated explanations may also be used to retrain andimprove the GNN models to correctly capture our desired patterns.Overall, the experimental results show that our proposed interpre-tation method XGNN can help verify, understand, and even helpimprove the trained GNN models. Graphs neural networks are widely studied recently and haveshown great performance for multiple graph tasks. However, graphmodels are still treated as black-boxes and hence cannot be fully trustable. It raises the need of investigating the interpretation tech-niques for graph neural networks. It is still a less explored areawhere existing methods only focus on example-level explanationsfor graph models. However, none of the existing work investigatesthe model-level interpretations of graph models which is more gen-eral and high-level. Hence, in this work, we propose a novel method,XGNN, to interpret graph models in the model-level. Specifically,we propose to find graph patterns that can maximize a certain pre-diction via graph generation. We formulate it as a reinforcementlearning problem and generate graph pattern iteratively. We traina graph generator and for each step, it predicts how to add an edgeinto the current graph. In addition, we incorporate several graphrules to encourage the generated graphs to be valid and human-intelligible. Finally, we conduct experiments on both synthetic andreal-world datasets to demonstrate the effectiveness of our pro-posed XGNN. Experimental results show that the generated graphshelp discover what patterns will maximize a certain predictionof the trained GNNs. The generated explanations help verify andbetter understand if the trained GNNs make a prediction in our ex-pected way. Furthermore, our results also show that the generatedexplanations can help improve the trained models.
CKNOWLEDGMENTS
This work was supported in part by National Science Founda-tion grants DBI-2028361, IIS-1714741, IIS-1715940, IIS-1845081, IIS-1900990 and Defense Advanced Research Projects Agency grantN66001-17-2-4031.
REFERENCES [1] Uri Alon. 2006.
An introduction to systems biology: design principles of biologicalcircuits . Chapman and Hall/CRC.[2] Uri Alon. 2007. Network motifs: theory and experimental approaches.
NatureReviews Genetics
8, 6 (2007), 450.[3] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen,Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations fornon-linear classifier decisions by layer-wise relevance propagation.
PloS one
International Conference on Machine Learn-ing (ICML) Workshops, 2019 Workshop on Learning and Reasoning with Graph-Structured Representations .[5] Piotr Dabkowski and Yarin Gal. 2017. Real time image saliency for black boxclassifiers. In
Advances in Neural Information Processing Systems . 6967–6976.[6] Asim Kumar Debnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan J Shus-terman, and Corwin Hansch. 1991. Structure-activity relationship of mutagenicaromatic and heteroaromatic nitro compounds. correlation with molecular or-bital energies and hydrophobicity.
Journal of medicinal chemistry
34, 2 (1991),786–797.[7] Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of inter-pretable machine learning. arXiv preprint arXiv:1702.08608 (2017).[8] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2009. Visu-alizing higher-layer features of a deep network.
Technical Report, University ofMontreal
Proceedings of the IEEE International Conferenceon Computer Vision . 3429–3437.[10] Hongyang Gao and Shuiwang Ji. 2019. Graph representation learning via hardand channel-wise attention networks. In
Proceedings of the 25th ACM SIGKDDInternational Conference on Knowledge Discovery & Data Mining . 741–749.[11] Hongyang Gao and Shuiwang Ji. 2019. Graph U-Net. In
International conferenceon machine learning . 2083–2092.[12] Muriel Gevrey, Ioannis Dimopoulos, and Sovan Lek. 2003. Review and comparisonof methods to study the contribution of variables in artificial neural networkmodels.
Ecological modelling
Proceedings ofthe 34th International Conference on Machine Learning-Volume 70 . JMLR. org,1263–1272.[14] Gabriel Lima Guimaraes, Benjamin Sanchez-Lengeling, Carlos Outeiral, PedroLuis Cunha Farias, and Alán Aspuru-Guzik. 2017. Objective-reinforced generativeadversarial networks (ORGAN) for sequence generation models. arXiv preprintarXiv:1705.10843 (2017).[15] Aric Hagberg, Pieter Swart, and Daniel S Chult. 2008.
Exploring network structure,dynamics, and function using NetworkX . Technical Report. Los Alamos NationalLab.(LANL), Los Alamos, NM (United States).[16] Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representationlearning on large graphs. In
Advances in neural information processing systems .1024–1034.[17] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. 2018. Junction Tree Varia-tional Autoencoder for Molecular Graph Generation. In
Proceedings of the 35thInternational Conference on Machine Learning . 2323–2332.[18] Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Opti-mization. In
Proceedings of the 3rd International Conference on Learning Represen-tations .[19] Thomas N Kipf and Max Welling. 2017. Semi-supervised classification withgraph convolutional networks. In
Proceedings of the International Conference onLearning Representations .[20] Junhyun Lee, Inyeop Lee, and Jaewoo Kang. 2019. Self-Attention Graph Pooling.In
International Conference on Machine Learning . 3734–3743.[21] Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing Neural Pre-dictions. In
Proceedings of the 2016 Conference on Empirical Methods in NaturalLanguage Processing . 107–117.[22] Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. 2018.Learning deep generative models of graphs. arXiv preprint arXiv:1803.03324 (2018).[23] Ron Milo, Shai Shen-Orr, Shalev Itzkovitz, Nadav Kashtan, Dmitri Chklovskii,and Uri Alon. 2002. Network motifs: simple building blocks of complex networks.
Science
Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition . 4467–4477.[25] Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015. Deep neural networks are easilyfooled: High confidence predictions for unrecognizable images. In
Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition . 427–436.[26] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. 2017. Feature visual-ization.
Distill
2, 11 (2017), e7.[27] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang,Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.2017. Automatic differentiation in PyTorch. In
Proceedings of the InternationalConference on Learning Representations .[28] Spencer L Seager and Michael R Slabaugh. 2013.
Chemistry for today: General,organic, and biochemistry . Cengage learning.[29] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedan-tam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual explanations fromdeep networks via gradient-based localization. In
Proceedings of the IEEE Interna-tional Conference on Computer Vision . 618–626.[30] Shai S Shen-Orr, Ron Milo, Shmoolik Mangan, and Uri Alon. 2002. Networkmotifs in the transcriptional regulation network of Escherichia coli.
Naturegenetics
31, 1 (2002), 64.[31] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep insideconvolutional networks: Visualising image classification models and saliencymaps. arXiv preprint arXiv:1312.6034 (2013).[32] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wat-tenberg. 2017. Smoothgrad: removing noise by adding noise. arXiv preprintarXiv:1706.03825 (2017).[33] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Ried-miller. 2014. Striving for simplicity: The all convolutional net. arXiv preprintarXiv:1412.6806 (2014).[34] James Andrew Storer. 2012.
An introduction to data structures and algorithms .Springer Science & Business Media.[35] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000.Policy gradient methods for reinforcement learning with function approximation.In
Advances in neural information processing systems . 1057–1063.[36] Kiran K Thekumparampil, Chong Wang, Sewoong Oh, and Li-Jia Li. 2018.Attention-based graph neural network for semi-supervised learning. arXivpreprint arXiv:1803.03735 (2018).[37] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, PietroLio, and Yoshua Bengio. 2018. Graph Attention Networks. In
International Confer-ence on Learning Representations . https://openreview.net/forum?id=rJXMpikCZ[38] Hongwei Wang, Jia Wang, Jialin Wang, Miao Zhao, Weinan Zhang, FuzhengZhang, Xing Xie, and Minyi Guo. 2018. GraphGAN: Graph representation learn-ing with generative adversarial nets. In
Thirty-Second AAAI Conference on Artifi-cial Intelligence . 2508–2515.[39] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019. How Powerfulare Graph Neural Networks?. In
International Conference on Learning Representa-tions . https://openreview.net/forum?id=ryGs6iA5Km[40] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec.2019. GNNExplainer: Generating Explanations for Graph Neural Networks. In
Advances in Neural Information Processing Systems 32 . 9244–9255.[41] Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. 2018. Graphconvolutional policy network for goal-directed molecular graph generation. In
Advances in Neural Information Processing Systems . 6410–6421.[42] L Yu, W Zhang, J Wang, and Y Yu. 2017. Seqgan: sequence generative adversarialnets with policy gradient. In
AAAI-17: Thirty-First AAAI Conference on ArtificialIntelligence , Vol. 31. Association for the Advancement of Artificial Intelligence(AAAI), 2852–2858.[43] Hao Yuan, Yongjun Chen, Xia Hu, and Shuiwang Ji. 2019. Interpreting DeepModels for Text Analysis via Optimization and Regularization Methods. In
Thirty-Third AAAI Conference on Artificial Intelligence . 5717–5724.[44] Hao Yuan and Shuiwang Ji. 2020. StructPool: Structured Graph Pooling via Con-ditional Random Fields. In
International Conference on Learning Representations .https://openreview.net/forum?id=BJxg_hVtwH[45] Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolu-tional networks. In
European Conference on Computer Vision . Springer, 818–833.[46] Muhan Zhang and Yixin Chen. 2018. Link prediction based on graph neuralnetworks. In
Advances in Neural Information Processing Systems . 5165–5175.[47] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. 2018. An end-to-end deep learning architecture for graph classification. In
Thirty-Second AAAIConference on Artificial Intelligence . 4438–4445.[48] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.2016. Learning deep features for discriminative localization. In