Effective and Scalable Clustering on Massive Attributed Graphs
Renchi Yang, Jieming Shi, Yin Yang, Keke Huang, Shiqi Zhang, Xiaokui Xiao
EEffective and Scalable Clustering on Massive Attributed Graphs β Technical Report
Renchi Yang
Nanyang Technological [email protected]
Jieming Shi β Hong Kong Polytechnic [email protected]
Yin Yang
Hamad bin Khalifa [email protected]
Keke Huang
National University of [email protected]
Shiqi Zhang
National University of [email protected]
Xiaokui Xiao
National University of [email protected]
ABSTRACT
Given a graph πΊ where each node is associated with a set of at-tributes, and a parameter π specifying the number of output clusters, π -attributed graph clustering ( π -AGC) groups nodes in πΊ into π dis-joint clusters, such that nodes within the same cluster share similartopological and attribute characteristics, while those in differentclusters are dissimilar. This problem is challenging on massivegraphs, e.g. , with millions of nodes and billions of attribute values.For such graphs, existing solutions either incur prohibitively highcosts, or produce clustering results with compromised quality.In this paper, we propose ACMin , an efficient approach to π -AGCthat yields high-quality clusters with costs linear to the size of theinput graph πΊ . The main contributions of ACMin are twofold: (i)a novel formulation of the π -AGC problem based on an attributedmulti-hop conductance quality measure custom-made for this prob-lem setting, which effectively captures cluster coherence in termsof both topological proximities and attribute similarities, and (ii) alinear-time optimization solver that obtains high quality clustersiteratively, based on efficient matrix operations such as orthogonaliterations, an alternative optimization approach, as well as an ini-tialization technique that significantly speeds up the convergenceof ACMin in practice.Extensive experiments, comparing 11 competitors on 6 realdatasets, demonstrate that
ACMin consistently outperforms allcompetitors in terms of result quality measured against groundtruth labels, while being up to orders of magnitude faster. In par-ticular, on the Microsoft Academic Knowledge Graph dataset with265.2 million edges and 1.1 billion attribute values,
ACMin outputshigh-quality results for 5-AGC within 1.68 hours using a singleCPU core, while none of the 11 competitors finish within 3 days.
Node clustering is a fundamental task in graph mining [28, 39,44, 61], and finds important real-world applications, e.g. , commu-nity detection in social networks [12], functional cartography ofmetabolic networks [15], and protein grouping in biological net-works [49]. Traditionally, node clustering is done based on thegraph topology, i.e. , by grouping together well-connected nodes.This approach, however, is often insufficient to obtain high-qualityclusters [13, 22], especially when the graph comes with attributes β This is the full version of the paper appearing in TheWebConf 2021. β Corresponding author. associated to nodes. In such attributed graphs , well-connected nodestend to share similar attributes; meanwhile, nodes with similar at-tributes are also likely to be well-connected, as observed in [26, 27].Therefore, to obtain high-quality node clustering, it is important toconsider both graph topology and node attributes. The resulting attributed graph clustering has use cases such as gene clusteringin biological networks [18], group-oriented marketing in commu-nication networks [54], service/app recommendation, and onlineadvertising in social networks [23, 30].This paper focuses on π - attributed graph clustering ( π -AGC),which takes as input an attributed graph πΊ and a parameter π ,and aims to partition πΊ into π disjoint node clusters πΆ , πΆ , Β· Β· Β· , πΆ π ,such that the nodes within the same cluster πΆ π are not only well-connected to each other, but also share similar attribute values,whereas the nodes in different clusters are distant to each otherand share less attributes. It is highly challenging to devise a π -AGCalgorithm that yields high-quality clusters, especially on massivegraphs, e.g. , with millions of nodes and billions of attribute values.Most existing solutions ( e.g. , [2, 7, 10, 29, 33, 36, 37, 42, 45, 52β54, 62,64, 67, 68]) fail to scale to such large graphs, since they either incurprohibitive computational overhead, or produce clustering resultswith compromised quality. For instance, a common methodology[7, 10, 36, 67] relies on materializing the attribute similarity betweenevery pair of nodes in the input graph πΊ , and, thus, requires π ( π ) space for π nodes, which is infeasible for a graph with numerousnodes. Methods based on probabilistic models ( e.g. , [21, 40, 54, 62,63]) generally require immense costs on large graphs to estimatethe likelihood parameters in their respective optimization programs.Among the faster solutions, some ( e.g. , [7, 33, 37, 42, 45]) reducethe problem to non-attributed graph clustering by re-weightingeach edge ( π’, π£ ) in πΊ based on the attribute similarity betweennodes π’ and π£ . This approach, however, ignores attribute similaritiesbetween nodes that are not directly connected, and, consequently,suffers from severe result quality degradation. Finally, π -AGC couldbe done by first applying attributed network embedding to the inputgraph ( e.g. , [17, 31, 34, 55β57, 60, 66]) to obtain an embedding vectorfor each node, and subsequently feeding the resulting embeddingsto a non-graph method such as π - Means clustering [19, 41]. Thistwo-stage pipeline leads to sub-optimal result quality, however,since the node embedding methods do not specifically target forgraph clustering, as demonstrated in our experiments.Facing the challenge of π -AGC on massive attributed graphs,we propose ACMin (short for Attributed multi-hop Conductance a r X i v : . [ c s . S I] F e b onferenceβ17, July 2017, Washington, DC, USA Yang and Shi, et al. Minimization), a novel solution that seamlessly incorporates bothgraph topology and node attributes to identify high-quality clus-ters, while being highly scalable and efficient on massive graphswith numerous nodes, edges and attributes. Specifically,
ACMin computes π -AGC by solving an optimization problem, in whichthe main objective is formulated based on a novel concept called average attributed multi-hop conductance , which is a non-trivialextension to conductance [6, 61], a classic measure of node clustercoherence. The main idea is to map both node relationships ( i.e. ,connections via edges) and similarities ( i.e. , common attributes) tomotions of a random walker. Then, we show that the correspond-ing concept of conductance in our setting, i.e. , attributed multi-hopconductance, is equivalent to the probability that a random walkerstarting from a node in a cluster (say, πΆ ) terminates at any nodeoutside the cluster πΆ . Accordingly, our goal is to identify a node par-titioning scheme that minimizes the average attributed multi-hopconductance among all π clusters in the result.Finding the exact solution to the above optimization problemturns out to be infeasible for large graphs, as we prove its NP-hardness. Hence, ACMin tackles the problem via an approximatesolution with space and time costs linear to the size of the inputgraph. In particular, there are three key techniques in the
ACMin algorithm. First, instead of actually sampling random walks,
ACMin converts the optimization objective into its equivalent matrix form,and iteratively refines a solution via efficient matrix operations, i.e. , orthogonal iterations [43]. Second, the
ACMin solver appliesan alternative optimization approach and randomized SVD [16] toefficiently generate and refine clustering results. Third,
ACMin in-cludes an effective greedy initialization technique that significantlyspeeds up the convergence of the iterative process in practice.We formally analyze the asymptotic time and space complexitiesof
ACMin , and evaluate its performance thoroughly by comparingagainst 11 existing solutions on 6 real datasets. The quality of aclustering methodβs outputs is evaluated by both (i) comparing themwith ground truth labels, and (ii) measuring their attributed multi-hop conductance, which turns out to agree with (i) on all datasets inthe experiments. The evaluation results demonstrate that
ACMin consistently outperforms its competitors in terms of clusteringquality, at a fraction of their costs. In particular, on the
Flickr dataset,the performance gap between
ACMin and the best competitor is aslarge as 28.6 percentage points, measured as accuracy with respectto ground truth. On the Microsoft Academic Knowledge Graph(
MAG ) dataset with 265.2 million edges and 1.1 billion attributevalues,
ACMin terminates in 1.68 hours for a 5-AGC task, whilenone of the 11 competitors finish within 3 days.The rest of this paper is organized as follows. Section 2 presentsour formulation of the π -AGC problem, based on two novel con-cepts: attributed random walks and attributed multi-hop conduc-tance. Section 3 overviews the proposed solution ACMin and pro-vides the intuitions of the algorithm. Section 4 describes the com-plete
ACMin algorithm and analyzes its asymptotic complexity.Section 5 contains an extensive set of experimental evaluations.Section 6 reviews related work, and Section 7 concludes the paperwith future directions.
Table 1: Frequently used notations.
Notation Description πΊ = ( π, πΈ π , π , πΈ π ) A graph πΊ with node set π , edge set πΈ π , attribute set π , andnode-attribute association set πΈ π . π,π The number of nodes ( i.e. , | π | ) and the number of attributes ( i.e. , | π | ) in πΊ , respectively. π The number of clusters. A , D , R The adjacency, out-degree and attribute matrices of πΊ . P π , P π The topological transition and attributed transition matrices of πΊ , respectively. πΌ, π½ Stopping and attributed branching probabilities. S The attributed random walk probability matrix (see Eq. (2)). F The top- π eigenvectors of S . Y , Ξ¨ ( Y ) A π Γ π node-cluster indicator ( i.e. , NCI) and the average attrib-uted multi-hop conductance ( i.e. , AAMC) of Y (see Eq. (8)). Section 2.1 provides necessary background and defines commonnotations. Section 2.2 describes a random walk model that incorpo-rates both topological proximity and attribute similarity informa-tion. Section 2.3 defines the novel concept of attributed multi-hopconductance, which forms the basis of the objective function in our π -AGC problem formulation, presented in Section 2.4. Let πΊ = ( π , πΈ π , π , πΈ π ) be an attributed graph consisting of a nodeset π with cardinality π , a set of edges πΈ π of size π , each connectingtwo nodes in π , a set of attributes π with cardinality π , and a setof node-attribute associations πΈ π , where each element is a tuple ( π£ π , π π , π€ π,π ) signifying that node π£ π β π is directly associated withattribute π π β π with a weight π€ π,π . Without loss of generality,we assume that each edge ( π£ π , π£ π ) β πΈ π is directed; an undirectededge ( π£ π , π£ π ) is simply converted to a pair of directed edges withopposing directions ( π£ π , π£ π ) and ( π£ π , π£ π ) . A high-level definition ofthe π -AGC problem is as follows. Definition 2.1 ( π -Attributed Graph Clustering ( π -AGC) [67] ). Givenan attributed graph πΊ and the number π of clusters, π -AGC aims topartition the node set π of πΊ into disjoint subsets: πΆ , πΆ , Β· Β· Β· , πΆ π ,such that (i) nodes within the same cluster πΆ π are close to each other,while nodes between any two clusters πΆ π , πΆ π are distant from eachother; and (ii) nodes within the same cluster πΆ π have homogeneousattribute values, while the nodes in different clusters may havediverse attribute values.Note that the above definition does not include a concrete opti-mization objective that quantifies node proximity and attribute ho-mogeneity. As explained in Sections 2.2-2.4, the design of effectivecluster quality measures is non-trivial, and is a main contributionof this paper. The problem formulation is completed later in Section2.4 with a novel objective function.Regarding notations, we denote matrices in bold uppercase, e.g. , M . We use M [ π ] to denote the π -th row vector of M , and M [ : , π ] todenote the π -th column vector of M . In addition, we use M [ π, π ] todenote the element at the π -th row and π -th column of M . Given anindex set I , we let M [I] (resp. M [ : , I] ) be the matrix block of M that contains the row (resp. column) vectors of the indices in I . Following common practice in the literature [54, 60], we assume that the attributeshave already been pre-processed, e.g. , categorical attributes such as marital status areone-hot encoded into binary ones.2 ffective and Scalable Clustering on Massive Attributed Graphs Conferenceβ17, July 2017, Washington, DC, USA v v v v v v v v v v v v v v r r r r r r v v v v v v v r r r (a) Clusters πΆ ,πΆ v v v v v v v v v v v v v v r r r r r r v v v v v v v r r r (b) Clusters πΆ β² ,πΆ β² Figure 1: Example attributed graph and clustering schemes.
Let A be the adjacency matrix of the input graph πΊ , i.e. , A [ π£ π , π£ π ] = ( π£ π , π£ π ) β πΈ π , otherwise A [ π£ π , π£ π ] =
0. Let D be the diagonalout-degree matrix of πΊ , i.e. , D [ π£ π , π£ π ] = (cid:205) π£ π β π A [ π£ π , π£ π ] . We definethe topological transition matrix of πΊ as P π = D β A . Furthermore,we define an attribute matrix R β R π Γ π , such that R [ π£ π , π π ] = π€ π,π is the weight associated with the entry ( π£ π , π π , π€ π π ) β πΈ π . We referto R [ π£ π ] as node π£ π βs attribute vector . Also, let π ππ’π‘ ( π£ π ) and π ππ ( π£ π ) represent the out-degree and in-degree of node π£ π in πΊ , respectively.Table 1 lists the frequently used notations throughout the paper. Random walk is an effective model for capturing multi-hop rela-tionships between nodes in a graph [32]. Common definitions ofrandom walk, e.g. , random walk with restart (RWR) [25, 47], con-sider only graph topology but not node attributes. Hence, we devisea new attributed random walk model that seamlessly integratestopological proximity and attribute similarity between nodes in acoherent framework, which plays a key role in our formulation ofthe π -AGC problem, elaborated later.Given an attributed graph πΊ , we first define the attributed transi-tion probability and topological transition probability between a pairof nodes π£ π and π£ π in πΊ . We say that π£ π and π£ π are connected viaattribute π π₯ , iff. π£ π and π£ π have a common attribute π π₯ . For example,in Figure 1, nodes π£ and π£ are connected via three attributes π β π (shown in blue dashed lines). The attributed transition probabilityfrom π£ π to π£ π via π π₯ is defined as R [ π£ π ,π π₯ ]Β· R [ π£ π ,π π₯ ] (cid:205) π£π β π (cid:205) ππ¦ β π R [ π£ π ,π π¦ ]Β· R [ π£ π ,π π¦ ] , whichcorresponds to the motion of the random walker that hops from π£ π to π£ π through a βbridgeβ π π₯ . Accordingly, we define the attributedtransition probability matrix P π of πΊ as: P π [ π£ π , π£ π ] = R [ π£ π ]Β· R [ π£ π ] β€ (cid:205) π£π β π R [ π£ π ]Β· R [ π£ π ] β€ . (1)Intuitively, P π [ π£ π , π£ π ] models the attributed transition probabilityfrom π£ π to π£ π via any attribute in π .Meanwhile, following conventional random walk definitions, forany two nodes π£ π and π£ π that are directly connected by an edge in πΊ , i.e. , ( π£ π , π£ π ) β πΈ π , the topological transition probability P π [ π£ π , π£ π ] from π£ π to π£ π is π ππ’π‘ ( π£ π ) , where π ππ’π‘ ( π£ π ) is the out-degree of node π£ π .The topological transition matrix P π can then be obtained by P π = D β A , where D and A are the node degree and adjacency matricesof πΊ , respectively. Based on the above concepts, we formally defineattributed random walk as follows. Definition 2.2 (Attributed Random Walk).
Given an attributedgraph πΊ , a stopping probability πΌ β ( , ) , and an attributed branch-ing probability π½ β ( , ) , an attributed random walk starting fromnode π£ π in πΊ performs one of the following actions at each step: (1) with probability πΌ , stop at the current node (denoted as π£ π ),(2) with probability 1 β πΌ , jump to another node π£ π as follows:(a) (attributed transition) with probability π½ , jump to anothernode π£ π via any attribute with probability P π [ π£ π , π£ π ] ,(b) (topological transition) with probability 1 β π½ , jump to anout-neighbor π£ π of π£ π with probability P π [ π£ π , π£ π ] .Based on Definition 2.2, the following lemma shows how todirectly compute the probability S [ π£ π , π£ π ] that an attributed randomwalk starting from node π£ π stops at node π£ π .Lemma 2.3. Given an attributed graph πΊ , the probability that anattributed random walk starting from node π£ π stops at node π£ π is S [ π£ π , π£ π ] = πΌ (cid:205) β β = ( β πΌ ) β Β· (( β π½ ) Β· P π + π½ Β· P π ) β [ π£ π , π£ π ] . (2)Note that computing S directly using Eq. (2) is inefficient, whichinvolves sampling numerous attributed random walks. Instead, theproposed solution ACMin , presented later, computes the probabil-ities in S based on an alternative matrix representation, withoutsimulating any attributed random walk. Conductance is widely used to evaluate the quality of a node clusterin a graph [6, 61]. A smaller conductance indicates a more coherentcluster, and vice versa. Specifically, given a cluster πΆ of graph πΊ ,the conductance of πΆ , denoted as (cid:98) Ξ¦ ( πΆ ) , is defined as follows. (cid:98) Ξ¦ ( πΆ ) = | cut ( πΆ ) | min { vol ( πΆ ) , vol ( π \ πΆ ) } , (3)where vol ( πΆ ) = (cid:205) π£ π β πΆ π ππ’π‘ ( π£ π ) , i.e. , the sum of the out-degreesof all nodes in πΆ , and cut ( πΆ ) = {( π£ π , π£ π ) | π£ π β πΆ, π£ π β π \ πΆ } , i.e. ,the set of outgoing edges with an endpoint in πΆ and the otherin π \ πΆ . Intuitively, (cid:98) Ξ¦ ( πΆ ) is smaller when πΆ has fewer outgoingedges linking to the nodes outside the cluster ( i.e. , lower inter-cluster connectivity), and more edges with both endpoints within πΆ (higher intra-cluster connectivity).In our setting, the classic definition of conductance (cid:98) Ξ¦ ( πΆ ) is in-adequate, since it captures neither attribute information nor multi-hop relationships between nodes. Figure 1 illustrates an example inwhich (cid:98) Ξ¦ ( πΆ ) leads to counter-intuitive cluster quality measurements.The example contains nodes π£ - π£ and attributes π - π . Suppose thatwe aim to partition πΊ into two clusters. As shown in Figure 1b, node π£ is mutually connected to nodes π£ and π£ , and also shares manyattributes ( i.e. , π , π , and π ) and neighbors ( i.e. , π£ and π£ ) withnode π£ ; in contrast, among nodes π£ - π£ , π£ is only mutually con-nected to π£ , and share no common attributes with them. Imaginethat this is in a social media setting where each node representsa user, and each edge indicates a follow relationship; then, π£ isclearly closer to nodes π£ - π£ than to nodes π£ - π£ , due to its strongerconnections and shared attributes to the former group. However,the conductance definition in Eq. (3) leads to the counter-intuitiveconclusion that favors the clustering scheme πΆ = { π£ , π£ , π£ } and πΆ = { π£ , π£ , π£ , π£ } in Figure 1a over πΆ β² and πΆ β² in Figure 1b, sincethe conductance (cid:98) Ξ¦ ( πΆ ) = (cid:98) Ξ¦ ( πΆ ) = β€ (cid:98) Ξ¦ ( πΆ β² ) = (cid:98) Ξ¦ ( πΆ β² ) = .To address the above issue, we propose a new measure of clusterquality dubbed attributed multi-hop conductance , which can beviewed as an adaptation of conductance to the problem setting All proofs appear in Appendix A3 onferenceβ17, July 2017, Washington, DC, USA Yang and Shi, et al. of π -AGC. Specifically, given a cluster πΆ of an attributed graph πΊ ,suppose that we perform π π attributed random walks from eachnode π£ π in πΆ . Let π€ ( π£ π , π£ π ) be the number of walks from π£ π stoppingat π£ π . Then, we can use the following quantity instead of Eq. (3) asa measure of cluster coherence: E (cid:20) (cid:205) π£π β πΆ,π£π β π \ πΆ π€ ( π£ π ,π π ) π π Β·| πΆ | (cid:21) = (cid:205) π£π β πΆ,π£π β π \ πΆ E [ π€ ( π£π,π£π ) ππ ]| πΆ | . Intuitively, the above value quantifies the expected portion of the at-tributed random walks escaping from πΆ , i.e. , stopping at any outsidenode π£ π β π \ πΆ . Hence, the smaller the number of escaped walks,the higher the cluster coherence. Further, observe that E [ π€ ( π£ π ,π£ π ) π π ] corresponds to the probability that an attributed random walk start-ing from π£ π terminates at π£ π , i.e. , S [ π£ π , π£ π ] in Eq. (2). Accordingly,we arrive at the following definition of attributed multi-hop con-ductance Ξ¦ ( πΆ ) . Definition 2.4 (Attributed Multi-Hop Conductance).
Given a clus-ter πΆ of an attributed graph πΊ , the attributed multi-hop conductance Ξ¦ ( πΆ ) of the cluster πΆ is defined as Ξ¦ ( πΆ ) = (cid:205) π£ π β πΆ,π£ π β π \ πΆ S [ π£ π ,π£ π ]| πΆ | . (4) Given an input attributed graph πΊ , we aim to partition all nodes into π disjoint clusters πΆ , πΆ , Β· Β· Β· , πΆ π , such that their average attributedmulti-hop conductance (AAMC) π of the π clusters is minimized, asfollows. π β = min πΆ ,πΆ , Β·Β·Β· ,πΆ π (cid:205) ππ = Ξ¦ ( πΆ π ) π . (5)The above objective, in combination with Definition 2.1, com-pletes our formuation of the π -AGC problem. As an example, inFigure 1, let πΌ = . , π½ = .
5. Then, we have Ξ¦ ( πΆ ) = . , Ξ¦ ( πΆ ) = .
125 for the clusters πΆ , πΆ in Figure 1a, and Ξ¦ ( πΆ β² ) = . , Ξ¦ ( πΆ β² ) = .
185 for the clusters πΆ β² , πΆ β² in Figure 1b. The AAMC values of thesetwo clustering results are Ξ¦ ( πΆ )+ Ξ¦ ( πΆ ) = . > Ξ¦ ( πΆ β² )+ Ξ¦ ( πΆ β² ) = . πΆ β² , πΆ β² are a better clustering of πΊ , whichagrees with our intuition explained in Section 2.3. This section provides a high-level overview of the proposed solution
ACMin for π -AGC computation, and explains the intuitions behindthe algorithm design. The complete ACMin method is elaboratedlater in Section 4.First, we transform the optimization objective in Eq. (5) to anequivalent form that is easier to analyze. For this purpose, weintroduce the following binary node-cluster indicator (NCI) Y β π Γ π to represent a clustering result: Y [ πΆ π , π£ π ] = (cid:40) π£ π β πΆ π ,0 π£ π β π \ πΆ π , (6)where πΆ π is the π -th cluster and π£ π is the π -th node in the node set π of the input graph πΊ . Based on NCI Y , the following lemma presentsan equivalent form of the AAMC objective function in Eq. (5).Lemma 3.1. Given a clustering result πΆ , πΆ , Β· Β· Β· , πΆ π , representedby NCI Y , the AAMC of πΆ , πΆ , Β· Β· Β· , πΆ π can be obtained by: (cid:205) ππ = Ξ¦ ( πΆ π ) π = π Β· trace ((( YY β€ ) β Y ) Β· ( I β S ) Β· (( YY β€ ) β Y ) β€ ) (7) Then, our optimization objective for π -AMC is transformed to: π β = min Y β π Γ π Ξ¨ ( Y ) (8)where Ξ¨ ( Y ) = π Β· trace ((( YY β€ ) β Y ) Β· ( I β S ) Β· (( YY β€ ) β Y ) β€ ) Note that Eq. (8) is equivalent to Eq. (5), and yet the formeris more friendly to analysis. In particular, we have the followingnegative result.Lemma 3.2.
The optimization problem of finding the optimal val-ues of Y from the objective function in Eq. (8) is NP-hard. β‘ Accordingly, to devise a solution for π -AGC on massive graphs,we focus on approximate techniques for optimizing our objective.Observe that the NP-hardness of our objective function in Eq. (8)is due to the requirement that elements of the NCI are binary, i.e. , Y β π Γ π . Thus, we apply a common trick that relaxes NCI ele-ments from binary to fractional , i.e. , Y β R π Γ π . The following lemmashows a sufficient condition to find the optimal values of fractionalNCI Y : when the row vectors of ( YY β€ ) β Y are the top- π eigen-vectors of matrix S (defined in Lemma 2.3), i.e. , the π eigenvectorscorresponding to the π largest eigenvalues of S .Lemma 3.3. Assume that we relax the requirement Y β π Γ π to Y β R π Γ π . Let F β R π Γ π denote the matrix consisting of the top- π eigenvectors of S . Then, the optimal value of Y for the objective min Y β R π Γ π Ξ¨ ( Y ) is obtained when ( YY β€ ) β Y = F , which leads toa value of Ξ¨ ( Y ) no larger than the solution of the original optimalobjective π β in Eq. (8) . β‘ The optimal value of fractional Y , however, does not directlycorrespond to a clustering solution, which requires the NCI to bebinary. The following lemma points to a way to obtain a goodapproximation of the optimal binary Y β π Γ π .Lemma 3.4. Given the top- π eigenvectors F of S , if we obtain abinary NCI Y that satisfies min β₯ XF β ( YY β€ ) β Y β₯ πΉ s.t. Y β π Γ π , X β€ X = I , (9) then Ξ¨ ( Y ) β π β in Eq. (8) . β‘ Based on Lemmata 3.3 and 3.4, to approximate the optimal binaryNCI Y , we can first compute the top- π eigenvectors F of S , andthen solve for the best NCI Y that optimizes Eq. (9). The proposedalgorithm ACMin follows this two-step approach.There remain two major challenges in realizing the above idea: β’ How to compute F for large graphs. Note that it is prohibitively ex-pensive to compute F directly by performing eigen-deompositionon a materialized matrix S (defined in Eq. (2)), which would con-sume Ξ© ( π ) space and Ξ© ( π π ) time. β’ Given F , how to efficiently compute Y β π Γ π based on Eq. (9),which in itself is a non-trivial optimization problem.To address the above challenges, the proposed method ACMin contains three key techniques. First, to compute the top- π eigen-vectors F of S , ACMin employs a scalable, iterative process basedon orthogonal iterations [43], which does not need to materialize S .Second, to find the best NCI Y , ACMin applies an alternative opti-mization approach and randomized SVD [16] to efficiently optimizeEq. (9). Third, to acclerate the above iterative processes,
ACMin ffective and Scalable Clustering on Massive Attributed Graphs Conferenceβ17, July 2017, Washington, DC, USA Algorithm 1:
ACMin
Input:
πΊ, π, πΌ, π½ . Output: Y . Compute (cid:98) R by Eq. (10); Y β InitNCI ( P π , (cid:98) R , R , πΌ, π½ ) ; F β ( Y Y β€ ) β Y ; Y β Y ; π β AppoxAAMC ( P π , (cid:98) R , R , πΌ, π½, Y ) ; for β β to π‘ π do Z β β ( β π½ ) Β· P π F β€ β β + π½ Β· (cid:98) R ( R β€ F β€ β β ) ; F β β QR ( Z β ) ; if F β = F β β then break ; Y β β GenNCI ( F β ) ; π β β AppoxAAMC ( P π , (cid:98) R , R , πΌ, π½, Y β ) ; if π β < π then π β π β , Y β Y β ; return Y ; includes an effective greedy algorithm to compute a high-qualityinitial value of Y , which significantly speeds up convergence inpractice. Overall, ACMin only requires space and time linear to thesize of the input graph πΊ . The next section presents the detailed ACMin algorithm and complexity analysis.
ACMin
ALGORITHM
This section presents the detailed
ACMin algorithm, shown in Al-gorithm 1. In the following, Sections 4.1-4.3 detail the three mostimportant components of
ACMin : the computation of top- π eigen-vectors F , binary NCI Y , and a greedy initialization of Y , respec-tively. Section 4.4 summarizes the complete ACMin algorithm andanalyzes its complexity. π Eigenvectors F Recall from Section 3 that
ACMin follows a two-step strategy thatfirst computes F , the top- π eigenvectors of S (Eq. (2)). Since materi-alizing S is infeasible on large graphs, this subsection presents ouriterative procedure for computing F without materializing S , whichcorresponds to Lines 6-9 of Algorithm 1.First of all, the following lemma reduces the problem of comput-ing F to computing the top- π eigenvectors of ( β π½ ) Β· P π + π½ Β· P π .Lemma 4.1. Let F be the top- π eigenvectors of ( β π½ ) Β· P π + π½ Β· P π .Then, F is also the top- π eigenvectors of S . β‘ Computing the exact top- π eigenvectors of ( β π½ ) Β· P π + π½ Β· P π is still rather challenging, however, since materializing P π also re-quires Ξ© ( π ) space. To tackle this issue, ACMin applies orthogonaliterations [43], as follows. First,
ACMin computes a normalizedattribute vector (cid:98) R [ π£ π ] for each node π£ π in the graph using the fol-lowing equation, leading to matrix (cid:98) R (Line 1 in Algorithm 1). (cid:98) R [ π£ π ] = R [ π£ π ] R [ π£ π ]Β· r β€ β π£ π β π , where r = (cid:205) π£ π β π R [ π£ π ] . (10)Comparing above equation with Eq. (1), it follows that P π = (cid:98) RR β€ .Hence, ( β π½ ) Β· P π + π½ Β· P π in Lemma 4.1 can be transformed to ( β π½ ) Β· P π + π½ Β· (cid:98) RR β€ , eliminating the need to materialize P π .Next, suppose that we are currently at the start of the β -th it-eration (Line 6 of Algorithm 1) with F β β obtained in previousiteration. Note that in the first iteration, F is computed from an Algorithm 2:
GenNCI
Input: F . Output: Y . X β² β I , X β I ; for β β to π‘ π do for π β to π do Compute πΎ π by Eq. (15) ; for π£ π β π do Pick π π by Eq. (14); Y [ : , π£ π ] β , Y [ π π , π£ π ] β U , πΊ , V β SVD ( ( YY β€ ) β YF β€ ) ; X β² β X , X β U Β· V β€ ; if X = X β² then break ; return Y ; initial value Y of Y , elaborated in Section 4.3. ACMin computes Z β = (( β π½ ) Β· P π + π½ Β· (cid:98) RR β€ ) F β€ β β = ( β π½ ) Β· P π F β€ β β + π½ Β· (cid:98) R Β· ( R β€ F β€ β β ) (Line 7 of the algorithm), which can be done in π ( π Β· (| πΈ π | + | πΈ π |)) time. Then, ACMin employs QR decomposition [9] (Line 8) to de-compose Z β into two matrices: F β and π² β , such that Z β = F β€ β Β· π² β ,where π² β is an upper-triangular matrix, and F β is orthogonal ( i.e. , F β F β€ β = I ). Clearly, the QR decomposition step can be done in π ( ππ ) time, leading to π ( π Β· (| πΈ π | + | πΈ π |) + ππ ) total time for oneiteration in the computation of F .Suppose that F β converges in iteration β = π‘ π , i.e. , F π‘ π is the sameas F π‘ π β (Line 9). Then, we have Z β = (( β π½ ) Β· P π + π½ Β· P π ) F β€ π‘ π = F β€ π‘ π Β· π² π‘ π . Considering that π² β is an upper-triangular matrix, and F β is orthogonal ( i.e. , F β F β€ β = I ), according to [43], we conclude that F π‘ π is the top- π eigenvectors of ( β π½ ) Β· P π + π½ Β· P π and the diagonalelements of π² π‘ π are the top- π eigenvalues. According to Lemma 4.1,the row vectors of F π‘ π are also the top- π eigenvectors of S .Note that throughout the process for computing F , there is nomaterialization of either S or P π , which avoids the correspondingquadratic space requirement. Meanwhile, with a constant π , eachiteration takes time linear to the size of the input graph πΊ , whichis far more scalable than decomposing S directly. In practice, thenumber of required iterations can be significantly reduced througha good initialization, detailed later in Section 4.3. Y As described in Section 3, after obtaining the top- π eigenvectors F of S , ACMin proceeds to compute the binary NCI Y by solvingthe optimization problem in Eq. (9). In Algorithm 1, this is done inLines 10-12. Note that in ACMin , the computation of Y is performedonce in every iteration for computing F , rather than only once afterthe final value of F is obtained. This is because our algorithm isapproximate, and, thus, the final value of F does not necessarilylead to the best clustering quality, measured by AAMC (Section2.4). Hence, ACMin computes Y β and the corresponding AAMC π β for each iteration β , and udpate the current best result Y and π whenever a better result is found (Lines 11-12 in Algorithm 1).Next we clarify the GenNCI function, shown in in Algorithm 2,which computes the binary NCI Y β β π Γ π with F β in the currentiteration β . First, based on properties of matrix trace, we transformthe optimization objective in Eq. (9), as follows. β₯ XF β ( YY β€ ) β Y β₯ πΉ = π β Β· trace (( YY β€ ) β YF β€ X β€ ) . (11) onferenceβ17, July 2017, Washington, DC, USA Yang and Shi, et al. GenNCI applies an alternative optimization approach to minimizeEq. (11). Specifically, the algorithm updates two variables, X and Y in an alternating fashion, each time fixing one of them and updatingthe other, according to the following rules. Updating Y with X fixed. Given F , according to Eq. (11), with X fixed, the function to optimize becomes:max Y β πΎ Γ π trace (( YY β€ ) β YF β€ X β€ ) (12)Let M = F β€ X β€ . Eq. (12) is equivalent tomax Y β π Γ π (cid:205) π£ π β π (cid:205) ππ = (cid:32) Y [ π π , π£ π ] Β· M [ π£ π ,π π ] βοΈ(cid:205) π£π β π Y [ π π ,π£ π ] (cid:33) . (13)Since Y β π Γ π , for each column Y [ : , π£ π ] ( π£ π β π ), we update theentry at π π of Y [ : , π£ π ] ( i.e. , Y [ π π , π£ π ] ) to 1, and 0 everywhere else,where π π is picked greedily as follows: π π = arg max β€ π π β€ π (cid:34) ( β Y [ π π ,π£ π ]) Β· M [ π£ π ,π π ] βοΈ πΎ π + + Y [ π π ,π£ π ]Β· M [ π£ π ,π π ] πΎ π (cid:35) , (14)where πΎ π = βοΈ(cid:205) π£ π§ β π Y [ π π , π£ π§ ] , (15)meaning that we always update each column Y [ : , π£ π ] ( π£ π β π ) suchthat the objective function in Eq. (12) is maximized. Since both M = F β€ X β€ and πΎ π can be precomputed at the beginning of eachiteration, which takes π ( ππ ) time, it takes π ( ππ ) time to updatethe whole Y in each iteration. Updating X with Y fixed. Given F , according to Eq. (11), with Y fixed, the function to optimize becomes:max X β€ X = I trace (( YY β€ ) β YF β€ X β€ ) (16)The following lemma shows that the optimal X in Eq. (16) canbe obtained via singular value decomposition (SVD) of matrix ( YY β€ ) β YF β€ .Lemma 4.2. The optimal solution to the objective function in Eq. (16) is X = UV β€ , where U and V are the left and right singular vectors of ( YY β€ ) β YF β€ respectively. β‘ To compute SVD of ( YY β€ ) β YF β€ β R π Γ π , GenNCI employs therandomized SVD algorithm [16], which finishes in π ( π ) time.With the above update rules for X and Y respectively, GenNCI (Algorithm 2) iteratively updates X and Y for a maximum of π‘ π iterations (Lines 2-9). In our experiments, we found that setting π‘ π to 50 usually leads to satisfactory performance. Note that theiterations may converge earlier than π‘ π iterations (Line 9). Since up-dating Y and X takes π ( ππ ) and π ( π ) time respectively, GenNCI terminates within π ( π‘ π Β· ( ππ + π )) time. Next we clarify the computation of the initial value Y of the NCI(Line 2 of Algorithm 1). If we simply assign random values toelements of Y , the iterative process in ACMin from Lines 6 to12 would converge slowly. To address this issue, we propose aneffective greedy initialization technique
InitNCI , which usuallyleads to fast convergence of
ACMin in practice, as demonstrated inour experiments in Section 5.4.
Algorithm 3:
InitNCI
Input: P π , πΌ, π½ . Output: Y . Y β , π π β β ; π β² π β { π£ π , π£ π , Β· Β· Β· , π£ π π } where π£ π π is the node in π with π -thlargest in-degree; π· β I [ : ,π β² π ] , π‘ β πΌ ; for β β to π‘ do π· β β ( β πΌ ) Β· P π π· β β + π· ; π· π‘ β πΌ Β· π· π‘ ; for π£ π β π β² π do compute (cid:205) π£ π β π π· π‘ [ π£ π , π£ π ] ; Select the top- π nodes π£ π β π β² π with the largest (cid:205) π£ π β π π· π‘ [ π£ π , π£ π ] into π π as the π center nodes; for π£ π β π do select π£ π π β π π with the largest π· π‘ [ π£ π , π£ π π ] , and set Y [ π, π£ π ] β return Y ; Given a cluster πΆ , recall that its attributed multi-hop conductance Ξ¦ ( πΆ ) (Eq. (4)) is defined based on the intuition that Ξ¦ ( πΆ ) is lowerwhen an attributed random walk from any nodes in πΆ is more likelyto stop at a node within πΆ . Further, we observe that in practice, ahigh-quality cluster πΆ tends to have high intra-cluster connectivityvia certain center nodes within πΆ , and such a center node usuallyhas high in-degree ( i.e. , many in-neighbors). In other words, thenodes belonging to the same cluster tend to have many paths tothe center node of the cluster, and consequently, a random walkwith restart (RWR) [25, 47] within a cluster is more likely to stopat the center node [46]. Based on these intuitions, we propose toleverage graph topology ( i.e. , π and πΈ π of the input attributedgraph πΊ ) as well as RWR to quickly identify π possible clustercenter nodes, π π = { π£ π , π£ π , Β· Β· Β· , π£ π π } β π , and greedily initializeNCI Y by grouping the nodes in π to a center node according totheir topological relationships to the center node.Algorithm 3 presents the pseudo-code of InitNCI . After initializ-ing Y to a π Γ π zero matrix and π π to an empty set at Line 1, themethod first selects from π a candidate set π β² π of size 5 π (Line 2),which consists of the top- ( π ) nodes with the largest in-degrees.The nodes in π β² π serve as the candidate nodes for the π center nodesto be detected. Then we compute the π‘ -hop RWR value π· π‘ [ π£ π , π£ π ] from every node π£ π β π to every node π£ π β π β² π from Lines 3 to 5according to the following equation [59]. π· π‘ = (cid:205) π‘β = πΌ ( β πΌ ) β P βπ Β· I [ : , π β² π ] (17)In particular, we set π‘ = πΌ at Line 3, which is the expected lengthof an RWR, and is usually sufficient for our purpose. If π· π‘ [ π£ π , π£ π ] is large, it means that the random walks starting from π£ π are morelikely to stop at π£ π , which matches our aforementioned intuition ofpossible cluster center nodes.Then, at Line 6, for each candidate center node π£ π β π β² π , wecompute the sum of π· π‘ [ π£ π , π£ π ] from all nodes π£ π β π to π£ π . If π£ π has larger (cid:205) π£ π β π π· π‘ [ π£ π , π£ π ] , it indicates that the random walksstarting from any nodes in π are more likely to stop at π£ π . There-fore, at Line 7, we select the top- π nodes π£ π β π β² π with the largest (cid:205) π£ π β π π· π‘ [ π£ π , π£ π π ] as the π possible center nodes in π π . At Line 8,for each node π£ π β π , we select the center node π£ π π β π π with thelargest π· π‘ [ π£ π , π£ π π ] and greedily group π£ π and π£ π π into the same π -thcluster by setting Y [ π, π£ π ] to 1, completing the computation of Y . ffective and Scalable Clustering on Massive Attributed Graphs Conferenceβ17, July 2017, Washington, DC, USA Algorithm 4:
AppoxAAMC
Input: P π , (cid:98) R , R , πΌ, π½, Y . Output: π . H β ( YY β€ ) β Y , π‘ β πΌ ; for β = to π‘ do H β β ( β πΌ ) Β· ( ( β π½ ) Β· P π H β€ β β + π½ Β· (cid:98) R ( R β€ H β€ β β )) + H ; π β π Β· (cid:205) ππ = H [ π ] Β· ( H β€ [ π ] β πΌ Β· H π‘ [ : , π ]) ; return π ; Note that Line 2 in Algorithm 3 takes π ( π + π log ( π )) time, andthe computation of π· π‘ requires π ( ππΌ Β·| πΈ π |) time. Therefore, InitNCI runs in π ( ππΌ Β· | πΈ π |) time. ACMin
Algorithm and Analysis
Algorithm 1 summarizes the pseudo-code of
ACMin , which takesas input an attributed graph πΊ , the number of clusters π , randomwalk stopping probability πΌ , and attributed branching probability π½ (defined in Definition 2.2). Initially (Line 1), ACMin computesmatrix (cid:98) R , explained in Section 4.1. Then (Line 2), ACMin computesan initial value Y for Y via InitNCI (Algorithm 3), and derives thecorresponding value F for F according to Lemma 3.3 in Line 3.Next (Line 5), we invoke AppoxAAMC (Algorithm 4) that uses Y to compute π , the best AAMC obtained so far. Note that the exactAAMC π = Ξ¨ ( Y ) in Eq. (8) is hard to evaluate since S in Eq. (2)is the sum of an infinite series. Instead, ApproxAAMC performs afinite number π‘ = πΌ of iterations in Eq. (2) to obtain an approximateAAMC, since the expected length of an attributed random walk is πΌ . Specifically, given P π , (cid:98) R , R , πΌ, π½ and Y as inputs, AppoxAAMC first initializes H as ( YY β€ ) β Y and the number of iterations π‘ to πΌ (Line 1 of Algorithm 4). Then, it computes the intermediate result H π‘ by π‘ iterations in Lines 2-3. Lastly AppoxAAMC computes π with H π‘ and H at Line 4. Algorithm 4 takes π ( ππΌ Β· (| πΈ π | + | πΈ π |)) time with the precomputed (cid:98) R .Utilizing algorithms GenNCI and
AppoxAAMC , ACMin obtainsthe binary NCI Y β and its corresponding quality measure π β foreach iteration β , after obtaining F β . ACMin may terminate uponconvergence, or reaching a preset maximum number of iterations π‘ π . In our experiments, we found that π‘ π =
200 is usually sufficientlylarge for convergence.Next we analyze the total time and space complexities of
ACMin .The computation of (cid:98) R at Line 1 in Algorithm 1 takes π (| πΈ π |) time.Algorithm 4 requires π ( ππ + ππΌ Β· (| πΈ π | + | πΈ π |)) time. In each itera-tion (Lines 6-12), Line 7 takes π ( π Β· (| πΈ π | + | πΈ π |)) time and the QR decomposition over Z β takes π ( ππ ) time. According to Section 4.2and Section 4.3, GenNCI and
InitNCI run in π ( π‘ π Β· ( ππ + π )) and π ( ππΌ Β· | πΈ π |) time, respectively. Thus, the total time complexityof ACMin is π (cid:16) π ( πΌ + π‘ π ) Β· (| πΈ π | + | πΈ π |) + ππ π‘ π π‘ π + ππ‘ π πΌ Β· | πΈ π |) (cid:17) when π βͺ π , which equals π (| πΈ π | + | πΈ π |) when π‘ π , π‘ π and π areregarded as constants. The space overhead incurred by ACMin is determined by the storage of P π , (cid:98) R , R , Z β , F β and H β , which isbounded by π (| πΈ π | + | πΈ π | + ππ ) . Table 2: Datasets. (K= , M= , B= ) Name | π | | πΈ π | | π | | πΈ π | | πΆ | Cora [29, 52, 53, 60, 64] 2.7K 5.4K 1.4K 49.2K 7
Citeseer [29, 52, 53, 60, 64] 3.3K 4.7K 3.7K 105.2K 6
Pubmed [52, 60, 64, 66] 19.7K 44.3K 0.5K 988K 3
Flickr [24, 29, 34, 58, 60] 7.6K 479.5K 12.1K 182.5K 9
TWeibo [60] 2.3M 50.7M 1.7K 16.8M 8
MAG-Scholar-C [3] 10.5M 265.2M 2.78M 1.1B 8
We experimentally evaluate
ACMin against 11 competitors in termsof both clustering quality and efficiency on 6 real-world datasets.All experiments are conducted on a Linux machine powered by anIntel Xeon(R) Gold [email protected] CPU and 377GB RAM. Sourcecodes of all competitors are obtained from the respective authors.
Datasets.
Table 2 shows the statistics of the 6 real-world directedattributed graphs used in our experiments. | π | and | πΈ π | denotethe number of nodes and edges, while | π | and | πΈ π | represent thenumber of attributes and node-attribute associations, respectively. | πΆ | is the number of ground-truth clusters in πΊ . In particular, Cora , Citeseer , Pubmed and MAG-Scholar-C are citation graphs, inwhich each node represents a paper and each edge denotes a citationrelationship. Flickr and TWeibo are social networks, in whicheach node represents a user, and each directed edge representsa following relationship. Further, notice that all 6 datasets haveground-truth cluster labels, and the number of ground-truth clusters | πΆ | is also included in Table 2. Competitors.
We compare
ACMin with 11 competitors, including7 π -AGC algorithms ( CSM [36], SA - Cluster [67],
BAGC [54],
MGAE [53],
CDE [29],
AGCC [64],
USC [50]), and 4 recent attributed net-work embedding algorithms (
TADW [55],
PANE [60],
LQANR [56],
PRRE [66]). The network embedding competitors are used togetherwith π - Means to produce clustering results. In addition, we alsocompare with the classic unnormalized spectral clustering method
USC [50], which directly works on S to extract clusters by materi-alizing S , computing the top- π eigenvectors of S , and then applying π - Means on the eigenvectors.
Parameter settings.
We adopt the default parameter settings ofall competitors as suggested in their corresponding papers. Specif-ically, for attributed network embedding competitors, we set theembedding dimensionality to 128. For
ACMin , we set π‘ π = , π‘ π = , πΌ = .
2, and π½ = .
35. Competitor
USC shares the same parame-ter settings of πΌ , π½ , and π‘ π with ACMin . Evaluation criteria.
For efficiency evaluation, we vary the num-ber of clusters π in { , , , , } , and report the running time(seconds) of each method on each dataset in Section 5.2. The re-ported running time does not include the time for loading datasets.We terminate a method if it fails to return results within 3 days. Interms of clustering quality, we report the proposed AAMC measure( i.e. , average attributed multi-hop conductance), modularity [38], http://linqs.soe.ucsc.edu/data (accessed October, 2020) https://figshare.com/articles/dataset/mag_scholar/12696653 (accessed October, 2020) https://github.com/xhuang31/LANE (accessed October, 2020) onferenceβ17, July 2017, Washington, DC, USA Yang and Shi, et al. ACMin USC CSM BAGC SA - Cluster MGAE CDE AGCC PANE TADW LQANR PRRE β running time (sec) (a) Cora β running time (sec) (b) Citeseer running time (sec) (c) Pubmed running time (sec) (d) Flickr running time (sec) (e) TWeibo running time (sec) (f) MAG-Scholar-C
Figure 2: Running time with varying π (best viewed in color).Table 3: CA, NMI and AAMC with ground-truth (Large CA, NMI, and small AAMC indicate high clustering quality). Solution
Cora Citeseer Pubmed Flickr TWeibo MAG-Scholar-C
CA NMI AAMC CA NMI AAMC CA NMI AAMC CA NMI AAMC CA NMI AAMC CA NMI AAMCGround-truth 1.0 1.0 0.546 1.0 1.0 0.531 1.0 1.0 0.505 1.0 1.0 0.691 1.0 1.0 0.719 1.0 1.0 0.63
TADW
LQANR
PRRE
PANE
CSM SA - Cluster
BAGC
MGAE
CDE
AGCC
USC
ACMin CA (clustering accuracy with respect to ground truth labels) and NMI (normalized mutual information) [1] to measure the cluster-ing quality in Section 5.3. Note that AAMC considers both graphtopology and node attributes to measure clustering quality, whilemodularity only considers graph topology. Also, note that CA andNMI rely on ground-truth clusters, while AAMC and modularitydo not. Therefore, when evaluating by CA and NMI, we set π to be | πΆ | as in Table 2 for each dataset; when evaluating by modularityand AAMC, we vary π in { , , , , } . Figure 2 presents the running time of all methods on all datasetswhen varying the number of clusters π in { , , , , } . The π¦ -axis is the running time (seconds) in log-scale. As shown in Figure 2, ACMin is consistently faster than all competitors on all datasets, of-ten by up to orders of magnitude.
ACMin is highly efficient on largeattributed graphs, e.g. , TWeibo and
MAG-Scholar-C in Figures 2e and2f, while most of the 11 competitors fail to return results withinthree days. For instance, in Figure 2e, when π = ACMin needs 630seconds to finish, which is 7 . Γ faster than AGCC (4634 seconds)and 71 Γ faster than PANE (44658 seconds), respectively. Further,
ACMin is the only method able to finish on
MAG-Scholar-C datasetthat has 265.2 million edges and 1.1 billion attribute values. Specifi-cally,
ACMin only needs 1.68 hours when π =
5. The high efficiencyof
ACMin on massive real datasets is due to the its high scalablealgorithmic components, whose total cost is linear to the size of theinput graph as analyzed in Section 4.4. On small/moderate-sized at-tributed graphs in Figures 2a-2d,
ACMin is also significantly fasterthan the competitors, especially when π is small. For instance, when π =
10, on
Flickr in Figure 2d,
ACMin takes 4 seconds, while thefastest competitor
PANE needs 381 seconds. Note that the total run-ning time of
ACMin increases linearly with π , which is consistent with our time complexity analysis in Section 4.4 when the numberof edges | πΈ π | + | πΈ π | far exceeds the number of nodes π . The runningtime results for the 4 attributed network embedding competitors( i.e. , TADW , LQANR , PRRE , and
PANE ) are not sensitive to π , sincetheir cost is dominated by the node embedding computation ratherthan π - Means . Even on settings with a large π , ACMin is still fasterthan all these competitors.
CA, NMI and AAMC with ground-truth.
Table 3 reports the CA,NMI, and AAMC scores of all methods, comparing to the ground-truth clusters of each dataset in Table 2. We also report the AAMCvalues of the ground-truth labels, which are the lower than theAAMC obtained by all methods except for
ACMin , which explicitlyaims to minimize AAMC. Meanwhile, we observe that relative per-formance of all methods measured by AAMC generally agrees withCA. These results demonstrate that AAMC effectively measureseffectively reflects clustering quality.
ACMin clearly and consistently achieves the best CA, NMI, andAAMC on all datasets. Specifically, on small attributed graphs, i.e. , Cora , Citeseer and
Pubmed , compared with the best competi-tors (underlined in Table 3)
ACMin improves CA by 1 . . . . . . ACMin are also significantly better than the competi-tors on moderate-sized/large attributed graphs ( i.e. , Flickr , TWeibo ,and
MAG-Scholar-C ). For instance, on
Flickr , ACMin has CA 75 . .
6% higher than that of the best competitor
AGCC , whichis only 47 . TWeibo , ACMin is slightly better than
AGCC ;note that on this dataset,
ACMin is orders of magnitude faster than
AGCC as shown in Figure 2e. Hence,
ACMin is overall preferablethan
AGCC in these settings. Finally,
ACMin is the only π -AGCmethod capable of handling MAG-Scholar-C , and achieves CA 65 . ffective and Scalable Clustering on Massive Attributed Graphs Conferenceβ17, July 2017, Washington, DC, USA ACMin USC CSM BAGC SA - Cluster MGAE CDE AGCC PANE TADW LQANR PRRE
AAMC (a)
Cora
AAMC (b)
Citeseer
AAMC (c)
Pubmed
AAMC (d)
Flickr
AAMC (e)
TWeibo
AAMC (f)
MAG-Scholar-C
Figure 3: AAMC with varying π (best viewed in color). . . . . modularity (a) Cora . . . . . . . modularity (b) Citeseer . . . . modularity (c) Pubmed . . . . modularity (d) Flickr . . . . modularity (e) TWeibo . . . . . modularity (f) MAG-Scholar-C
Figure 4: Modularity with varying π (best viewed in color). ACMin ACMin - RI AAMC (a)
Cora
AAMC (b)
Citeseer
AAMC (c)
Pubmed
AAMC (d)
Flickr
AAMC (e)
TWeibo
AAMC (f)
MAG-Scholar-C
Figure 5: AAMC with varying π‘ π (best viewed in color). and NMI 49 . ACMin demonstrates the effectiveness of the proposed AAMC optimizationobjective in Section 2, as well as our approximate solution for thisoptimization program described in Section 4.
AAMC with varying π . Figure 3 reports the AAMC achieved by
ACMin against all competitors on all datasets when varying thenumber π of clusters in { , , , , } . Observe that ACMin con-sistently produces the smallest AAMC under all π settings on alldatasets (smaller AAMC indicates better results), which confirmsthat the proposed ACMin algorithm (Algorithm 1) effectively mini-mizes the proposed AAMC objective function defined in Section2.4. In particular, as shown in Figure 3, when π = ACMin hasAAMC better than the best competitor by a margin of 0 . . . .
84% and 2 .
6% on
Cora , Citeseer , Pubmed , Flickr and
TWeibo respectively. Figure 3f reports the AAMC achieved by
ACMin onMAG-Scholar-C, which is the only method able to return results.Further, considering the relative performance of all methods mea-sured by CA and NMI generally agree with that measured by AAMCas shown in the results in Table 3, and the fact that
ACMin is farmore efficient and scalable compared to its competitors as shownin Section 5.2, we conclude that
ACMin is the method of choice for π -AGC on massive graphs in practice. Modularity with varying π . Figure 4 reports the modularity of allmethods on all datasets when varying π in { , , , , } . Again,observe that, for all settings of π and all datasets (except TWeibo ), ACMin has the highest modularity. In particular,
ACMin obtains a substantial imporvement of up to 5% , . , . .
1% on
Cora , Citeseer , Pubmed and
Flickr , compared to the best competitor,respectively. Note that modularity only considers graph topologyand ignores node attributes, indicating that modularity may not beable to fully evaluate clustering quality of attributed graphs. Thismay explain why on
TWeibo the modularity of
ACMin is slightlylower than some competitors. Even so,
ACMin still achieves highmodularity under most cases, meaning that the proposed attributedrandom walk model can still preserve graph topological featuresfor clustering, in addition to node attributes.
ACMin
In this section, we evaluate the convergence properties of
ACMin ,focusing on the effects of the greedy initialization technique
InitNCI described in Section 4.3 on convergence speed. In particular, wecompare
ACMin with an ablated version
ACMin - RI that replaces InitNCI at Line 2 of Algorithm 1 with random initialization of Y .The number π of clusters to be detected is set to be | πΆ | as in Table2 for each dataset. Figure 5 reports the AAMC ( i.e. , Ξ¨ ( Y ) in Eq. (8))produced by ACMin and
ACMin - RI per iterations (Lines 6-12 in Al-gorithm 1), when π‘ π is set to 200. Observe that the AAMC producedby ACMin decreases significantly faster than that of
ACMin - RI in the early iterations, and also converges faster than ACMin - RI .For instance, in Figure 5b, on Citeseer , ACMin requires about 80iterations to reach a plateaued AAMC, while
ACMin - RI needs 140iterations. Moreover, GenNCI is able to help
ACMin to achieve onferenceβ17, July 2017, Washington, DC, USA Yang and Shi, et al. lower AAMC at convergence as shown in Figure 5. This experimen-tal evaluation demonstrates the efficiency and effectiveness of theproposed greedy initialization technique in Section 4.3. Attributed graph clustering has been extensively studied in liter-ature, as surveyed in [4, 5, 11]. In the following, we review theexisting methods that are most relevant to this work.
Edge-weight-based clustering.
A classic methodology is to con-vert the input attributed graph to a weighted graph by assigningeach edge a weight based on the attribute and topological similaritybetween the two nodes of the edge; then, traditional weighted graphclustering algorithms are directly applied [7, 33, 37, 42, 45]. For in-stance, Neville et al. [37] assign a weight to each edge ( π’, π£ ) of theinput attributed graph πΊ based on the number of attribute valuesthat π’ and π£ have in common, and construct a weighted graph πΊ β² .Then they apply the classic spectral clustering [50] over πΊ β² to pro-duce clusters. However, these methods only consider the attributesof two directly connected nodes and use hand-crafted weights torepresent attributes, and thus, result in inferior clustering quality. Distance-based clustering.
Existing distance-based clustering so-lutions construct a distance matrix M by combining the topolog-ical and attribute similarity between nodes, and then apply clas-sic distance-based clustering methods, such as π - Means [19] and π - Medoids [41], on M to generate clusters. For instance, SA - Cluster [67] extends the original input attributed graph πΊ to an attribute-augmented graph πΊ β² by treating each attribute as a node, and thensamples random walks over πΊ β² to compute the distance betweennodes in πΊ β² , in order to construct M , which is then fed into a π -Centroids method to generate clusters. Further, DCom [7] applieshierarchical agglomerative clustering on a constructed distance ma-trix.
CSM [36] computes the distance matrix M based on a shortestpath strategy that considers both structural and attribute relevanceamong nodes, and applies π - Medoids over M to generate clusters. ANCA [10] applies π - Means for the sum of eigenvectors of the dis-tance and similarity matrices to generate clusters. Distance-basedclustering methods suffer from severe efficiency issues since theyrequire to compute the distance of every node pair, resulting in π ( π ) time and space overhead, which is prohibitive in practice.For instance, as shown in our experiments, both SA - Cluster and
CSM suffer from costly running time and poor clustering quality.
Probabilistic-model-based clustering.
Based on the assumptionthat the structure, attributes, and clusters of attributed graphs aregenerated according to a certain parametric distribution, there ex-ist a collection of probabilistic-model-based clustering methods,which statistically infer a probabilistic model for attributed graphclustering, in order to generate clustering results. In particular,
PCL - DC [62] combines a conditional model of node popularityand a discriminative model that reduces the impact of irrelevantattributes into a unified model, and then finds the clustering resultthat optimizes the model. CohsMix [63] formulates the clusteringproblem by
MixNet model [40] and then utilizes a varient of EMalgorithm to optimize it, in order to generate clustering results.
BAGC [54] designs a generative Bayesian model [8] that producesa sample of all the possible combinations of a graph based on adja-cency matrix A and attribute matrix X , and aims to find a clustering result πΆ maximizing a conjoint probability P ( πΆ | A , X ) . Note thatthe optimization process to estimate the likelihood parameters inthese probabilistic-model-based clustering methods often incurssubstantial time overheads, as validated in our experiments (Section5.2). Embedding-based methods.
In recent years, a plethora of net-work embedding techniques are proposed for attributed graphs.The objective of network embedding is to learn an embeddingvector for each node such that the graph topology and attributeinformation surrounding the nodes can be preserved. We can di-rectly employ traditional clustering methods ( e.g. , π - Means ) overthe embedding vectors to generate clusters [19, 41]. AA - Cluster [2] builds a weighted graph based on graph topology and nodeattributes, and then applies network embedding on the weightedgraph to generate embeddings.
MGAE [53] proposes a marginalizedgraph convolutional network to learn embeddings.
CDE [29] learnsnode embeddings by optimizing a non-negative matrix factoriza-tion problem based on community structure embeddings and nodeattributes.
DAEGC [52] fuses graph topology and node attributesvia an attention-based autoencoder [48] to obtain embeddings, andthen generates soft labels to guide a self-training graph cluster-ing procedure.
AGCC [64] utilizes an adaptive graph convolutionmethod to learn embeddings, and then applies the spectral clus-tering on the similarity matrix computed from the learnt embed-dings to obtain clusters. The above methods either incur immenseoverheads in learning embeddings or suffer from unsatisfactoryclustering quality. There are many attributed network embeddingmethods proposed, e.g. , [17, 31, 34, 55β57, 60, 66]. However, most ofthem are not specially designed for clustering purpose, leading tosuboptimal clustering quality, as demonstrated in our experimentswhen comparing with
TADW , LQANR , PRRE and
PANE . This paper presents
ACMin , an effective and scalable solution for π -AGC computation. ACMin achieves high scalability and effective-ness through a novel problem formulation based on the proposedattributed multi-hop conductance measure for cluster quality, aswell as a carefully designed iterative optimization framework and aneffective greedy clustering initialization method. Extensive experi-ments demonstrate that
ACMin achieves substantial performancegains over the previous state of the art in terms of both efficiencyand clustering quality. Regarding future work, we plan to studyparallelized versions of
ACMin , running on multi-core CPUs andGPUs, as well as in a distributed setting with multiple servers, in or-der to handle even larger datasets. Meanwhile, we intend to extend
ACMin to handle attributed heterogeneous graphs with differenttypes of nodes and edges.
ACKNOWLEDGMENTS
This work is supported by the National University of Singaporeunder SUG grant R-252-000-686-133, by NPRP grant ffective and Scalable Clustering on Massive Attributed Graphs Conferenceβ17, July 2017, Washington, DC, USA A PROOFS
Proof of Lemma 2.3.
Let π β ( π£ π , π£ π ) be the probability that an at-tributed random walk starting from π£ π stops at π£ π at the β -th hop.We first prove that π β ( π£ π , π£ π ) = πΌ ( β πΌ ) β Β· (( β π½ ) Β· P π + π½ Β· P π ) β [ π£ π , π£ π ] . (18)Note that if Eq. (18) holds, the overall probability that an attributedrandom walk from π£ π terminates at π£ π is (cid:205) β β = π β ( π£ π , π£ π ) = S [ π£ π , π£ π ] ,which establishes the equivalence in Eq. (2). To this end, we proveEq. (18) by induction. First, let us consider the initial case thatthe attributed random walk terminates at source node π£ π withprobability πΌ . In this case, π ( π£ π , π£ π ) = πΌ if π£ π = π£ π ; otherwise π ( π£ π , π£ π ) =
0, which is identical to the r.h.s of Eq. (18) when β =
0. Therefore, Eq. (18) holds when β =
0. Assume that Eq.(18) holds at the β β² -th hop. Then the probability that an attributedrandom walk from π£ π visits any node π£ π β π at the β β² -th hop is ( β πΌ ) β β² Β· (( β π½ ) Β· P π + π½ Β· P π ) β β² [ π£ π , π£ π ] . Based on this assumption,for the case β = β β² +
1, with probability 1 β πΌ , it will navigate to node π£ π according to the probability ( β π½ ) Β· P π [ π£ π , π£ π ] + π½ Β· P π [ π£ π , π£ π ] ,and finally stop at π£ π with probability πΌ . Thus, π β β² + ( π£ π , π£ π ) = (cid:205) π£ π β π ( β πΌ ) β β² Β· (( β π½ ) Β· P π + π½ Β· P π ) β β² [ π£ π , π£ π ] Β· ( β πΌ ) πΌ (( β π½ ) Β· P π [ π£ π , π£ π ] + π½ Β· P π [ π£ π , π£ π ]) = πΌ ( β πΌ ) β β² + Β· (( β π½ ) Β· P π + π½ Β· P π ) β β² + [ π£ π , π£ π ] , which completes the proof. β‘ Proof of Lemma 3.1.
By Eq. (6), for cluster πΆ π , we have vector (( YY β€ ) β Y )[ π π ] , where each entry (( YY β€ ) β Y )[ π π , π£ π ] = / βοΈ | πΆ π | if π£ π β πΆ π and otherwise (( YY β€ ) β Y )[ π π , π£ π ] =
0. Note that2 Β· (( YY β€ ) β Y )[ π π ] Β· ( I β S ) Β· (( YY β€ ) β Y )[ π π ] β€ = (cid:205) π£ π ,π£ π β π S [ π£ π , π£ π ] Β· ((( YY β€ ) β Y )[ π π , π£ π ] β (( YY β€ ) β Y )[ π π , π£ π ]) = (cid:205) π£ π β πΆ π ,π£ π β π \ πΆ π S [ π£ π , π£ π ] Β· (( YY β€ ) β Y )[ π π , π£ π ] = Ξ¦ ( πΆ π ) . Then we have (cid:205) ππ = Ξ¦ ( πΆ π ) π = π (cid:205) ππ π = (( YY β€ ) β Y )[ π π ] Β· ( I β S ) Β· (( YY β€ ) β Y )[ π π ] β€ = π Β· trace ((( YY β€ ) β Y ) Β· ( I β S ) Β· (( YY β€ ) β Y ) β€ ) , which completes our proof. β‘ Proof of Lemma 3.2.
First, we construct a weighted graph G = (V , E) based on the input graph πΊ = ( π , πΈ π , π , πΈ π ) by letting V = π and E = {( π£ π , π£ π , S [ π£ π , π£ π ]) | π£ π , π£ π β π and S [ π£ π , π£ π ] > } ,where S [ π£ π , π£ π ] signifies the weight of edge ( π£ π , π£ π ) . Thus, Eq. (8)can be reduced to the objective function of the min-cut problem on G , which is proven to be NP-hard in [14, 51]. β‘ Proof of Lemma 3.3.
Let π π ( M ) be the π -th smallest eigenvalue ofmatrix M . Note that β Y β R π Γ π , (( YY β€ ) β Y ) Β· (( YY β€ ) β Y ) β€ = I ,meaning that π ( Y ) = (( YY β€ ) β Y ) β€ Β· (( YY β€ ) β Y ) is a projec-tion matrix of rank π . Therefore, β π < π β π + , π π ( π ( Y )) = β π β₯ π β π + , π π ( π ( Y )) =
1. By Von Neumannβs trace inequality[35] and the property of matrix trace, for any Y β R π Γ π , we havethe following inequality: Ξ¨ ( Y ) = π Β· trace ((( YY β€ ) β Y ) Β· ( I β S ) Β· (( YY β€ ) β Y ) β€ ) = π Β· trace (( I β S ) Β· π ( Y )) β₯ π Β· (cid:205) ππ = π π ( I β S ) Β· π π β π + ( π ( Y )) = π Β· (cid:205) ππ = π π ( I β S ) = π Β· (cid:205) ππ = ( β π π β π + ( S )) . (19) Note that F be the top- π eigenvectors of S , implying FF β€ = I and ( I β S ) Β· F [ π π ] β€ = ( β π π β π + ( S )) Β· F [ π π ] β€ for 1 β€ π β€ π . Hence, π Β· trace ( F ( I β S ) F β€ ) = π Β· (cid:205) ππ = ( β π π β π + ( S )) , (20)which implies that Ξ¨ ( Y ) is minimized when (( YY β€ ) β Y ) = F .Suppose Y β β π Γ π is the optimal solution to Eq. (8). Therefore,with Eq. (19) and Eq. (20), the following inequality holds π β = Ξ¨ ( Y β ) = π Β· trace ((( Y β Y ββ€ ) β Y β )( I β S )(( Y β Y ββ€ ) β Y β ) β€ )β₯ π Β· (cid:205) ππ = ( β π π β π + ( S )) = π Β· trace ( F ( I β S ) F β€ ) , which finishes our proof. β‘ Proof of Lemma 3.4.
Eq. (9) implies that XF β ( YY β€ ) β Y where X β€ X = I . By the property of matrix trace, we have Ξ¨ ( Y ) = π Β· trace (( YY β€ ) β Y Β· ( I β S ) Β· (( YY β€ ) β Y ) β€ )β π Β· trace ( XF ( I β S ) F β€ X β€ ) = π Β· trace ( X β€ XF ( I β S ) F β€ ) = π Β· trace ( F ( I β S ) F β€ ) . (21)By Lemma 3.3, we have Ξ¨ ( Y ) β π β , completing our proof. β‘ Proof of Lemma 4.1.
We need the following lemmas for the proof.Lemma A.1 ([65]). If [ π, x ] is an eigen-pair of matrix M β R π Γ π ,then [ (cid:205) π‘β = π€ β π β , x ] is an eigen-pair of matrix (cid:205) π‘β = π€ β M β . Lemma A.2 ([20]).
Given a M β R π Γ π satisfying (cid:205) ππ = M [ π, π ] = β β€ π β€ π and each entry M [ π, π ] β₯ β β€ π, π β€ π , the largesteigenvalue π of M is 1. Suppose that [ π π , x π ] is an eigen-pair of ( β π½ ) Β· P π + π½ Β· P π and π π is its π -th largest eigenvalue. Note that each row sum of ( β π½ ) Β· P π + π½ Β· P π is equal to 1 and each entry of ( β π½ ) Β· P π + π½ Β· P π is non-negative. Then, by Lemma A.2, we have π π β [β , ] for1 β€ π β€ π . Let π ( π π ) = (cid:205) π‘β = πΌ ( β πΌ ) β π βπ . Lemma A.1 implies thatany eigen-pair β π β [ , π ] , [ π ( π π ) , x π ] of ( β π½ ) Β· P π + π½ Β· P π isan eigen-pair of S . By the sum of geometric sequence, we have π ( π π ) = πΌ Β· β( β πΌ ) π‘ + π π‘ + π β( β πΌ ) π π = πΌ β( β πΌ ) π π , which is is monotonouslydecreasing when 1 β€ π β€ π . Hence, for 1 β€ π β€ π , π ( π π ) and x π are the π -th largest eigenvalue and the π -th largest eigenvector of S . Recall that F is the top- π eigenvectors of ( β π½ ) Β· P π + π½ Β· P π .Therefore, F is the top- π eigenvectors of S . The lemma is proved. β‘ Proof of Lemma 4.2.
Let Z = V β€ X β€ U . Since U and V are the leftand right singular vectors, we have UU β€ = I and VV β€ = I . Note that ZZ β€ = I , which implies that each Z [ π, π ] satisfies β β€ Z [ π, π ] β€ πΊ [ π, π ] is a singular value and thus πΊ [ π, π ] >
0. Then,trace (( YY β€ ) β YF β€ X β€ ) = trace ( U πΊ V β€ X β€ ) = trace ( πΊ V β€ X β€ U ) = (cid:205) ππ = πΊ [ π, π ] Β· Z [ π, π ] β€ (cid:205) ππ = πΊ [ π, π ] . Therefore, trace (( YY β€ ) β YF β€ X β€ ) is maximized when Z = I , whichimplies that X = UV β€ . The lemma is proved. β‘ B COMPARISON WITH SPECTRALCLUSTERING
It is straightforward to apply the classic spectral clustering [50](dubbed as
USC ) on S (see Eq. (2)) to generate clusters. Here, wetheoretically analyse its major difference from ACMin . onferenceβ17, July 2017, Washington, DC, USA Yang and Shi, et al. Given an attributed input graph πΊ , the number π of clusters, andthe random walk factors πΌ, π½ as inputs, USC runs with three phases:(i) computing the attributed random walk probability matrix S ; (ii)finding the top- π eigenvectors F of the attributed random walkprobability matrix S ; and (iii) generating an NCI Y via π - Means with F . Specifically, given the top- π eigenvectors F β R π Γ π , math-ematically, π - Means finds π disjoint clusters πΆ , πΆ , Β· Β· Β· , πΆ π of πΊ such thatmin πΆ ,πΆ , Β·Β·Β· ,πΆ π (cid:205) ππ = (cid:205) π£ π β πΆ π (cid:13)(cid:13)(cid:13) F [ : , π£ π ] β | πΆ π | (cid:205) π£ π β πΆ π F [ : , π£ π ] (cid:13)(cid:13)(cid:13) , which can be rewritten as an NCI-based matrix form, i.e. ,min Y β π Γ π (cid:13)(cid:13) F β€ β Y β€ X (cid:13)(cid:13) πΉ , (22)where X = ( YY β€ ) β YF β€ . (23) π - Means solves Eq. (22) by using the expectation-maximization(EM) algorithm. Initially, it sets Y as a random incicator matrix ( i.e. , π Γ π ) and computes X via Eq. (23) accordingly. Then, with X fixed,each entry Y [ π π , π£ π ] for cluster πΆ π and node π£ π is updated as Y [ π π , π£ π ] =  , π = arg min β€ π β€ π (cid:13)(cid:13) F [ : , π£ π ] β X [ π ] (cid:13)(cid:13) , , else . After that, with Y fixed, it recomputes X by Eq. (23) accordingly. π - Means repeats the above optimization process in an alternativeand iterative manner until convergence. Suppose that we find theoptimal Y to Eq. (22). Then we have F = FY β€ ( YY β€ ) β Y , which im-plies that USC actually finds an NCI Y that minimizes the followingobjective function π Β· trace ( FY β€ ( YY β€ ) β Y Β· ( I β S ) Β· ( FY β€ ( YY β€ ) β Y ) β€ ) , where F is the top- π eigenvectors of S , which is totally different from our objective function in Eq. (8). In anutshell, the NCI Y returned by USC fails to provide us an attributedconductance that approaches the optimal attributed conductance π β of πΊ . REFERENCES [1] C. C. Aggarwal and C. K. Reddy.
Data Clustering: Algorithms and Applications .CRC Press, 2014.[2] E. Akbas and P. Zhao. Attributed graph clustering: An attribute-aware graphembedding approach. In
ASONAM , 2017.[3] A. Bojchevski, J. Klicpera, B. Perozzi, A. Kapoor, M. Blais, B. RΓ³zemberczki,M. Lukasik, and S. GΓΌnnemann. Scaling graph neural networks with approximatepagerank. In
SIGKDD , 2020.[4] C. Bothorel, J. D. Cruz, M. Magnani, and B. Micenkova. Clustering attributedgraphs: models, measures and methods.
Network Science , 2015.[5] P. Chunaev. Community detection in node-attributed social networks: a survey. arXiv preprint arXiv:1912.09816 , 2019.[6] F. R. Chung and F. C. Graham.
Spectral graph theory . 1997.[7] D. Combe, C. Largeron, E. Egyed-Zsigmond, and M. GΓ©ry. Combining relationsand text in scientific network clustering. In
ASONAM , 2012.[8] P. Congdon.
Bayesian statistical modelling . 2007.[9] J. W. Demmel.
Applied numerical linear algebra . Siam, 1997.[10] I. Falih, N. Grozavu, R. Kanawati, and Y. Bennani. Anca: Attributed networkclustering algorithm. In
Complex Networks , 2017.[11] I. Falih, N. Grozavu, R. Kanawati, and Y. Bennani. Community detection inattributed network. In
WWW , 2018.[12] S. Fortunato. Community detection in graphs.
Physics reports , 2010.[13] L. C. Freeman. Cliques, galois lattices, and the structure of human social groups.
Social networks , 1996.[14] O. Goldschmidt and D. S. Hochbaum. Polynomial algorithm for the k-cut problem.In
FOCS , 1988.[15] R. Guimera and L. A. N. Amaral. Functional cartography of complex metabolicnetworks.
Nature , 2005. [16] N. Halko, P.-G. Martinsson, and J. A. Tropp. Finding structure with randomness:Probabilistic algorithms for constructing approximate matrix decompositions.
SIAM review , 2011.[17] W. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on largegraphs. In
NeurIPS , 2017.[18] D. Hanisch, A. Zien, R. Zimmer, and T. Lengauer. Co-clustering of biologicalnetworks and gene expression data.
Bioinformatics , 2002.[19] J. A. Hartigan and M. A. Wong. Algorithm as 136: A k-means clustering algorithm.
J R Stat Soc Ser C , 1979.[20] T. Haveliwala and S. Kamvar. The second eigenvalue of the google matrix.Technical report, Stanford, 2003.[21] D. He, Z. Feng, D. Jin, X. Wang, and W. Zhang. Joint identification of networkcommunities and semantics via integrative modeling of network topologies andnode contents. In
AAAI , 2017.[22] D. Hric, R. K. Darst, and S. Fortunato. Community detection in networks: Struc-tural communities versus ground truth.
Physical Review E , 2014.[23] H. Huang, H. Shen, and Z. Meng. Community-based influence maximization inattributed networks.
Applied Intelligence , 2020.[24] X. Huang, J. Li, and X. Hu. Label informed attributed network embedding. In
WSDM , 2017.[25] G. Jeh and J. Widom. Scaling personalized web search. In
WWW , 2003.[26] G. Kossinets and D. J. Watts. Empirical analysis of an evolving social network. science , 2006.[27] T. La Fond and J. Neville. Randomization tests for distinguishing social influenceand homophily effects. In
WWW , 2010.[28] A. Lancichinetti and S. Fortunato. Community detection algorithms: a compara-tive analysis.
Physical review E , 2009.[29] Y. Li, C. Sha, X. Huang, and Y. Zhang. Community detection in attributed graphs:An embedding approach. In
AAAI , 2018.[30] U. Liji, Y. Chai, and J. Chen. Improved personalized recommendation based onuser attributes clustering and score matrix filling.
CSI , 2018.[31] J. Liu, Z. He, L. Wei, and Y. Huang. Content to node: Self-translation networkembedding. In
SIGKDD , 2018.[32] L. LovΓ‘sz et al. Random walks on graphs: A survey.
Combinatorics, Paul erdos iseighty , 1993.[33] F. Meng, X. Rui, Z. Wang, Y. Xing, and L. Cao. Coupled node similarity learningfor community detection in attributed networks.
Entropy , 2018.[34] Z. Meng, S. Liang, H. Bao, and X. Zhang. Co-embedding attributed networks. In
WSDM , 2019.[35] L. Mirsky. A trace inequality of john von neumann.
Monatshefte fΓΌr mathematik ,1975.[36] W. Nawaz, K.-U. Khan, Y.-K. Lee, and S. Lee. Intra graph clustering using collab-orative similarity measure.
DAPD , 2015.[37] J. Neville, M. Adler, and D. Jensen. Clustering relational data using attribute andlink information. In
IJCAI , 2003.[38] M. E. Newman and M. Girvan. Finding and evaluating community structure innetworks.
Physical review E , 2004.[39] A. Y. Ng, M. I. Jordan, and Y. Weiss. On spectral clustering: Analysis and analgorithm. In
NeurIPS , 2002.[40] K. Nowicki and T. A. B. Snijders. Estimation and prediction for stochastic block-structures.
J Am Stat Assoc , 2001.[41] H.-S. Park and C.-H. Jun. A simple and fast algorithm for k-medoids clustering.
Expert systems with applications , 2009.[42] Y. Ruan, D. Fuhry, and S. Parthasarathy. Efficient community detection in largenetworks using content and links. In
WWW , 2013.[43] H. Rutishauser. Computational aspects of fl bauerβs simultaneous iterationmethod.
Numerische Mathematik , 1969.[44] S. E. Schaeffer. Graph clustering.
Computer science review , 2007.[45] K. Steinhaeuser and N. V. Chawla. Community detection in a large real-worldsocial network. In
SBP . 2008.[46] S. A. Tabrizi, A. Shakery, M. Asadpour, M. Abbasi, and M. A. Tavallaie. Personal-ized pagerank clustering: A graph clustering algorithm based on random walks.
Physica A , 2013.[47] H. Tong, C. Faloutsos, and J.-Y. Pan. Fast random walk with restart and itsapplications. In
ICDM , 2006.[48] P. VeliΔkoviΔ, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graphattention networks.
ICLR , 2018.[49] K. Voevodski, S.-H. Teng, and Y. Xia. Finding local communities in proteinnetworks.
BMC bioinformatics , 2009.[50] U. Von Luxburg. A tutorial on spectral clustering.
Statistics and computing , 2007.[51] D. Wagner and F. Wagner. Between min cut and graph bisection. In
MFCS , 1993.[52] C. Wang, S. Pan, R. Hu, G. Long, J. Jiang, and C. Zhang. Attributed graphclustering: a deep attentional embedding approach. In
IJCAI , 2019.[53] C. Wang, S. Pan, G. Long, X. Zhu, and J. Jiang. Mgae: Marginalized graphautoencoder for graph clustering. In
CIKM , 2017.[54] Z. Xu, Y. Ke, Y. Wang, H. Cheng, and J. Cheng. A model-based approach toattributed graph clustering. In
SIGMOD , 2012.12 ffective and Scalable Clustering on Massive Attributed Graphs Conferenceβ17, July 2017, Washington, DC, USA [55] C. Yang, Z. Liu, D. Zhao, M. Sun, and E. Chang. Network representation learningwith rich text information. In
AAAI , 2015.[56] H. Yang, S. Pan, L. Chen, C. Zhou, and P. Zhang. Low-bit quantization forattributed network representation learning. In
IJCAI , 2019.[57] H. Yang, S. Pan, P. Zhang, L. Chen, D. Lian, and C. Zhang. Binarized attributednetwork embedding. In
ICDM , 2018.[58] J. Yang, J. McAuley, and J. Leskovec. Community detection in networks withnode attributes. In
ICDM , 2013.[59] R. Yang, J. Shi, X. Xiao, Y. Yang, and S. S. Bhowmick. Homogeneous networkembedding for massive graphs via reweighted personalized pagerank.
PVLDB ,2020.[60] R. Yang, J. Shi, X. Xiao, Y. Yang, J. Liu, and S. S. Bhowmick. Scaling attributednetwork embedding to massive graphs.
PVLDB , 2021.[61] R. Yang, X. Xiao, Z. Wei, S. S. Bhowmick, J. Zhao, and R.-H. Li. Efficient estimationof heat kernel pagerank for local clustering. In
SIGMOD , 2019. [62] T. Yang, R. Jin, Y. Chi, and S. Zhu. Combining link and content for communitydetection: a discriminative approach. In
SIGKDD , 2009.[63] H. Zanghi, S. Volant, and C. Ambroise. Clustering based on random graph modelembedding vertex features.
Pattern Recognition Letters , 2010.[64] X. Zhang, H. Liu, Q. Li, and X.-M. Wu. Attributed graph clustering via adaptivegraph convolution. In
IJCAI , 2019.[65] Z. Zhang, P. Cui, X. Wang, J. Pei, X. Yao, and W. Zhu. Arbitrary-order proximitypreserved network embedding. In
SIGKDD , 2018.[66] S. Zhou, H. Yang, X. Wang, J. Bu, M. Ester, P. Yu, J. Zhang, and C. Wang. Prre:Personalized relation ranking embedding for attributed networks. In
CIKM , 2018.[67] Y. Zhou, H. Cheng, and J. X. Yu. Graph clustering based on structural/attributesimilarities.
PVLDB , 2009.[68] Y. Zhou, H. Cheng, and J. X. Yu. Clustering large attributed graphs: An efficientincremental approach. In