A Hebbian approach to complex network generation
AAPS/123-QED
A Hebbian approach to complex network generation
E. Agliari
1, 2, 3 and A. Barra
4, 5 Dipartimento di Fisica, Universit`a degli Studi di Parma, viale Usberti 7/A, 43100 Parma, Italy Istituto Nazionale di Fisica Nucleare, Gruppo Collegato di Parma Theoretische Polymerphysik, Albert-Ludwig-Universit¨at, Freiburg, Germany Dipartimento di Fisica, Sapienza Universit`a di Roma, P.le A. Moro 5, 00182, Roma, Italy Gruppo Nazionale per la Fisica Matematica, Sezione di Roma1 (Dated: November 5, 2018)Through a redefinition of patterns in an Hopfield-like model, we introduce and develop an approachto model discrete systems made up of many, interacting components with inner degrees of freedom.Our approach clarifies the intrinsic connection between the kind of interactions among componentsand the emergent topology describing the system itself; also, it allows to effectively address thestatistical mechanics on the resulting networks. Indeed, a wide class of analytically treatable,weighted random graphs with a tunable level of correlation can be recovered and controlled. Weespecially focus on the case of imitative couplings among components endowed with similar patterns(i.e. attributes), which, as we show, naturally and without any a-priori assumption, gives rise tosmall-world effects. We also solve the thermodynamics (at a replica symmetric level) by extendingthe double stochastic stability technique: free energy, self consistency relations and fluctuationanalysis for a picture of criticality are obtained.
PACS numbers: 05.50.+q,02.10.Ox,05.70.Fh
The performance of most complex systems, from thecell to the Internet, emerges from the collective activity ofmany inner components; At an abstract level, the lattercan be reduced to a series of nodes that are connectedeach other by links, envisaging the interaction. Nodesand links together form a network [1–3].The view offered by network description has led toidentify classes of (topological) universality [4], andto evidence how experimentally-revealable features, e.g.cliquishness, modularity, assortativity or peculiar degreedistribution, not only underlie a certain degree of corre-lation among components and/or links but also cruciallyaffect the behavior of the system [1]. This constituted areal breakthrough with respect to the previous tendencyto model complex networks either as regular objects, suchas square or diamond lattices, or as (purely uncorrelated)random networks `a la Erd¨os-Renyi (ER) [6].In the first part of this work we show that, within ourapproach, the kind of interaction (e.g. imitative, repul-sive, etc.) among components naturally gives rise to abroad class of weighted random graphs, whose topolog-ical properties can be properly tuned. Such a connec-tion also allows to infer about the plausibility of a givenmodelization: the choice of a particular network must beconsistent with the kind of interactions governing the sys-tem itself and viceversa, ultimately reflecting the internalstructure of the considered nodes.In the second part of this work, we study the thermo-dynamic properties of a subset of these structures gen-erated by a “Hebbian-like kernel”; interestingly, such anapproach allows to work out the thermodynamics of awide class of diluted graphs, even in the presence of fer-romagnetic disorder on couplings.
Modelization.
Given a set of V components, each char-acterized by a “pattern” ξ (similarly to the “hidden vari- able” approach [7, 8]) drawn from a given probability dis-tribution, a pair of nodes i and j is linked according toa proper rule r ( ξ i , ξ j ). More precisely, here, ξ is a vectorgiven by binary string of length L , which encodes a set ofattributes characterizing each node; Then, the function r associates each couple of strings to a real value whichprovides the pertaining coupling: r ( ξ i , ξ j ) = J ij .Now, crucial for the whole approach are the way thestrings are generated and the rule r , both to be definedaccording to the processes one wishes to model. The Hebbian-like kernel
We investigate in detail thecase of biased patterns where the probability to extractany entry is P ( ξ µi = 0) = 1 − P ( ξ µi = 1) = (1 − a ) / , a ∈ [ − , r given by the scalar product amongstrings J ij = J ji = L (cid:88) µ =1 ξ µi ξ µj . (1)Notice that such a rule resembles the Hebbian kernelwell-known in neural networks [5], apart from the shift[ − , +1] → [0 , +1] in the definition of patterns; this plainreplacement converts frustration into ferromagnetic dilu-tion. We are therefore focusing on systems where theinteraction among components is stronger the larger thenumber of attributes they share.An important parameter characterizing a given node,is the number ρ of non-null entries present in the re-lated string: Due to the independence underlying theextraction of each entry, ρ follows a binomial distri-bution P ( ρ ; a, L ) = B ( ρ ; L, (1 + a ) /
2) with average¯ ρ a,L = L (1 + a ) /
2. Moreover, it can be shown thatthe probability for two string ξ i and ξ j , displaying re-spectively ρ i and ρ j non-null entries, to be connected is P link ( ρ i , ρ j ; L ) = 1 − ( L − ρ i )!( L − ρ j )! / [ L !( L − ρ i − ρ j )!]. By a r X i v : . [ c ond - m a t . s t a t - m ec h ] S e p (cid:239) k P ( k )
300 400 500 600 700 800 90000.511.522.533.544.55x 10 (cid:239) k V = 1 0 0 0 L = 5 0 0 V = 7 0 0 0 L = 7 0 FIG. 1: (Color on line) Degree distribution ¯ P degree ( z ; a, L, V )for systems displaying small ratio L/V and multimodal dis-tribution (left panel) and large
L/V with modes collapsinginto a unimodal distribution. In the former case we plot datapoints ( • ) as well as the the analytical curve provided byEq. 2. In the latter case we show the distributions corre-sponding to each mode P degree ( z ; a, ρ, V ) in agreement withnumerical data, as well as the overall distribution (thicker,bright curve). averaging over, say ρ j , one finds the expected link proba-bility P link ( ρ i , a ) = 1 − [(1 − a ) / ρ i for the node i . Then,the probability that i has degree (number of neighbors)equal to z follows as the binomial P degree ( z ; a, ρ i , V ) = B ( z ; V, P link ( ρ i , a )). Due to the average over ρ j , such adegree distribution corresponds to a mean-field approachwhere we treat all the remaining nodes in the average;this works finely for V large and ρ i not too small (seealso Fig. 1). Therefore, the number of null-entries con-trols the degree distribution of the pertaining node: Alarge ρ gives rise to narrow (small variance) distributionspeaked at large values of z . Then, the global degree dis-tribution reads off as a combination of binomials¯ P degree ( z ; a, L, V ) = L (cid:88) ρ =0 P degree ( z ; a, ρ, V ) P ( ρ ; a, L ) , (2)giving rise to a L -modal distribution, where “modes”,each corresponding to a different value of ρ , are solved aslong as the connectivity and L are not too large in orderto ensure spread distributions for z and ρ (see Fig. 1, leftand right panels, respectively and [12] for more details).Multimodal degree distributions constitute an inter-esting feature of the model as they allow to naturallydiscriminate between different classes of nodes possiblyfulfilling different functions. In particular, nodes corre-sponding to a large degree are often associated to a rel-atively small reactivity and vice versa [9, 11].A more global characterization can be attained by theaverage link probability, applying to a generic couple of nodes, neglecting any information about correlations p = 1 − (cid:34) − (cid:18) a (cid:19) (cid:35) L , (3)so that the average degree reads as ¯ z = pV . Now, for L → ∞ , p approaches a discontinuous function assumingvalue 1 when a > − a = −
1. To fixideas, let us focus on the the so-called high-storage regimewhere L is linearly divergent with V , i.e. L = αV , then,the range of values for a yielding a non-trivial topologycan be characterized by means of the scaling a = − γV θ , (4)where θ ≥ γ is a finite parameter [20].Following [12], it is possible to distinguish the followingregimes (see Fig. 2): • θ < / p ≈
1, ¯ z ≈ V ⇒ Fully connected graph • θ = 1 / p ∼ − e − γ α/ ∼ γ α/
4, ¯ z = O ( V ) ⇒ Linearly diverging connectivity; Within a mean-field description the ER random graph with finiteprobability is recovered. • / < θ < p ∼ γ αV − θ / z = O ( V − θ ) ⇒ Extreme dilution regime:lim V →∞ ¯ z − = lim V →∞ ¯ z/V = 0 [10]. • θ = 1, p ∼ γ α V , ¯ z = O ( V ) ⇒ Finite con-nectivity regime; Within a mean-field description γ α/ θ determine a disconnected graph withvanishing average degree.Therefore, while θ controls the connectivity regime ofthe network, γ allows a fine tuning.Up to now we just focused on topological disorder; asfor couplings, we can still detect “modes”, each charac-terized by J ρ representing the average strength for linksstemming from a node associated to a string with ρ non-null entries. While J ρ provides a measure of thelocal “field” seen by a single node, a global descrip-tion can be attained by the overall average coupling¯ J ≡ (cid:80) ρ J ρ P ( ρ ; a, L ), taken over the whole graph. Thetwo quantities are related via J ρ = ρ/L √ ¯ J [12] and, as wewill see, despite the self-consistence relation (see Eq.10),more sensible to local conditions, is influenced by √ ¯ J , thecritical behavior occurs at β c = ¯ J − consistently with amanifestation of a collective, global effect.By looking in more detail at the coupling distributionholding in the thermodynamic limit and for values of a determined by Eq. 4, we find that for 1 / < θ ≤
1, nodesare pairwise either non-connected or connected due toone single matching among the relevant strings, namely,
FIG. 2: (Color on line) Phase diagram representing the topol-ogy of the graph as the parameters θ , γ and α are varied.In the region on the left (blue) disorder on couplings is stillpresent, while on the right side (yellow) disorder on couplingsis lost while topological inhomogeneity is still present. In thenarrow central region (green) a coexistence of both kinds ofdisorder is achieved. neglecting higher order corrections, J = 0 with proba-bility p ∼ exp( αγ V − θ / ∼ − γ α/ (4 V θ − ), while J = L − with probability p ∼ p γ α/ (4 V θ − ) ∼ − p .For θ = 1 / αγ / (cid:28)
1, whichcorresponds to a relatively high dilution regime, other-wise some degree of disorder is maintained, being that p k ∼ ( αγ / k /k !. On the other hand, for θ < /
2, whiletopological disorder is lost, disorder on couplings is stillpresent. However, for θ = 0 and γ = 2, the couplingdistribution gets peaked at J = 1 and, again, disorderon couplings is lost so that a pure Curie-Weiss model isrecovered (see also [12]).Before concluding this part, we stress that slight vari-ations in the rule provided by Eq. 1 can yield dramaticchanges in the global layout of the graph, e.g. scale-freedegree and/or coupling distributions. Small-World properties
A “small-world” network [13]displays, by definition, diameter growing as log N , hencecomparably with the case of ER random graphs, and highcliqueshness indicating a high level of redundancy.The cohesiveness of the neighbourhood of a node i isusually quantified by the (local) clustering coefficient c i ,defined as the fraction of permitted edges between nodesadjacent to i that actually exist. The average clusteringcoefficient c = (cid:80) Vi =1 c i can be compared with the oneexpect for a comparable (i.e. displaying same averagedegree) ER graph, namely c ER = ¯ z/V .For the graph generated by Eq. 1, the neighborhood V i of i is made up of all nodes displaying at least onenon-null entry corresponding to any non-null entries of ξ i . This condition biases the distribution of strings rel-evant to nodes ∈ V i , so that they are more likely to be connected with each other. This is also confirmed nu-merically: Fig. 3 shows that c/c ER > α, a ), and, in particular, in the region ofhigh dilution.Interestingly, this kind of link correlation allows to de-tect, within the graph, communities densely and stronglylinked up, so that the role of weak ties is crucial in bridg-ing such communities and maintaing the graph connected[14, 15]. Thermodynamics
When dealing with thermodynami-cal properties of these networks, we first paste V vari-ables σ on the nodes and define the Hamiltonian H ( σ ; ξ ) = − αN − θ ) V (cid:88) i 5, demarcate the regionwhere the graph is made up of 20 components (a macroscopicone plus isolated nodes). then we enlarge the technique of the double stochasticstability [17] by introducing the following interpolatingstructure A ( t ) = E V log (cid:88) σ (cid:90) + ∞−∞ L (cid:89) µ dµ ( z µ ) exp[ t (cid:112) β/αV − θ (cid:88) i,µ (6) ξ iµ σ i z µ + (1 − t )( L (cid:88) l c =1 b l c V (cid:88) i η i σ i + V (cid:88) l b =1 c l b L (cid:88) µ χ µ z µ )] , where now E = E ξ E η E χ , b l c [with l c ∈ (1 , ..., L )], and c l b [with l b ∈ (1 , ..., V )] are real numbers (possibly functionsof β, γ, θ ) to be set a posteriori.Note that A ( t = 1) is our goal, while A ( t = 0) is straight-forward as it involves only one-body calculations. So wewant to use the fundamental theorem of calculus to geta sum rule, as A (1) = A (0) + (cid:82) [ ∂A ( t (cid:48) ) /∂t (cid:48) ] t (cid:48) = t dt , whichultimately implies the evaluation of the t -streaming ofeq. (6). Now, we need to sort out our order parameters:as replica symmetry (RS) is expected to be conservedin ferromagnetic diluted networks, we (naturally) avoid(multi)-overlaps by defining M l b = V − (cid:80) i ω l b +1 ( σ i )(and analogously for the other party trough N l c ), wherethe index in ω means that we are considering all the pos-sible magnetizations built trough only l b + 1 links insidethe graph (as the graph is no longer weighted, it is a mi-crocanonical decomposition of the observable in subclus-ters close to the one introduced in [16]). Once introducedthe averaged order parameters as (cid:104) M (cid:105) = (cid:80) l b P ( l b ) M l b ,and (cid:104) N (cid:105) = (cid:80) l c P ( l c ) N l c , we find that ∂ t A ( t ) = S ( t ) −√ βγ/ (cid:80) l b (cid:80) l c P ( l b ) P ( l c ) ¯ M l b ¯ N l c ; where the fluctuationsource S ( t ) is proportional to (cid:104) ( M − ¯ M )( N − ¯ N ) (cid:105) andcan be neglected at the RS level (whose order parametervalues are denoted via a bar, namely ¯ M , ¯ N ).As a consequence it is possible to solve for the RS freeenergy to get A ( β, γ, θ ) = log 2 + γ V θ (cid:104) log cosh( (cid:112) β ¯ N V θ ) (cid:105) ++ βγ (cid:104) ¯ M (cid:105) − √ βγ (cid:104) ¯ M (cid:105)(cid:104) ¯ N (cid:105) . Trough the self consistent relation (cid:104) ¯ N (cid:105) = √ βγ (cid:104) ¯ M (cid:105) , wecan express the whole theory only via (cid:104) M (cid:105) (as expectedlooking at the original Hamiltonian (5)) and we startdiscussing the various case of interest: • θ = 0: Fully connected weighted regimeThe case θ = 0 reduces to the Curie-Weiss modeland in particular, the upper bound on γ (i.e. γ = 2)describes the un-weighted fully connected topology.Choerently its termodynamics turns out to be A ( β, γ = 2 , θ = 0) = log 2 + γ βγ M ) − βγ M , ¯ M = tanh( βγ M ) . (7)Note that, as there is only one possible networkbuilt with all the links, all the subgraph magne-tizations collapse into only one, namely P ( ¯ M ) = δ ( ¯ M − M CW ), where with M CW we mean thestandard Curie-Weiss magnetization. Furthermore,note that for γ = 2 we recover exactly the CW ther-modynamics. • θ = 1 / 2: Standard dilution and ER regimeWith a scheme perfectly analogous to the previousone we can write down the free energy and its re-lated self-consistent equation as A ( β, γ, θ = 1 / 2) = log 2 + lim V →∞ ( γ √ V · (8)log cosh( βγ √ V (cid:104) ¯ M (cid:105) ) − βγ (cid:104) ¯ M (cid:105) ) , (cid:104) ¯ M (cid:105) = lim V →∞ tanh( βγ √ V (cid:104) ¯ M (cid:105) ) . (9)Ficticious diverging contributions emerge in thethermodynamic limit because the normalization wechose for the Hamiltonian (5) gives the correct ex-tensivity for the fully connected case only, while for θ = 1 / O ( V − ) on an ER graph. Note in factthat the argument of the logarithm of the hyper-bolic cosine can be read as β √ JV (cid:104) ¯ M (cid:105) . As in stan-dard approaches [18] it is enough to renormalizethe coupling by scaling [19] it with the amount ofnearest neighbours (that in the ER dilution scalesexactly linearly with the volume size). • θ > / 2: Extremely dilution regime and all theother cases of interestWith a scheme analogous to the previous one it ispossible to write down the pertaining free energy,coupled with its self-consistent relation for its ex-tremization.Interestingly, the effective field felt by the trial spin -obtained extremizing eq. (7) with respect to (cid:104) M (cid:105) - scalesas the square root of the coupling strength instead of alinear behavior (this effect disappears approaching theCW limit where √ J → ¯ J ). As the interaction matrix inthe Hamiltonian has been normalized (i.e. is bounded byone), √ ¯ J > ¯ J , the local field is “higher with respect to anaive expectation” [21]. This feature, which is a conse-quence of the diverging variance in the bitstring distribu-tion [12], however does not affect the transition line thatscales consistently with a manifestation of a collective,global effect, as β c ∝ ¯ J − .In order to prove the last statement, we have to tackle thecontrol of the fluctuations of the rescaled order parame-ters (cid:104)M(cid:105) = √ V (cid:104) M − ¯ M (cid:105) , (cid:104)N (cid:105) = √ L (cid:104) N − ¯ N (cid:105) : At firstwe need to derive the t -streaming of the squares of theseobjects (i.e. (cid:104)M (cid:105) , (cid:104)MN (cid:105) , (cid:104)N (cid:105) ), which lead to a systemof coupled ordinary differential equations in t , namely (cid:104) ˙ M (cid:105) = 2 (cid:104)M (cid:105)(cid:104)MN (cid:105) , (10) (cid:104) ˙ MN (cid:105) = (cid:104)M (cid:105)(cid:104)N (cid:105) + (cid:104) ( MN ) (cid:105) , (11) (cid:104) ˙ N (cid:105) = 2 (cid:104)N (cid:105)(cid:104)MN (cid:105) , (12)whose Cauchy condition at t = 0 can be obtained imme-diately, such that we can solve the system, evaluate it at t = 1 (the original statistical mechanics framework) andcheck where these fluctuations do diverge (identifying theexpected second order phase transition). By using Wicktheorem to express four point correlations trough seriesof two point ones and due to internal symmetries reflect-ing the mean field interactions among the two parties [12]the plan is fully solvable and all these fluctuations (and the correlation (cid:104)MN (cid:105) ) are found to diverge on the sameline: β c = ¯ J − , as intuitively expected.This work is supported by the FIRB grant: RBF R EKEV [1] R. Albert, A.-L. Barab´asi, Rev. Mod. Phys. , , 47(2002); S.N. Dorogovtesev, J.F.F. Mendes, Adv. Phys. , , 1079 (2002); M.E.J. Newman, SIAM Rev. , , 167(2003)[2] Complex Systems , Science, , 1212 (1999).[3] H.D. Rozenfeld, Structure and Properties of ComplexNetworks: Models, Dynamics, Applications (VDM Ver-lag, 2008)[4] L. L. F. Chung, Complex Graphs and Networks (AMS,2006)[5] D.J. Amit, Modeling brain functions. The world of at-tractor neural networks , (Cambridge Press, 1988).[6] P. Erd¨os, A. R´enyi, Publicationes Mathematicae ,290297 (1959).[7] G. Caldarelli et al., Phys. Rev. Lett. , 258702 (2002).[8] M. Bogu˜n´a, R. Pastor-Satorras, Phys. Rev. E , 036112(2003).[9] A. Barra, E. Agliari, J. Stat. Mech. P07004 (2010). [10] T.L.H. Watkin and D. Sherrington, Europhys. Lett. ,791 (1991).[11] G. Palla, L. Lov´asz, T. Vicsek, PNAS , 7640 (2010).[12] A. Barra, E. Agliari, submitted (available at arXive:)[13] D.J. Watts, S.H. Strogatz, Nature , (1998).[14] M.S. Granovetter, Amer. J. Soc. , 1360, (1973).[15] E. Agliari, A. Barra, forthcoming.[16] A. Barra, F. Guerra, J. Math. Phys. , 125217, (2008).[17] A. Barra et al., J. Stat. Phys. , 4, (2010).[18] E. Agliari et al., J. Stat. Mech. P10003, (2008).[19] L. De Sanctis, F. Guerra, J. Stat. Phys. , 759 (2008)[20] Since a ∈ [ − , ≤ γ ≤ V θ ; in particu-lar, when θ = 0 the upper bound for γγ