Promoting cooperation by preventing exploitation: The role of network structure
Zoran Utkovski, Viktor Stojkoski, Lasko Basnarkov, Ljupco Kocarev
aa r X i v : . [ q - b i o . P E ] A ug Promoting cooperation by preventing exploitation: The role of network structure
Zoran Utkovski , ∗ Viktor Stojkoski , Lasko Basnarkov , , and Ljupco Kocarev , Fraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587, Berlin, Germany Macedonian Academy of Sciences and Arts, P.O. Box 428, 1000 Skopje, Republic of Macedonia and Faculty of Computer Science and Engineering,Ss. Cyril and Methodius University,P.O. Box 393, 1000 Skopje, Republic of Macedonia (Dated: November 16, 2018)A growing body of empirical evidence indicates that social and cooperative behavior can be af-fected by cognitive and neurological factors, suggesting the existence of state-based decision-makingmechanisms that may have emerged by evolution. Motivated by these observations, we propose asimple mechanism of anonymous network interactions identified as a form of generalized reciprocity– a concept organized around the premise “help anyone if helped by someone”, and study its dynam-ics on random graphs. In the presence of such mechanism, the evolution of cooperation is related tothe dynamics of the levels of investments (i.e. probabilities of cooperation) of the individual nodesengaging in interactions. We demonstrate that the propensity for cooperation is determined by anetwork centrality measure here referred to as neighborhood importance index and discuss relevantimplications to natural and artificial systems. To address the robustness of the state-based strategiesto an invasion of defectors, we additionally provide an analysis which redefines the results for thecase when a fraction of the nodes behave as unconditional defectors.
PACS numbers: 87.23.Ge, 87.23.Kg, 02.50.Ey, 02.50.Le
I. INTRODUCTION
Cooperation has played a fundamental role in manyof the major transitions in biological evolution and isessential to the functioning of a large number of biologicalsystems [1]. Cooperative interactions are required formany levels of network organization ranging from singlecells to groups of animals and, ultimately, humans.While kin selection [2] and group (multilevel) selection[3, 4] have been able to explain the emergence and stabil-ity of cooperation in related individuals, the occurrenceof cooperation between unrelated individuals is more in-triguing [5, 6]. Theoretical models provide evidence thatcooperative behavior can nevertheless evolve and persistif it is based on reciprocity [7–9]. An important role inthe emergence of cooperation plays the network struc-ture. Since the early work of Novak and May [10] whichdemonstrated that a lattice structure enhances coopera-tion in a Prisoner’s Dilemma (PD) game, this issue hasattracted a great deal of attention, see for example [11–18]. In particular, the consequences of population struc-ture on the evolution of reciprocal cooperation were stud-ied in [19–22]. It has also been recognized that underlyingnetwork structures such as network heterogeneity, scale-freeness etc., crucially determine the outcome of multipledynamical phenomena [23].To what extent reciprocity can explain the behaviorof biological organisms is a subject of active debate (see ∗ This work was done while the author was at MacedonianAcademy of Sciences and Arts and Faculty of Computer Science,University Goce Delcev, Stip, Republic of Macedonia. e.g. [24, 25] for a recent review). According to [24], a ma-jor concern is that the assumptions of theoretical modelsdiffer in important ways from the observed structure ofreal interactions, as supported by experimental evidence[26–28]. As a result, the mechanisms proposed in someof the theoretical models are unlikely to be realized byevolution in real organisms [6]. Particularly, direct andindirect reciprocity require cognitive abilities to registeridentity of social partners and their behavior in previousinteractions, which has been shown to constrain cooper-ation in animals [29], including humans [30].Recent empirical studies have shown that cooperationin animals (rats [31], monkeys [32], dogs [33]), as wellas humans [34, 35], can work between non-relative con-specifics by generalized reciprocity - a simple mechanismwhich does not require higher cognitive demands. Thismechanism, simply described as “help anyone if helpedby someone”, assumes that an individual who receivedhelp in the past is more likely to help any new individ-ual in subsequent interactions. Generalized reciprocity,which can be traced back to “upstream tit-for-tat” [36]and “upstream indirect reciprocity” [37]), has been re-cently addressed in detail in [24, 25].A growing body of recent empirical research indicatesthat social and cooperative behavior can be affected bycognitive and neurological factors, such as experienceand hormone titres [38]. In this context, it has beensuggested that the proximate mechanism of generalizedreciprocity is based on changes of the individuals phys-iological/neurological state [31, 34, 39]. The first stepsin the direction of understanding the evolutionary pro-cesses underlying generalized reciprocity have been madein [36, 37]. In [40] the formation of a decision-makingmechanism based on an internal state has been inves-tigated by evolutionary simulation. There it has beendemonstrated that a mechanism where the individualsbase their decision to cooperate based on a state variableupdated by the outcome of the last interaction with ananonymous partner, can emerge through small evolution-ary steps under a wide range of conditions.In the presence of supporting empirical evidence forthe evolutionary development of a state-based decision-making mechanism, we propose a general framework toaddress the dynamics of such mechanisms on complexnetworks. We adopt a simple (stochastic) model for net-work interactions, where nodes regularly send coopera-tion requests to randomly chosen neighbors. The selectednodes accept the requests (cooperate) with probabilitydetermined by a single variable - an internal cooperativestate which reflects their current “well being”. The re-sulting behavioral mechanism relates the nodes’ behaviorin the network interactions to their fitness, i.e. accumu-lated payoff in a game-theoretic jargon, with immediateimplications to a large plethora of real-life networks.From a game-theoretic perspective, the behavioralmechanism we address may be framed in the continu-ous Prisoner dilemma context [41] according to whichthe level of investment (i.e. the probability of coopera-tion) is adjusted to the accumulated experience. Whilein our stochastic model interactions happen between twoindividuals, the fact that the behavioral mechanism isoblivious to the identity of donors and receivers effec-tively provides a framework where the network nodes en-gage in a game with their neighborhood. This, in a sense,is conceptually similar to some versions of the N-playeriterated Prisoners Dilemma [42, 43].We point out that in our model we do not consider“competition” between strategies in the sense of [44].Also, we do not assume evolutionary updates in the senseof e.g. [37], or imitation of the neighbor’s strategy (e.g.imitation dynamics) [45]. Instead, we presume that ageneral form of state-based generalized reciprocity mech-anism is in place in the complex network of interest (i.e.has evolved as a result of evolution, as suggested by em-pirical evidence), effectively resulting in a continuum ofinvestment strategies with levels of investment changingaccording to the nodes individual states.With this in mind, the aim of our approach is to evalu-ate the implications of this behavioral mechanism on thecooperation in the network and, importantly, to revealthe role of network structure. In particular, we show that,in the presence of the here addressed state-based behav-ioral mechanism, the levels of individual investments mayevolve from very low to significant, eventually saturatingto a point which is solely determined by the networktopology. To address the robustness of the state-basedstrategies to an invasion by defectors, we restate the re-sults for the case when a fraction of the network nodesbehave as unconditional defectors.An important aspect of our model is the fact that theanonymity of the network interactions is not associatedwith an increased vulnerability to exploitation (as it is generally the case with generalized reciprocity [24, 25]. Infact, we are able to show that the behavioral mechanismpromotes cooperation by “driving” the network towardsa steady state beyond which the individual nodes areprotected from exploitation by the rest of the network.The remaining of the paper is structured as follows. InSec. II we define the stochastic model for network inter-actions and describe the proposed behavioral mechanism(update rule). In the same section we present the de-terministic counterpart of the stochastic model, which isanalytically tractable and represents a valid approxima-tion in the steady state regime. In Sec. III we analyze thedeterministic model and address the issue of cooperationfrom the perspective of the network topology. We furtherderive important properties of the model related to thespread and stability of cooperation in the network. Theanalytical findings are supported by numerical results.We conclude the paper by discussing the implications ofthe model and drawing parallels with other theoreticalmodels and real-life networks.
II. MODEL DESCRIPTIONA. Network model
The network is modeled as a random graph A on afinite set N of N nodes, with binary edge variables A ij ∈{ , } between pairs of nodes i, j ∈ N ( A ij = 1 , i = j indicating neighborhood relation). The interactions be-tween the nodes are modeled as follows: in each round t , node i sends a cooperation request to a randomly (onuniform) chosen node from its neighborhood, for example j ∈ N i ; upon selection, node j accepts the request (i.e.cooperates) with probability p j ( t ), which may be consid-ered as its internal cooperative state at time t ; if node j accepts the request, i.e. cooperates, it pays a cost c fornode i to receive a benefit b (we assume these quanti-ties to be the constant over the network). The (random)payoff of node i at round t isy i ( t ) = b x j ( t ) − c x i ( t ) X k ∈N i ρ k ( t ) . (1)In (1): the selected index from the neighborhood of i is a random variable uniformly distributed on the set N i , j ∼ U( N i ); x l ( t ), l = 1 , . . . , N , are Bernoulli ran-dom variables, each with parameter p l ( t ) (the coopera-tive state); ρ h is a Bernoulli random variable with param-eter 1 /d h , where d h is the degree of node h , d h = P l A h,l ;the term P h ∈N i ρ h ( t ) captures the (random) number ofnodes (neighbors of i ) which send a cooperative requestto i during round t . We note that the model (1) may beeasily extended to weighted graphs by substituting theuniform distribution with categorical. B. Behavioral mechanism (update rule)
For simplicity, we assume a synchronous behavioral up-date, based on of the accumulated (i.e. total) payoff ofthe node i by time t , Y i ( t ) = Y i ( t −
1) + y i ( t ), with Y i (0)being the initial condition and y i (0) = 0. The coopera-tive state of node i at time t + 1 is defined asp i ( t + 1) = f (Y i ( t )) , (2)where we assume that the function f : R → [0 1] is mono-tonic (nondecreasing). A plausible choice which reflectsreal-world behavior is the sigmoid (logistic) functionf( ω ) = h e − κ ( ω − ω ) i − , where the parameters κ and ω define the steepness, re-spectively the midpoint of the function.We note that it is straightforward to extend the modelto account for an asynchronous behavioral update, wherein each step t node i updates its probability of cooper-ation with probability u . In that case, the cooperativestate of node i at time instant t + 1 is defined asp i ( t + 1) = p − λi ( t ) · f λ (Y i ( t )) . The dynamics of the behavioral update, is then dictatedby the payoff accumulated by each node in the (random)time period between two updates t o and t o + T i ∆Y i ( t o , t o + T i ) . = Y i ( t o + T i ) − Y i ( t o )= b t o + T i X τ = t o +1 x j ( τ ) − c x i ( t o + 1) t o + T i X τ = t o +1 R i ( τ ) , with the comment that the index j ∈ N i is updated ineach step τ .In the following we will only address the scenario withsynchronous update, with the remark that the conclu-sions also apply to the asynchronous scenario. C. Random graphs
The starting point for studying games on graphs arethe models used in evolutionary biology, where the evo-lution of the population over time can be determined bysolving a coupled set of differential equations (the repli-cator equations, see, e.g. [46]). Besides being determin-istic (no stochasticity in the decisions), this frameworkassumes infinite, well-mixed populations.To account for stochastic game dynamics and fi-nite populations, evolutionary graph theory provides amathematical tool for representing population structure:nodes correspond to individuals and edges indicate in-teractions [19]. Graphs can describe spatially structuredpopulations of bacteria, plants or animals, tissue archi-tecture and differentiation in multi-cellular organisms, or social networks. In this context, the well-mixed popula-tion, which is a classical scenario for mathematical stud-ies of evolution, is given by the complete graph.In this setting, the structure of the underlying randomgraph dictates the final result of many real world systems,including cooperation. In general, real world networksare characterized with three properties [47]: i) high clus-tering - two nodes have a higher probability to share anedge if they have similar neighborhoods, ii) small-world- short, on average, distance (shortest path length) fromone node to another, and iii) scale-freeness - power lawdegree probability density function (pdf).Many models have been developed for generating ran-dom graphs that have (some of) these properties. Westudy the behavior of our model on the four models thatare most often implemented i) Random d -regular graph,ii) Erdos-Renyi (ER) random graph, iii) Watts-Strogatz(WS) random graph, and iv) Barabasi-Albert randomscale-free network. In the following, we describe them. • Random d -regular graph – the simplest randomgraph that can be found in the literature. Formally de-fined as a random graph A ( N, d ) in which all nodes havethe same degree d [48]. As such, it has a degree pdf (andhence a z pdf) described with the Dirac delta function,whereas clustering and shortest path length generally de-pend on the parameter d . To generate a d -regular graphwe implement the pairing algorithm described in [49].A special type of a regular graph that has been com-monly studied in evolutionary biology is the two dimen-sional (2D) N × N square lattice [10]. A such lattice ischaracterized with low (zero) clustering and long averagepath length. The main difference between a 2D squarelattice and other random regular graphs is that the struc-ture of the former is not random. Namely, in it the nodesare distributed at the integer coordinate points of the twodimensional Euclidean space and each node is connectedto other nodes that are one unit away from it. • Erdos-Renyi (ER) random graph – also known asthe A ( N, π ) model [50]. In it, two nodes share an edgewith probability π , independently from the presence ofother edges. A random graph constructed through thisalgorithm is characterized with very low clustering, longshortest path length and Poisson degree pdf. • Watts-Strogatz (WS) random graph – a model forgenerating random graphs introduced in [51] and definedas A ( N, d, β ), where d is the average degree and β ∈ [0 , d -regular ring lattice is constructed byputting the nodes on integer values of a circle with cir-cumference N + 1 and connecting them to their d nearestneighbors. Then, each generated edge ( i, j ) (with i < j )is rewired to ( i, k ), where k = i is a uniformly chosennode, with probability β . We point out that d -regularring lattices and ER random graphs emerge as specialtypes of the WS graph when β = 0 and β = 1, respec-tively. When 0 < β < • Barabasi-Albert (BA) random scale-free network – amodel based on the preferential attachment mechanismfor generating random graphs [52]. The construction ofa BA network, written as A ( N, m ), is represented as adynamical process. Concretely, in the beginning a fullyconnected network of m nodes ( N i = N \ i for all i ) iscreated. Then, at each time step a new node i is bornthat makes connections to m other nodes that are presentin the network. The node connects to a particular node j with probability proportional to its current degree. Be-sides having the same properties of high clustering andsmall-world as the WS graph, the BA graph has a scale-free degree pdf. Therefore, the BA model has been ex-tensively applied for studying real world systems, rangingfrom social to biological networks and beyond [52, 53]. D. Deterministic approximation
We approximate the stochastic model (1) by a de-terministic model (under the same behavioral update),where the random variables are substituted with theirrespective expectationsy i ( t ) = b X j A ij d i p j ( t ) − cz i p i ( t ) . (3)In (3), z i is defined as z i = X j A ji /d j . This quantity acts as a local centrality measure of a node,with node i being more “important” if it has many neigh-bors, and the neighbors themselves have few neighbors.In our model of interactions, this node would be calledupon rather frequently. The measure, that we refer toas “neighborhood importance index”, reflects the role ofnetwork topology in the promotion and stability of co-operation. When considering random walks on complexnetworks, one can show that z i is exactly the sum of thejump probabilities towards node i from its neighbors [54].When written in vector form, for (3) we have y ( t ) = Θ · p ( t ) , where Θ ii = − cz i , and Θ ij = b A ij d i , for i = j .In Fig. 1 we present the comparison between thestochastic and the deterministic model in terms of themean squared error (MSE) between the realizations ofthe individual cooperation probabilities and their deter-ministic counterparts (analytical solutions of the deter-ministic model). We observe that in the time limit thesteady state behavior of both models is almost identical.In addition, the steady state solution depends only onthe benefit-to cost ratio b/c , and the particular values of b and c only determine the rate of convergence, i.e theduration of the transient regime. III. RESULTS
Here, we address in more details the issues of coop-eration in relation to the network topology. We therebyhighlight the role of the neighborhood importance index z (more precisely its distribution over the network nodes).In particular, we describe the steady state behavior ofthe deterministic model and derive important propertiesrelated to the existence and stability of cooperation. A. Steady state behavior
The update rule (2) yields the following set of iterativeequations for i = 1 , . . . , N p i ( t + 1) = f (Y i ( t −
1) + Θ i · p ( t )) , where Θ i is the i − th row of Θ . In steady state it has tobe fulfilled p ∗ i = f (cid:0) f − (p ∗ i ) + Θ i p ∗ (cid:1) , for i = 1 , . . . , N . By applying the inverse map we getf − (p ∗ i ) = f − (p ∗ i ) + Θ i p ∗ . (4)The above requires y ∗ i . = Θ i p ∗ = 0, unless either p ∗ i = 1(i.e. Y ∗ i = f − (p ∗ i ) = ∞ ), or p ∗ i = 0 (i.e.Y ∗ i = −∞ ).It is easy to verify that if there exists i such that p ∗ i = 0,then the same is true for all i ∈ N . Indeed, when p ∗ i = 0,then from (3) it must hold that either: 1) y ∗ i >
0, or:2) p ∗ j = 0 for all j in the neighborhood of i , j ∈ N i .The condition 1 implies p ∗ i = 1, which is a contradiction.The condition 2 yields p ∗ i = 0 for all i ∈ N by repeatingthe same argument to the nodes in the neighborhood of i , until all nodes are reached. We note that this caseis also covered by the requirement Θ i p ∗ = , with thesolution p ∗ = . Hence, a steady state solution fulfills p ∗ ∈ ∪ (0 1] N and is thereby characterized by non-negative steady state payoffs y ∗ i ≥
0. We note that asteady state p ∗ = is also reached whenever the initialconditions are p (1) = . We will, however, exclude thistrivial possibility in the analysis that follows.In steady state, the nodes may thus be attributed totwo (disjoint) sets, W = { w ∈ N : y ∗ w = 0 } and S = { s ∈ N : y ∗ s > } , depending on the steady state payoffy ∗ i . As a consequence of (4), the nodes in S are furthercharacterized by p ∗ i = 1, while the nodes in W may takeboth values p ∗ i = 1 and p ∗ i <
1, depending on the net-work parameters. We will refer to the nodes in the sets W and S as “weak”, respectively “strong” nodes, withan intention to emphasize their role in the bifurcationanalysis performed later. Accordingly, there are two setsof relations that have to be satisfied0 = bd i X j A ij p ∗ j − cz i p ∗ i , i ∈ W y ∗ i = bd i X j A ij p ∗ j − cz i , i ∈ S . (5) × t01 M S E (a) 0 5 × t01 M S E (b) 0 5 × t01 M S E (c) 0 5 × t01 M S E (d)0 5 × t01 M S E (e) 0 5 × t01 M S E (f) 0 5 × t01 M S E (g) 0 5 × t01 M S E (h)0 5 × t01 M S E (i) 0 5 × t01 M S E (j) 0 5 × t01 M S E (k) 0 5 × t01 M S E (l) FIG. 1. Mean squared error (MSE) comparison of the stochastic and the deterministic models. For the stochastic model, theresults are averaged across 100 network realizations. (a)
Regular graph. (b)
ER graph. (c)
BA graph. (d)
WS graph. (a-d) b/c = 1 . (e-h) Same as (a-d) , only b/c = 1 . (i-l) Same as (a-d) , only b/c = 1 . (a-l) A lighter shade indicates lowerparameter values. All graphs have 100 nodes and average degree 8. In each run we set the initial values Y i (0) = − p i (0) = 0 .
05 for all i . We assume synchronous update. Note that in (5) the sets W , S , the steady state valuesp ∗ i , i ∈ W and the constants y ∗ i , i ∈ S are unknown. B. Properties of the model
In the following we derive some important propertiesof the model. In particular, we investigate the necessaryand sufficient conditions for the spread and stability ofcooperation, in relation to the network topology. . Robustness to exploitation: The non-negativity of theindividual steady state payoffs y ∗ i in (5) has an impor-tant implication on the promotion of cooperation in net-works as it ultimately protects the individual nodes fromexploitation by the rest of the network. This is, in gen-eral, at contrast to other mechanisms based on generalreciprocity where the anonymity of donors and receiversmakes it difficult to single out and punish defectors, leav-ing the nodes vulnerable to exploitation. Fig. 2 illustrates the range of the individual steady state payoffs and theaverage network payoff h y ∗ i as a function of the bene-fit/cost ratio b/c , for the different network models. Necessary condition for the existence of cooperation:
It is easy to show that b/c < i = 0 for all i ∈ N . Indeed, if there exists i such that p ∗ i >
0, thenthe total steady state network payoff X i y ∗ i = b X i X j A ij d i p ∗ j − c X i X j A ji d j p ∗ i = ( b − c ) X i z i p ∗ i , (6)is strictly negative, implying that there is some i forwhich y ∗ i < b/c ≥ ∗ i > Promotion of cooperation:
We observe that when b/c >
1, the steady state probabilities are strictly greaterthan 0, p ∗ i > i ∈ N . Indeed, if there exists i suchthat p ∗ i = 0 then, as already discussed, it must hold thatp ∗ i = 0, for all i ∈ N . This, however, would yield a totalnetwork payoff P i y ∗ i = 0, which contradicts (6). Sufficient condition for the existence of unconditionalcooperators (strong nodes):
When b/c >
1, there is al-ways at least one strong node in the network. This fol-lows directly from the observation that when b/c > i for which y ∗ i > ∗ i = 1.Combined together, property 3. and property 4. statethat, as a consequence of the update rule 2, the networknodes cooperate with the maximum possible probability,such as their pay-off is non-negative. In other words,they are not exploited by their environment. Necessary condition for the existence of unconditionalcooperators (strong nodes):
The condition z i > b/c im-plies p ∗ i <
1, which follows by substituting P j A ij d i p ∗ j ≤ i to bestrong (i.e. unconditional cooperator p ∗ i = 1) is z i ≤ b/c . Necessary and sufficient condition for full network co-operation:
We show that b/c ≥ z max , (7)where z max is the largest neighborhood importance in-dex in the graph, z max = max i ( z i ), is both necessary andsufficient for all nodes to be strong (full network cooper-ation).We note that the proof that p ∗ i = 1, ∀ i ∈ N , implies b/c ≥ z max , follows directly from property 3 . To provethe converse, we use contradiction. We first define p min =inf p ∗ i , i ∈ N , and set b/c to be greater than one ( b/c > b/c ≥ z i for all i , and there exists some i such thatp ∗ i <
1. Under this assumption, for all i we would have:y ∗ i = b X j A ij d i p ∗ j − cz i p ∗ i , ≥ b X j A ij d i p ∗ j − c bc p ∗ i , which implies p ∗ i + y ∗ i b ≥ X j A ij d i p ∗ j . However, for those i satisfying p ∗ i <
1, we know thaty ∗ i = 0, implyingp ∗ i ≥ X j A ij d i p ∗ j ≥ p min . (8)For all i satisfying b/c > z i , (8) holds with strict inequal-ity, whereas those i ′ for which b/c = z i ′ must satisfyp ∗ i ′ = p min . This, however, can hold if and only if thenodes corresponding to these indices are only linked to b/c h y ∗ i (a) 1 2 3 4 5 b/c h y ∗ i (b)1 2 3 4 5 b/c h y ∗ i (c) 1 2 3 4 5 b/c h y ∗ i (d) FIG. 2. Average steady state payoff h y ∗ i as a function of b/c (solid line), averaged over 100 graph realizations. The regions(in lighter shade) enclose the area between the minimum andthe maximum node payoff observed in the network. (a) Reg-ular graph. (b)
ER graph. (c)
BA graph. (d)
WS graph. (a)-(d)
All graphs have 1000 nodes and average degree 8. each other, i.e. form a connected component. In thatcase z max = 1 = b/c which contradicts the assumption b/c >
1. Hence, the converse must also be true, whichconcludes the proof.
C. Derivation of the steady state solution
Having addressed the general steady state behavior ofthe model and derived its most important properties,here we provide bifurcation analysis to determine thesteady state solution of (5). To do so, we keep the cost c fixed and vary the benefit b in order to determine theirinfluence on the system. Fig. 3 serves as a graphicalillustration for the analysis.We start with the remark that b/c ≥ z max ensuresthat all nodes are unconditional cooperators (i.e. strong),p ∗ i = 1, for all i ∈ N . This follows directly from nec-essary and sufficient condition for full network coopera-tion (property .) . We denote b = cz max and introduce W as the set of nodes with indices w ∈ W satisfying z w = b /c (note that W may have more than one el-ement). For i ∈ W the payoff becomes y ∗ i = 0, whileit is still p ∗ i = 1. By reducing b beyond b , for i ∈ W the probability of cooperation becomes p ∗ i < ∗ i = 1 would imply a payoffy ∗ i < ∗ i are determinedfrom (5), by plugging in p ∗ i < i ∈ W . Thus, b isthe first bifurcation point and the nodes in the set W arethe first to become weak, i.e. to break up with uncondi-tional cooperation. The remaining nodes S (1) = N \ W are still strong.We proceed by using induction to determine the setof weak nodes W ( n +1) = S ni =1 W i and the (remaining)strong nodes S ( n +1) = N \W ( n +1) for any b in the interval FIG. 3. (a)
A visualization of a small random graph with10 nodes. The nodes are colored according to the order ofswitching from strong to weak when decreasing the b/c ratio– a darker color indicates higher switching propensity. Thenode size is proportional to the index z i . (b) Fraction ofstrong nodes σ as a function of the (decreasing) b/c ratio. between two bifurcation points, b n > b ≥ b n +1 . Aftersubstituting p ∗ i = 1 for the nodes in S ( n +1) , we determinep ∗ i for the nodes in W ( n +1) from the remaining equations.By applying some simple algebra to reorganize theequations for the nodes in the set W ( n ) , for all b in theinterval between b n and the next bifurcation point b n +1 , b n > b ≥ b n +1 , we obtain X j ∈S ( n ) bcz i d i A ij = p ∗ i − X j ∈W ( n ) bcz i d i A ij p ∗ j , i ∈ W ( n ) . The last equation can be written in matrix form a = ( I − A W ( n ) ) p ∗ , (9)where the vector a is the vector of sums appearing on theleft-hand side of (9), while A W ( n ) is a weighted version ofthe neighborhood matrix of the subgraph associated with W ( n ) . The solution is unique provided that the matrix I − A W ( n ) is nonsingular, i.e. the inverse ( I − A W ( n ) ) − exists. In that case the solution reads p ∗ = ( I − A W ( n ) ) − a . We note that in reality, and particularly in large net-works, the nonsingularity condition for I − A W ( n ) is ful-filled almost certainly. D. Alternate projection method for the steadystate solution
The bifurcation analysis provides an analytical solu-tion to the steady state cooperation probabilities for each b/c >
1. By the definition of steady state, the same so-lution would be obtained by starting from an arbitraryinitial condition p (0) ∈ (0 1] N and letting the networkevolve according to our payoff and update rules. Note that the steady state conditions may be reformulated asy ∗ i = b X j A ij d i p ∗ j − cz i p ∗ i ∗ i (1 − p ∗ i ) , resulting in a nonlinear (quadratic) system of 2 N equa-tions with 2 N variables in total. Hence, a simplified, iter-ative approach based on the alternate projection methodcan be used for finding the steady state solution. Thismethod may be summarized as follows:1. Set y ∗ i = 0 for all i satisfying the condition z i ≥ b/c .Set p ∗ i = 1 for the remaining nodes. Solve the N -dimensional linear system to find the remaining p ∗ i and y ∗ i ( N unknowns in total)2. For all i satisfying y ∗ i < ∗ i = 0 and let their corresponding p ∗ i tobe unknown. Solve again the corresponding linearsystem of N equations with N unknowns in total.3. Repeat steps 1 . and 2 . until there are no y ∗ i < E. Implications of the model
While the connection is not imminent, the addressedmodel may still be framed in the context of evolutionarygame theory, by associating the accumulated payoff withfitness of the individual nodes. In this sense, our maincontribution can be related to the results for the knownrules for evolution of cooperation (see Table I). In par-ticular, it can be seen as a rule for cooperation based ongeneralized reciprocity on graphs. One has to be care-ful, however, when comparing the different rules, as theyusually arise from different setups and are dependent onthe interaction model/game update. For example, themodel in [40] assumes pairwise node interactions whereeach node pair is chosen on random during each iteration,yielding a uniform distribution on the number of interac-tion instances per node. In our model, on the other hand,some nodes (i.e. those with a large z -index) engage moreregularly in the interactions with their neighbors, thus re-flecting the role of network structure. With this in mind,and after accounting for the differences in the interactionmodel, a natural connection may be made between thepresented mechanism based on generalized reciprocity ongraphs, and the mechanisms based on indirect reciprocity[55], network reciprocity (revisited) [56], and generalizedreciprocity [40], as highlighted in Table I.We turn the attention to Fig. 4a which sheds light onthe effect of network structure on the promotion of coop-eration under the addressed model. As an indicator forthe level of network cooperation we consider the steadystate fraction of unconditional cooperators σ as a func-tion of the benefit/cost ratio b/c ( b > c is the prerequisitefor cooperation). The figure reveals that the Barabasi-Albert (BA) scale-free graph requires the largest b/c for b/c σ (a) Regular ER BA WS1.1 1.2 1.3 b/c σ z p z ( z ) (b) FIG. 4. (a)
Fraction of unconditional cooperators σ as a function of b/c for Regular, ER, BA, and WS graphs. Dashed linesindicate the threshold for full network cooperation according to equation (7). (b) Probability density function (pdf) of theindex z for the same random graphs. We note that for the Regular graph the pdf is concentrated on a single point (i.e theindex z is the same across all nodes, z = 1). (a-b) The results are obtained by averaging over 100 different graph realizations.All graphs have 1000 nodes and average degree 8. full network cooperation to take place (all nodes uncon-ditional cooperators), followed by the Erdos-Renyi (ER)graph and the Watts-Strogatz (WS) small-world graph.In contrast, for a small b/c ratio, the BA graph has thelargest fraction of cooperators among these three graphtypes. The Regular graph presents itself as the mostsupportive to cooperation, as b > c implies full, uncondi-tional cooperation on network level.The reason for this behavior may be directly inferredfrom Fig. 4b, where we depict the probability densityfunction (pdf) of the index z across the network nodesfor the same random graphs. We recall that that the indi-vidual cooperative behavior in the network is determinedby this index. The nodes with a higher value of it arein “less favorable” position in the network, as they willbe called upon more often in our interactions model. Forhigher values of the benefit-to-cost ratio b/c , the globalcooperative behavior is dominated by the tail (the right-hand side) of the distribution, i.e. the fraction of networknodes with high values of the z -index. This explains ex-actly the lower fraction of unconditional cooperators inthe BA graph in this regime, as compared to the otherrandom graph configurations. On the other hand, thebehavior in the low b/c regime is determined by the left-hand side of the distribution, where again a large massis distributed in the case of the BA graph. As a result,in this regime we find higher fraction of unconditionalcooperators in the BA graph compared to the ER andthe WS graph, as shown in the box of Fig. 4a. F. Stability of cooperation
So far, we addressed the case when all nodes were sub-ject to the same behavioral mechanism (2) to updatetheir individual probabilities of cooperation. In the fol-lowing, we assume that a fraction δ = D/N of the nodes become unconditional defectors (p i ( t ) = 0), independenton their accumulated payoff. The aim is to assess therobustness of the (in general cooperative) behavioral up-date to the presence of defectors.The bifurcation analysis as well as the alternate pro-jection method can be easily accommodated to accountfor this characteristic. Additionally, several propertiesderived for the original case can be generalized to thissetup. Particularly, in the presence of unconditional de-fectors, the necessary condition for existence of uncon-ditional cooperators ( property . ), becomes d i q i z i ≤ b/c ,where q i is the number of neighbors of i that are notunconditional defectors. In addition, the necessary andsufficient condition for full network cooperation ( property . ) now reads b/c ≥ max i d i q i z i . The remaining proper-ties can not be easily generalized as they depend on theinitial selection of defecting nodes.Numerical results are summarized in Fig. 5 where weplot a heat map for the fraction of “defective instances”as a function of δ and b/c . By defective instance we un-derstand the absence of a strong node in the networkin steady state. Interestingly, the BA graph presents it-self as the most robust to an invasion of defectors, asit registers the smallest number of defective instances δ for the same ratio b/c . The ER graph provides to beslightly less robust than the BA graph. As least robustto an invasion of defectors come the WS and the Regulargraph, which present similar behavior in this setup. Thereason for this behavior becomes apparent if we, again,look at the distribution of the index z in different randomgraphs. Since in this case we are interested in the num-ber of defective instances (i.e. graph realizations withoutunconditional cooperators in the network), decisive is theshape of the left-hand-side of the pdf of z . As the exis-tence of unconditional defectors in the neighborhood ofa node i decreases the payoff of node i , its influence onthe cooperative behavior of node i may be understood as TABLE I. Cooperation Mechanisms
Mechanism Rule Note
Kin selection b/c > /g g denotes the probability that two agents share a gene [2].Group selection b/c > n/m n and m , are respectively, the maximum group sizeand number of groups [4].Direct reciprocity b/c > /w w is the probability that a game will last one more round [7].Network reciprocity b/c > h d i h d i is the average number of neighbors [19].Network reciprocity b/c > h d i h d i is the average degree of the nearest neighbors [56].(revisited)Indirect reciprocity b/c > /q q is the probability to know someones reputation [55].Generalized reciprocity b/c > ( v + n ) / ( v − n ) v denotes the number of interactions, and n is the group size [40].Generalized reciprocity b/c ≥ z max z max is the maximum neighborhood importance index.on graphs (a)1 3 5 7 b/c . δ (b)1 3 5 7 b/c . δ (c)1 3 5 7 b/c . δ (d)1 3 5 7 b/c . δ FIG. 5. Heat map for the fraction of defective instances asa function of δ = D/N and b/c . D out of N nodes are ran-domly chosen to be unconditional defectors and the resultsare averaged over 100 realizations. (a) Regular graph. (b)
ER graph. (c)
BA graph. (d)
WS graph. (a)-(d)
All graphshave 100 nodes and average degree 8. Results are averagedacross 100 graph realizations. an “effective” increase of the value of the index z i . This,on the other hand may drive unconditional cooperatorsto extinction (a defective distance is declared for the par-ticular configuration). As depicted in Fig. 4b, due to thehigher fraction of nodes with small values of the index z ,the BA graph is on average less affected by this condition,i.e. it is more robust to an invasion of defectors. G. Accounting for restricted memory
The state-based behavioral rule (2) implicitly assumesthat each past event is valued equally. However, some-times the agents (nodes) have short memory, and thus may give higher importance to recent interactions. Wecan account for this effect by adding a weight r , 0 ≤ r ≤
1, to the accumulated payoff, i.e. Y i ( t ) = rY i ( t −
1) + Θ i p ( t ) . We note that the case r = 1 corresponds to the update(2). When r <
1, the steady state probability for coop-eration for node i isp ∗ i = f (cid:18) Θ i p ∗ − r (cid:19) = f (cid:18) y ∗ i − r (cid:19) . (10)In this case, the accumulated payoff converges to a finitevalue since the steady state payoffs represent converginggeometric series, implying 0 < p ∗ i <
1. We point outthat, with the introduction of the weight r , exploitationcan not be prevented by employing the update rule (2) forsmall enough b/c ratios, as in some cases it may happenthat y ∗ i < i . This can be seen by applying theinverse map to (10), as we gety ∗ i = (1 − r ) f − (p ∗ i ) , which can be negative if f − (p ∗ i ) is negative. This hap-pens, for example, if f ( · ) is the standard logistic function,with steepness k = 1 and midpoint ω = 0, and p ∗ i < / β ∗ for which if b/c ≥ β ∗ exploitation is prevented. As can be seen from equation(10) the threshold depends on the underlying networktype and the parameter r , and can not be easily derived.Nevertheless, we can provide its lower and upper bounds, β LB and β UB that are independent from r . In particular,we can safely assume that β LB = 1 because, as previ-ously said, b/c > β UB , we again assume the ex-istence of the minimal probability for cooperation p min (its presence is a valid assumption since, from (10), weknow that p i > i ). Then, from (5), it follows0that y i ≥ b p min − cz i p i . Therefore, in order for y i to benonnegative for all i it must be that b p min ≥ cz i p i ≥ cz i p min for all i , or b/c ≥ z max . Interestingly, this is the samevalue as the condition for full network cooperation in theinfinite memory case. When the network is regular, thebounds coincide, thus leading to β ∗ = 1, a result whichis independent of r .Finally, it is worth mentioning that, when r = 0, f( · )is very steep (such that p i ( t + 1) = 1 if y i ( t ) >
0, andp i ( t + 1) = 0, otherwise) and setting N = 2, we get thetit-for-tat strategy in the iterated prisoner’s dilemma. IV. DISCUSSION
We argue that the (stochastic) equations (1) and (2)provide a realistic model for the dynamics of networkinteractions in a large plethora of real-life networks. Un-der this model, network cooperation comes as a resultof the inherent feature of the behavioral mechanism thatprevents participating nodes from being exploited by theenvironment. In particular, the simple update rule (2)requires minimal cognitive abilities and very little infor-mation retention and retrieval as the decisions of individ-uals to cooperate or not, only depend on their internalstate which captures the past experience of their inter-actions with the network. The internal state may mirrorfitness in biological systems, wealth or well-being in ani- mal and human societies, or battery level (energy) in ar-tificial systems (e.g. wireless ad-hock networks). Due toits simplicity, this behavioral mechanism is more likely toevolve in real networks than, e.g. direct or indirect typesof reciprocity, which require much more specific memory,cognitive ability and effort.A corollary of our findings is that, under the addressedmodel for network interactions (1), and the behavioralmechanism (2), the BA scale-free graph is the most in-hibitive to cooperation, followed by the ER graph and theWS small-world graph, while the regular graph presentsitself as the most supportive for cooperation (as displayedin Fig. 4 and Fig. 2). The picture is inverted in the pres-ence of pure defectors in the network, as then, the BAgraph provides the highest degree of robustness, followedby the ER graph (as captured by Fig. 5).We point out that, while most of our conclusions arealong the same lines in [10, 56, 57], they do not contra-dict the findings in [58, 59] which suggest that, underthe model addressed there, scale-free networks enhancecooperation. We argue that the apparent inconsistenciesfound in the literature, may be attributed to the differ-ences in the way that network interactions are modeled.Therefore, it is important that all findings in this contextare, in general, interpreted in light of the specifics of theaddressed model only.
ACKNOWLEDGEMENT
This research was supported in part by DFG throughgrant “Random search processes, L´evy flights, and ran-dom walks on complex networks”. [1] R. M. Axelrod,
The evolution of cooperation (Basicbooks, 2006).[2] W. Hamilton, J. Theor. Biol. , 1 (1964).[3] G. C. Williams, Evolution , 398 (1957).[4] A. Traulsen and M. A. Nowak,Proc. Natl. Acad. Sci. USA , 10952 (2006).[5] S. A. West, A. S. Griffin, and A. Gardner, J. Evol. Biol. , 415 (2007).[6] T. Clutton-Brock, Nature , 51 (2009).[7] R. L. Trivers, Quart. Rev. Biol. , 35 (1971).[8] M. A. Nowak, Science , 1560 (2006).[9] L. Lehmann and L. Keller, J. Evol. Biol. , 1365 (2006).[10] M. A. Nowak and R. M. May, Int. J. Bifurc. Chaos , 35(1993).[11] M. A. Nowak, A. Sasaki, C. Taylor, and D. Fudenberg,Nature , 646 (2004).[12] G. Szab´o and J. Vukov, Phys. Rev. E , 036107 (2004).[13] E. Lieberman, C. Hauert, and M. A. Nowak, Nature , 312 (2005).[14] J. M. Pacheco, A. Traulsen, and M. A. Nowak, Phys.Rev. Lett. , 258103 (2006).[15] Z. Rong, X. Li, and X. Wang, Phys. Rev. E , 027101(2007).[16] S. Morita, Progr. Theor. Phys. , 29 (2008). [17] Z. Wang, L. Wang, and M. Perc, Phys. Rev. E , 052813(2014).[18] A. Skanata and E. Kussell,Phys. Rev. Lett. , 038104 (2016).[19] H. Ohtsuki, C. Hauert, E. Lieberman, and M. A. Nowak,Nature , 502 (2006).[20] F. C. Santos and J. M. Pacheco,J. Evol. Biol. , 726 (2006).[21] P. D. Taylor, T. Day, and G. Wild, Nature , 469(2007).[22] D. J. Rankin and M. Taborsky, Evolution , 1913(2009).[23] S. N. Dorogovtsev, A. V. Goltsev, and J. F. Mendes,Rev. Mod. Phys. , 1275 (2008).[24] G. S. van Doorn and M. Taborsky,Evolution , 651 (2012).[25] M. Taborsky, J. G. Frommen, and C. Riehl,Philos. Trans. Roy. Soc. B. (2016).[26] C. Hauert, S. De Monte, J. Hofbauer, and K. Sigmund,Science , 1129 (2002).[27] R. J. Darren P. Croft and J. Krause, Exploring AnimalSocial Networks (Princeton University Press, 2008).[28] R. C. Connor, Philos. Trans. Roy. Soc. B. , 2687(2010). [29] J. R. Stevens, F. A. Cushman, and M. D. Hauser, Annu.Rev. Ecol. Evol. Syst. , 499 (2005).[30] M. Milinski and C. Wedekind, Proc. Natl. Acad. Sci. USA , 13755 (1998).[31] C. Rutte and M. Taborsky, PLoS Biol , e196 (2007).[32] K. L. Leimgruber, A. F. Ward, J. Widness, M. I. Norton,K. R. Olson, K. Gray, and L. R. Santos, PloS one ,e87035 (2014).[33] N. Gfrerer and M. Taborsky, Sci. Rep. (2017).[34] M. Y. Bartlett and D. DeSteno, Psychol. Sci. , 319(2006).[35] L. Stanca, J. Econ. Psychol. , 190 (2009).[36] R. Boyd and P. Richerson,Soc. Networks , 213 (1989).[37] M. A. Nowak and S. Roch,Philos. Trans. Roy. Soc. B. , 605 (2007).[38] E. Fox, A. Ridgewell, and C. Ashwin, Philos. Trans. Roy.Soc. B. , 1747 (2009).[39] A. M. Isen, Adv. Exp. Soc. Psychol. , 203 (1987).[40] Z. Barta, J. M. McNamara, D. B. Husz´ar, andM. Taborsky, Philos. Trans. Roy. Soc. B. , 843 (2011).[41] L. M. Wahl and M. A. Nowak, J. Theor. Biol. , 307(1999).[42] M. Doebeli and C. Hauert,Ecology Letters , 748 (2005).[43] C. O’Riordan and H. Sorensen, “Stable co-operation in the n-player prisoner’s dilemma:The importance of community structure,” in ALAMAS 2005-2007, Revised Selected Papers (Springer, 2008) pp. 157–168.[44] R. Axelrod and W. D. Hamilton, Science , 1390(1981).[45] J. M. Smith and G. Price, Nature , 15 (1973).[46] J. Hofbauer and K. Sigmund,
Evolutionary games andpopulation dynamics (Cambridge university press, 1998).[47] M. E. J. Newman, Siam Review , 167 (2003).[48] B. Bollob´as, Modern graph theory , Vol. 184 (Springer Sci-ence & Business Media, 2013).[49] J. H. Kim and V. H. Vu, in
Proceedings of the thirty-fifthannual ACM symposium on Theory of computing (ACM,2003) pp. 213–222.[50] P. Erdos and A. R´enyi, Publ. Math. Inst. Hung. Acad.Sci , 17 (1960).[51] D. J. Watts and S. H. Strogatz, Nature , 440 (1998).[52] A.-L. Barab´asi and R. Albert, science , 509 (1999).[53] A.-L. Barabasi and Z. N. Oltvai, Nature reviews. Genet-ics , 101 (2004).[54] Y.-Y. Chen, Q. Gan, and T. Suel, in Proceedings of thethirteenth ACM international conference on Informationand knowledge management (ACM, 2004) pp. 381–389.[55] M. A. Nowak and K. Sigmund, Nature , 573 (1998).[56] T. Konno, J. Theor. Biol. , 224 (2011).[57] F. Fu, L. Wang, M. A. Nowak, and C. Hauert,Phys. Rev. E , 046707 (2009).[58] F. C. Santos and J. M. Pacheco,Phys. Rev. Lett. , 098104 (2005).[59] F. C. Santos, J. M. Pacheco, and T. Lenaerts,Proc. Natl. Acad. Sci. USA103