On Gradient like Properties of Population games, Learning models and Self Reinforced Processes
aa r X i v : . [ m a t h . D S ] S e p On Gradient like Properties ofPopulation games, Learning models andSelf Reinforced Processes
Michel BenaimInstitut de Math´ematiques,University of NeuchˆatelAugust 20, 2018
Abstract
We consider ordinary differential equations on the unitsimplex of R n that naturally occur in population games, mod-els of learning and self reinforced random processes. Gener-alizing and relying on an idea introduced in [11], we pro-vide conditions ensuring that these dynamics are gradientlike and satisfy a suitable ”angle condition”. This is used toprove that omega limit sets and chain transitive sets (undercertain smoothness assumptions) consist of equilibria; andthat, in the real analytic case, every trajectory converges to-ward an equilibrium. In the reversible case, the dynamics areshown to be C close to a gradient vector field. Properties ofequilibria -with a special emphasis on potential games - andstructural stability questions are also considered. Keywords
Gradient-Like dynamics, Mean field approximation, Pro-cesses with reinforcement, Non linear Markov chains, Populationgames, Potential games, Nash Equilibria
Contents Motivating examples 5 F and F π are generally unrelated 105 Gradient Like Structure 12 Let S be a finite set, say S = { , . . . , n } . A rate matrix over S is a n × n matrix L such that L ij ≥ i = j and P j L ij = 0 . We let R ( S ) denote the space of such matrices. For x ∈ R n and L ∈ R ( S )we let xL denote the vector defined by ( xL ) i = P j x j L ji . Let ∆ = { x ∈ R n : x i ≥ , X i x i = 1 } be the unit simplex of probabilities over S. In this paper we areinterested in ordinary differential equations on ∆ having the form dxdt = xL ( x ) := F ( x ) (1)2here L : ∆ R ( S ) is a sufficiently smooth function. Such dy-namics occur - through a natural averaging procedure- in modelsof games describing strategic interactions in a large population ofplayers, as well as in certain models of learning and reinforcement.These models are usually derived from qualitative assumptions de-scribing the ”microscopic” behavior of anonymous agents, and it isusually believed or assumed that similar qualitative microscopic be-haviors should lead to similar global dynamics. However there is nosatisfactory general theory supporting this belief.To be more precise, under the assumption that L ( x ) is irre-ducible, there exists a unique ”invariant probability” for L ( x ) , π ( x ) ∈ ∆ characterized by π ( x ) L ( x ) = 0 . (2)Several models corresponding to different rate functions x L ( x )have the same invariant probability function x π ( x ) . For instance,to each population game (see section 2.1) which average ODE isgiven by (1), there is a canonical way to define a learning process(see section 2.2) which average ODE is given by dxdt = − x + π ( x ) := F π ( x ) , (3)but there is no evidence that the dynamics of (1) and (3) are relatedin general.The purpose of this paper is to provide sufficient conditions on π ( x ) ensuring that (1) has a gradient-like structure. This heavilyrelies on an idea introduced in [11] where it was shown that therelative entropy between x and π ( x ) is a strict Lyapounov functionfor systems of Gibbs type. We extend this idea to other class ofsystems beyond systems of Gibbs type, including population gamesand reinforcement process with imitative dynamics, and investigatefurther dynamics properties.To give the flavor of the results presented in this paper, let π :∆ ˙∆ be a smooth function mapping ∆ into its relative interior.Let χ π be the set of vector fields having the form given by (the righthand side of) (1), where for each x, L ( x ) is irreducible and verifies(2). Note that χ π is a convex set of vector fields on ∆ and that F π ∈ χ π . heorem A For all F ∈ χ π . (i) Equilibria (respectively non degenerate equilibria) of F coincidewith equilibria (respectively non degenerate equilibria) of F π . (ii) In general, global dynamics of F and F π are ”unrelated.” Weconstruct an example for which F π is globally asymptoticallystable (every trajectory converge toward a linearly stable equi-librium) while every non equilibrium trajectory for F convergeto a limit cycle. (iii) Assume that there exists a C k , k ≥ s : R R such that x ∈ ˙∆ s (( x i π i ( x ) )) i ∈ S is the gradient(or quasi gradient) of some function V : ˙∆ R . Then (a) V is a strict Lyapounov function for F ( F is gradient-like)and verifies an angle condition , (b) Omega limit sets and chain-transitive sets of F are equi-libria, (c) In the real analytic case, every solution to (1) convergetoward an equilibrium, (d)
In the reversible case, hyperbolic equilibria of F coincidewith non degenerate critical points of V and, providedthere are finitely many equilibria, F is C close to a gra-dient vector field for a certain Riemannian metric, (e) The set χ π is not (in general) structurally stable.Section 2 describes a few examples that motivate this work. Sec-tion 3 contains some preliminary results and the main assumptions.Section 4 is devoted to Theorem A, ( ii ); Section 5 to ( iii ) , ( a ) , ( b ) , ( c );Sections 6 and 7 to ( iii )( d ) and Section 8 to ( iii )( e ) . Other resultsand examples are also discussed in these sections. For instance, inSection 6.1, the local dynamics (dynamics in the neighborhood ofequilibria) of mean field systems associated to potential games isprecisely described in term of Nash equilibria.4
Motivating examples
Throughout this section we see S as set of pure strategies . A Markovmatrix over S is a n × n matrix K such that K ij ≥ P j K ij = 1 . We let M ( S ) denote the sets of such matrices and we assume givena lipschitz map K : ∆ M ( S ) . For further reference we may call such a map a revision protocol.
This terminology is borrowed from [24].
Good references on the subject include [23] and the survey paper[24] from which some of the examples here are borrowed.Consider a population of N agents, each of whom chooses a strat-egy in S at discrete times k = 1 , , . . . . Depending on the context,an agent can be a player, a set of players, a biological entity, a com-munication device, etc. The state of the system at time k ∈ N is thevector X Nk = ( X Nk, , . . . X Nk,n ) ∈ ∆ where N X
Nk,i equals the number ofagents having strategy i. The system evolves as follows. Assume thatat time k the system is in state X Nk = x. Then an agent is randomlychosen in the population. If the chosen agent is an i − strategist,he/she switches to strategy j with probability K ij ( x ) . This makes( X Nk ) k ≥ a discrete time Markov chain, which transition probabilitiesare P ( X Nk +1 = x + 1 N ( e j − e i ) | X Nk = x ) = x i K ij ( x )where ( e , . . . , e n ) is the standard basis of R n . Let L ( x ) = − Id + K ( x ) . (4)By standard mean-field approximation (see [16], [8] for precise state-ments, and [23], [24] for discussions in the context of games), theprocess { ( X Nk ) : kN ≤ T } can be approximated by the solution to(1) (with L ( x ) given by (4)) with initial condition x = X N , over thetime interval [0 , T ] . evision protocols Assume, as it is often the case in economic or biological applications,that the population game is determined by a continuous
Payoff-function U : ∆ R n . The quantity U i ( x ) represents the payoff(utility, fitness) of an i − strategist when the population state is x. An attachment-function is a continuous map w : ∆ R n + . Theweight w ij ( x ) can be seen as an a priori attachment of an i − strategistfor strategy j. It can also encompasses certain constraints on thestrategy sets. For instance w ij ( x ) = 0 (respectively w ij ( x ) <<
1) ifa move from i to j is forbidden (respectively costly). We call theattachment function imitative if w ij ( x ) = x j ˜ w ij ( x ) (5)Most, not to say all, revision protocols in population games fall intoone of the two next categories: (i) [Sampling] K ij ( x ) = w ij ( x ) f ( U j ( x )) P k w ik ( x ) f ( U k ( x )) (6)where f is a non negative increasing function. (ii) [Comparison] K ij ( x ) = w ij ( x ) g ( U i ( x ) , U j ( x )) for i = j (7) K ii ( x ) = 1 − X j = i K ij ( x )where g ( u, v ) is nonnegative, decreasing in u , increasing in v and such that P j = i K ij ( x ) ≤ . Suppose now there is only one single agent in the population. In thecontext of games, one can imagine that this agent consists of a finiteset of players and that S is the cartesian product of the strategy sets6f the players. let X k ∈ S denote the strategy of this agent at time k. Let µ k ∈ ∆ denote the empirical occupation measure of ( X k ) upto time k. That is µ k = 1 k k X j =1 δ X j where δ : S ∆ is defined by δ i = e i . Suppose now that the agentrevises her strategies as follows: P ( X k +1 = j | X , . . . , X k − , X k = i ) = K ij ( µ k ) . The process ( X k ) is no longer a Markov process but a process withreinforcement (see [21] for a survey of the literature on the subject).Using tools from stochastic approximation theory, it can be shown(see [1]) that, under certain irreducibility assumptions, the long termbehavior of ( µ k ) can be precisely related (see [1], [2], [4] and the briefdiscussion preceding corollary 5.15) to the long term behavior of thedifferential equation on ∆ dxdt = − x + π ( x ) (8)where π ( x ) ∈ ∆ is the invariant probability of K ( x ) . Note that (8)can be rewritten as (1) with L ij ( x ) = δ ij − π j ( x ) . Let L be a rate matrix, as defined in the introduction. Then L is theinfinitesimal generator of a continuous time Markov chains on S. Aprobability π ∈ ∆ is called invariant for L if it is invariant for theassociated Markov chain, or equivalently πL = 0 . A sufficient condition ensuring that π ∈ ∆ is invariant is that L is reversible with respect to π, meaning that π i L ij = π j L ji . irreducible if for all i, j ∈ { , . . . , n } there existsome integer k and a sequence of indices i = i , i , . . . , i k − , i k = j such that L i l ,i l +1 > l = 1 , . . . , k − . An irreducible rate matrix admits a unique invariant probabilitywhich can be expressed as a rational function of the coefficients ( L ij )(see e.g Chapter 6 of [12]).The relative interior of ∆ is the set˙∆ = { x ∈ ∆ : ∀ i ∈ S, x i > } . From now on we assume given a C map L : ∆ R ( S ) satisfyingthe following assumption: Hypothesis 3.1 (standing assumption)
For all x ∈ ˙∆ , L ( x ) isirreducible. We sometimes assume
Hypothesis 3.2 (occasional assumption)
For all x ∈ ∆ , L ( x ) is irreducible. In view of the preceding discussion Hypotheses 3.1 and 3.2 implythe following
Lemma 3.3
There exists a C map π : ˙∆ ˙∆ such that for all x ∈ ˙∆ , y = π ( x ) is the unique solution to the equation yL ( x ) = 0 , y ∈ ∆ . If L is C k , C ∞ or real analytic, the same is true for π. Under Hy-pothesis 3.2, π is defined on all ∆ and maps ∆ into ˙∆ . We let F π denote the map defined as F π ( x ) = − x + π ( x ) . (9)Throughout, it is implicitly assumed that the domain of F π is ˙∆under Hypothesis 3.1 and ∆ under Hypothesis 3.2. By this we mean that L is the restriction to ∆ of a C map defined in aneighborhood of ∆ in af f (∆) = { x ∈ R n : P i x = 1 }
8e now consider the dynamics induced by (1). Without loss ofgenerality, we may assume that (1) is defined on all R n and induces aflow Φ = (Φ t ) leaving ∆ positively invariant. Indeed, by convexity of∆ , the retraction r : R n ∆ defined by r ( x ) = argmin y ∈ ∆ k x − y k is Lipschitz so that the differential equation dydt = yL ( r ( y )) (10)is Lipchitz and sub-linear on all R n . By standard results, it theninduces a flow Φ : R × R n R n where t Φ( t, y ) = Φ t ( y ) is theunique solution to (10) with initial condition y. For all x ∈ ∆ and t ≥ , Φ t ( x ) ∈ ∆ and the map t ∈ R + Φ t ( x ) is solution to (1).In the following we sometime use the notation Φ t ( x ) = x ( t ) =( x ( t ) , . . . , x n ( t )) . The tangent space of ∆ is the space T ∆ = { u ∈ R n : n X i =1 u i = 0 } . Lemma 3.4 (i)
There exists α ≥ such that for all x ∈ ∆ x i ( t ) ≥ e − αt x i (0) . In particular, ˙∆ is positively invariant. (ii) If for all x ∈ ∂ ∆ L ( x ) is irreducible, then Φ t (∆) ⊂ ˙∆ for all t > and the dynamics (1) admits a global attractor A = \ t ≥ Φ t (∆) ⊂ ˙∆ . Proof. ( i ) Let α = sup x ∈ ∆ − L ii ( x ) . For all j = i and x ∈ ∆˙ x i ≥ − αx i + x j L ji ( x ) . Hence x i ( t ) ≥ e − αt [ x i (0) + Z t e αs x j ( s ) L ji ( x ( s )) ds ] ≥ e − αt x i (0) . The second inequality is the first statement. From the first inequalityand the continuity of L ( x ( t )) it follows that x i ( t ) > t > x j L ji ( x ) > . Let now x ∈ ∂ ∆ . Assume without loss9f generality that x > . By irreducibility there exists a sequence1 = i , i , . . . i k = j such that L i l ,i l +1 ( x ) > . Hence, by continuity, L i l ,i l +1 ( x ( t )) > t small enough. It then follows that x j ( t ) > t > (cid:4) Remark 3.5
Assumption 3.1 is not needed in Lemma 3.4.Throughout we let Eq ( F ) = { x ∈ ∆ : F ( x ) = 0 } denote the equilibria set of F. Note that in view of the precedingLemmas Eq ( F ) ∩ ˙∆ = { x ∈ ˙∆ : F π ( x ) = 0 } and, in case L ( x ) is irreducible for all x ∈ ∆ , Eq ( F ) ⊂ ˙∆ . An equilibrium p is called non degenerate for F provided theJacobian matrix DF ( p ) : T ∆ T ∆ is invertible. Lemma 3.6
Let p ∈ Eq ( F ) ∩ ˙∆ . Then p is non degenerate for F ifand only if it is non degenerate for F π . Proof.
Let L T ( x ) : T ∆ T ∆ be defined by L T ( x ) h = hL ( x ) . Thenfor all x ∈ ˙∆ F ( x ) = xL ( x ) = ( x − π ( x )) L ( x ) = L T ( x )( x − π ( x )) . Hence at every equilibrium p ∈ ˙∆ DF ( p ) = − L T ( p )( DF π ( p )) . Byirreducibility, L T ( p ) is invertible (see Lemma 9.1 in the appendix).Thus DF ( p ) is invertible if and only if DF π ( p ) is invertible. (cid:4) F and F π are generallyunrelated While F and F π have the same equilibria, they may have quite dif-ferent dynamics as shown by the following example.Suppose n = 3 so that ∆ is the unit simplex in R . Let G be asmooth vector field on ∆ such that: (i) G points inward ˙∆ on ∂ ∆ , ii) Every forward trajectory of G converge to p = (1 / , / , / , (iii) DG ( p ) − = (cid:18) − η − − η (cid:19) , η > , where : T ∆ R is defined by ( u , u , u ) = ( u , u ) . It is easy to construct such a vector field.Choose ε > εG ( x ) + x lies in ˙∆ for all x ∈ ∆ and set π ( x ) = εG ( x ) + x. Then, F π and G have the sameorbits.Let W be a 3 × L ij ( x ) = W ij π j ( x ) for i = j and L ii ( x ) = − P j = i L ij ( x ) . The matrix L ( x ) is an irreducible rate matrix, re-versible with respect to π ( x ) . It follows from Lemmas 3.4 and 3.6that F ( x ) = xL ( x ) has a global attractor contained in ˙∆ and aunique equilibrium given by p. Furthermore, DF ( p ) = − L ( p ) T DF π ( p ) = − εL ( p ) T DG ( p ) = − ε W DG ( p )where the last equality follows from the definition of L and the factthat π ( p ) = p. To shorten notation, set b = ε W , c = ε W and d = ε W . Then DF ( p ) − = (cid:18) ( b + 2 c ) c − bd − b ( b + 2 d ) (cid:19) (cid:18) − η − − η (cid:19) The determinant of this matrix is positive and its trace equals( c − d ) − η ( b + c + d ) . If one now choose c > d and η small enough, the trace is positive.This makes p linearly unstable. By Poincar´e-Bendixson theorem,it follows that every forward trajectory distinct from p convergestoward a periodic orbit. Remark 4.1
It was pointed out to me by Sylvain Sorin and JosefHofbauer that this example is reminiscent of the following phe-nomenon. Consider a population game which revision protocol takesthe form K ij ( x ) = x j R max(0 , U j ( x ) − U i ( x )) for i = j R is chosen so that P j = i K ij ( x ) ≤ x i = x i ( U i ( x ) − X j ∈ S x j U j ( x )) (11)Here the rate matrix L ( x ) is not irreducible and its set of invariantprobabilities is easily seen to be the Best Reply set BR ( x ) = conv ( { e i : U i ( x ) = max j ∈ S U j ( x ) } ) . where conv ( A ) stands for the convex hull of A. The vector field (9)is not defined but can be replaced by the differential inclusion˙ x ∈ − x + BR ( x ) . (12)If one assume that U i ( x ) = P j U ij x j with U the payoff matrix givenby a Rock-Paper-Scissors game, U = − − − ;Then p = (1 / , / , /
3) is the unique equilibrium of (11) in ˙∆(corresponding to the unique Nash equilibrium of the game) andevery solution to (11) with initial condition in ˙∆ \ { p } is a periodicorbit. On the other hands, solutions to (12) converge to p. Phaseportraits of these dynamics can be found in ([24], section 5) and adetailed comparison of the replicator and the best reply dynamics isprovided in [15].
For u, v ∈ R n we let h u, v i = P i u i v i . A map h : ˙∆ R n , is called a gradient if there exists a C map V : ˙∆ R such that for all x ∈ ˙∆ and u ∈ T ∆ h h ( x ) , u i = DV ( x ) .u := h∇ V ( x ) , u i .
12t is called a quasigradient or a α - quasigradient if x α ( x ) h ( x )is a gradient for some continuous map α : ˙∆ R ∗ + . That is α ( x ) h h ( x ) , u i = h∇ V ( x ) , u i (13)for all x ∈ ˙∆ and u ∈ T ∆ . Remark 5.1 If V is the restriction to ˙∆ of a C map W : R n R , then ∇ V ( x ) is the orthogonal projection of ∇ W ( x ) onto T ∆ . Thatis ∇ V i ( x ) = ∂W∂x i ( x ) − n n X j =1 ∂W∂x j ( x ) , i = 1 , . . . n. Remark 5.2
A practical condition ensuring that h is a gradient isthat (a) h is the restriction to ˙∆ of a C map h : R n R n , (b) For all x ∈ ˙∆ and i, j, k ∈ { , . . . , n } ∂h i ∂x j ( x ) + ∂h j ∂x k ( x ) + ∂h k ∂x i ( x ) = ∂h i ∂x k ( x ) + ∂h k ∂x j ( x ) + ∂h j ∂x i ( x ) . This follows from ([14], Theorem 19.5.5.)
Notation
We use the following convenient notation. If x, y arevectors in R n and s : R R we let x.y ∈ R n (respectively xy and s ( x )) be the vector defined by ( xy ) i = x i y i (respectively ( xy ) i = x i y i , s i ( x ) = s ( x i )) . A C map V : ˙∆ R is called a strict Lyapounov function for F (or Φ) if for all x ∈ ˙∆ F ( x ) = 0 ⇒ h F ( x ) , ∇ V ( x ) i < . Theorem 5.3
Let s :]0 , ∞ [ R be a C function with positivederivative and let h s : ˙∆ R n be the map defined by h s ( x ) = s ( xπ ( x ) ) . Assume that h s is a α -quasigradient. Then i) The map V (given by (13)) is a strict Lyapounov function for F on ˙∆; (ii) The critical points of V coincide with Eq ( F ) ∩ ˙∆; (iii) V satisfies the following angle condition : For every compactset K ⊂ ˙∆ there exists c > such that | h∇ V ( x ) , F ( x ) i |≥ c k ∇ V ( x ) kk F ( x ) k for all x ∈ K. Remark 5.4 [Gibbs systems] If π ( x ) is a Gibbs measure , π β,i ( x ) = e − ( U i + β P j U ij x j ) Z ( x ) (14)where U = ( U ij ) is a symmetric matrix, β ≥ , and Z ( x ) = X j e − ( U j + β P k U ik x k ) , parts ( i ) and ( ii ) of Theorem 5.3 have been proved in [11], Theorems5.3 and 5.5. Here s ( t ) = log( t ) and V ( x ) = X i x i log( x i ) + X j U j x j + β X ij U ij x i x j . (15) Proof of theorem 5.3
Part ( i ) relies on the following Lemma. Lemma 5.5
Let L be an irreducible transition matrix with invariantprobability π. Let x ∈ ∆ , f i = x i π i , s ( f ) i = s ( f i ) and c f = inf i s ′ ( f i ) > . Then there exists λ ( L ) > depending continuously on L such that h xL, s ( f ) i ≤ − c f λ ( L ) V ar π ( f ) where V ar π ( f ) = P i ( f i − π i = P i ( x i − π i ) π i . L = L ( x ) and π = π ( x ) gives h F ( x ) , ∇ V ( x ) i < x = π ( x ) . ( ii ) The set Eq ( F ) ∩ ˙∆ coincides with fixed points of π in ˙∆ . Let x ∈ ˙∆ . ∇ V ( x ) = 0 ⇔ h s ( x ) ∈ R . The function s being injective this isequivalent to x i π i ( x ) = x j π j ( x ) for all i, j. That is x = π ( x ) . ( iii ) Let K ⊂ ˙∆ . By Lemma 5.5 (applied with L = L ( x ) and π = π ( x )) and continuity of the maps involved, there exists c > K such that |h∇ V ( x ) , F ( x ) i| ≥ c X i ( x i − π i ( x )) = c k x − π ( x ) k . To prove the angle condition it then suffices to show that both k F ( x ) k and k∇ V ( x ) k are bounded by some constant times k x − π ( x ) k . Now, F ( x ) = xL ( x ) = xL ( x ) − π ( x ) L ( x ) so that k F ( x ) k ≤ c k x − π ( x ) k with c = sup x ∈ ∆ k L ( x ) k . By Lipschitz continuity of s and compactness, there exist c , c > K such that | s ( x i π i ( x ) ) − s (1) | ≤ c | x i π i ( x ) − | ≤ c | x i − π i ( x ) | . Thus, for all u ∈ T ∆ such that k u k = 1 h h s ( x ) , u i = h h s ( x ) − s (1)1 , u i ≤ k h s ( x ) − s (1) . k ≤ c k x − π ( x ) k . This implies that k∇ V ( x ) k ≤ c k x − π ( x ) k and concludes the proof. (cid:4) The following result proves to be useful for certain dynamicsleaving invariant the boundary of the simplex. Such dynamics occur15n population games using imitative protocols (see equation 5) aswell as in certain models of vertex reinforcement (see example 5.11below).For x ∈ ∆ let Supp ( x ) = { x ∈ ∆ : x i > } . Proposition 5.6
Assume that assumptions of Theorem 5.3 hold.Assume furthermore that (a)
For all x ∈ ∆ x i = 0 ⇒ L ji ( x ) = 0 and the reduced rate matrix [ L ij ( x )] i,j ∈ Supp ( x ) is irreducible (b) The maps V : ˙∆ R n and α : ˙∆ R ∗ + (given by equation 13)extend to C (respectively continuous) maps V : ∆ R n and α : ∆ R ∗ + . Then V is strict Lyapounov function for F on ∆ . Proof.
Let T ∆( x ) = { u ∈ T ∆ : u i = 0 for i Supp ( x ) } . Byassumption ( a ) the map x π ( x ) is defined for all x ∈ ∆ continuousin x and π i ( x ) = 0 ⇔ x i = 0 . Therefore, using assumption ( b ), the equation ∀ x ∈ ˙∆ , ∀ u ∈ T ∆ α ( x ) h h s ( x ) , u i = h∇ V ( x ) , u i extends to ∀ x ∈ ∆ , ∀ u ∈ T ∆( x ) X i ∈ Supp ( x ) s ( x i π i ( x ) ) u i = h∇ V ( x ) , u i Thus X i ∈ Supp ( x ) s ( x i π i ( x ) )( xL ( x )) i = h∇ V ( x ) , F ( x ) i for all x ∈ ∆ . By Lemma 5.5 the left hand side is nonpositive andzero if and only if x i = π i ( x ) for all i ∈ Supp ( x ) . (cid:4) Remark 5.7
Note that under the assumptions of Proposition 5.6,the angle inequality of Theorem 5.3 doesn’t hold on the boundaryof the simplex 16 xample 5.8
Let W : R n R be a C k map, k ≥
1. Suppose thatfor all x ∈ ˙∆ π i ( x ) = f i ( x i ) ψ ( ∂W∂x i ( x )) P nj =1 f j ( x j ) ψ ( ∂W∂x j ( x ))Then, Theorem 5.3 applies in the following cases: Case 1 ψ ( u ) = e − βu with β ≥ , and f i ( t ) > t > . Itsuffices to choose s ( t ) = log( t ) and V ( x ) = n X i =1 x i log( x i ) − n X i =1 Z x i log( f i ( u )) du + βW ( x ) (16)Then h s is the gradient of V. Case 2 ψ ( u ) = u β , β > , f i ( t ) = t and ∂W∂x i > { x ∈ ∆ : x i > } . It suffices to choose s ( t ) = − t − /β and V ( x ) = − W ( x ) . Then h s is the α -quasigradient of V with α ( x ) = [ X j x j ( ∂W∂x j ) β )] − /β . Example 5.9 [Potential Games]
Examples of applications of Example 5.8, case 1, are given by
Potential Games (see [23] for an comprehensive presentation andmotivating examples). We use the notation of section 2. A
PotentialGame is a game for which the payoff function is such that for all x ∈ ∆ U i ( x ) = − ∂W∂x i ( x ) , i = 1 . . . n Consider a population game with a revision protocol given by (7).Suppose that the attachment matrix takes the form w ij ( x ) = f j ( x j ) ˜ w ij ( x )17ith ˜ w irreducible and symmetric. Let β ≥ . Assume furthermorethat g ( u, v ) takes one of the following form: Pairwise comparison g ( u, v ) = e β ( v − u ) e β ( v − u ) or g ( u, v ) = min(1 , e β ( v − u ) ) , Imitation driven by dissatisfaction g ( u, v ) = e − βu , Imitation of success g ( u, v ) = e βv . In all these situations, K ( x ), hence L ( x ) is reversible with respectto π β ( x ) with π β,i ( x ) = f i ( x i ) e − β ∂W∂xi ( x ) P j f j ( x j ) e − β ∂W∂xj ( x ) . Theorem 5.3 applies with V given by (16). Remark 5.10 [Gibbs systems, 2]
A particular case of potentialgames is obtained with W ( x ) = P ij U ij x i x j with U = ( U ij ) sym-metric, and f i ( x ) = e − U i . Here payoffs are linear in x : U i ( x ) = − X j U ij x j and we retrieve the situation considered in [11]. See Remark 5.4. Example 5.11 [Vertex reinforcement]
Let K be the revisionprotocol defined by K ij ( x ) = A ij x γj P k A ik x γk where A is a matrix with positive entries and γ ≥ . For populationgames (see section 2.1) this gives a simple model of imitation: anagent of type i, when chosen, switches to j with a probability pro-portional to the ( number of agents of type j ) γ . For processes withreinforcement (as defined in section 2.2) the probability to jump from i to j at time n is proportional to ( the time spent in j up to time n) γ . vertex reinforced random walks was intro-duced by Diaconis and first analyzed in Pemantle [20] (see also [1]and [7] for more references on the subject).When A is symmetric, K ( x ) is reversible with respect to π i ( x ) = x γi P k A ik x γk P ij A ij x γi x γj = x i ∂W∂x i P j x j ∂W∂x j (17)with W ( x ) = X i,j A ij x γi x γj (18)We are then in the situation covered by Example 5.8, case 2, with ψ ( u ) = u, f i ( t ) = t, s ( t ) = − t and V = − W. Both Theorem 5.3 and Proposition 5.6 apply.
Example 5.12 [Interacting Urn processes]
Closely related tovertex reinforced random walks are models of interacting urns (see[3], [9], [25]). For these models π i ( x ) = x i ∂W∂x i for some smoothfunction W. This is a particular case of Example 5.8, case 2.
Using Lasalle’s invariance principle we deduce the following conse-quences from Theorem 5.3.
Corollary 5.13
Assume that assumptions of Theorem 5.3 hold. Thenevery omega limit set of Φ contained in ˙∆ is a connected subset of Eq ( F ) ∩ ˙∆ . Combining this results with Lemma 3.4 (ii) and Proposition 5.6 gives
Corollary 5.14
Assume that one of the following condition hold: (a)
Assumptions of Theorem 5.3 and Hypothesis 3.2 or; (b)
Assumptions of Proposition 5.6.Then every omega limit set of Φ is a connected subset of Eq ( F ) .
19 set L is called attractor free or internally chain transitive providedit is compact, invariant and Φ | L has no proper attractor. For rein-forced random processes like the ones defined in section 2.2, limitsets of ( µ n ) are, under suitable assumptions, attractor free sets ofthe associated mean field equation (8) (see [1]). More generally at-tractor free sets are limit sets of asymptotic pseudo trajectories (see[4]). It is then useful to characterize such sets. Note however thatthe existence of a strict Lyapounov function, doesn’t ensure in gen-eral, that internally chain transitive sets consist of equilibria (seee.g Remark 6.3 in [2]). Corollary 5.15
Assume that assumptions of Theorem 5.3 hold andthat h s is C k for some k ≥ n − T ∆) − . Then every inter-nally chain transitive set of Φ contained in ˙∆ is a connected subset of Eq ( F ) ∩ ˙∆ . If we furthermore assume that L ( x ) is irreducible for all x ∈ ∆ , then every internally chain transitive set of Φ is a connectedsubset of Eq ( F ) Proof.
Let C = Eq ( F ) ∩ ˙∆ and A ⊂ ˙∆ an attractor free set. ByTheorem 5.3, C coincide with critical points of V. By the assump-tion V is C k +1 so that by Sard’s theorem (see [13]), V ( C ) has emptyinterior. It follows (see e.g Proposition 6.4 in [2]) that A ⊂ C. (cid:4) In case equilibria are isolated, Corollary 5.13 implies that every tra-jectory bounded away from the boundary converge to an equilib-rium and that every trajectory converges in case L ( x ) is irreduciblefor all x. However, when equilibria are degenerate, the gradient-likeproperty is not sufficient to ensures convergence. There are knownexamples of smooth gradient systems which omega limit sets are acontinuum of equilibria (see [19]). However, in the real analytic case,gradient like systems which verify an angle condition are known toconverge
Theorem 5.16
Suppose that assumptions of Theorem 5.3 hold andthat V is real analytic. Then every omega limit set meeting ˙∆ reducesto a single point. roof. Let p be an omega limit point. If V is real analytic, itsatisfies a Lojasiewicz inequality at p in the sense that there exist0 < η ≤ / , β > U ( p ) of p such that | V ( x ) − V ( p ) | − η ≤ β k∇ V ( x ) k for all x in a U ( p ). Such an inequality called a ”gradient inequality”was proved by Lojasiewicz [17] and used (by Lojasiewicz again) toshow that bounded solutions of real analytic gradient vector fieldshave finite length, hence converge. When the dynamics is not a gra-dient, but only gradient like with V as a strict Lyapounov function,the same results holds provided that V satisfies an angle condition: h∇ V ( x ) , F ( x ) i ≥ c k F ( x ) kk∇ V ( x ) k for all x ∈ U ( p ) . This is proved in [10] (see also [18], Theorem 7). (cid:4)
Example 5.17 [Gibbs systems, 3] If π is given by (14) with U symmetric, V given by (15) is real analytic so that every solution to(1) converges toward an equilibrium. Recall that point p ∈ Eq ( F ) is called non degenerate if the jacobianmatrix DF ( p ) : T ∆ T ∆ is invertible. It is called hyperbolic ifeigenvalues of DF ( p ) have non zero real parts. If p is hyperbolic, T ∆ admits a splitting T ∆ = E up ⊕ E sp invariant under DF ( p ) such that the eigenvalues of DF ( p ) | E sp (re-spectively DF ( p ) | E up ) have negative (respectively positive) real parts.Point p ∈ Crit ( V ) = { x ∈ ˙∆ : ∇ ( V )( x ) = 0 } is called non-degenerate if Hess ( V )( p ) the Hessian or V at p has full rank. In asuitable coordinate systems Hess ( V )( p )( u, u ) = P n + i =1 u i − P n − j =1 u j with n + + n − = dim ( T ∆) = n − . The number n − is called the index of p (with respect to V ) and is written Ind ( p, V ) . roposition 6.1 Assume that assumptions of Theorem 5.3 hold.Let p ∈ Eq ( F ) ∩ ˙∆ . Then (i)
Point p is non degenerate if and only if it is a non degeneratecritical point of V. (ii) If furthermore L is C and p is hyperbolic, dim( E up ) = Ind ( p, V ) . Proof.
From Lemma 3.6, p is non degenerate if and only if DF π ( p )is invertible and (see the proof of Lemma 3.6) DF ( p ) = − L T ( p ) DF π ( p ) (19)Now, a direction computation (details are left to the reader) of theHessian of V at x leads to h Hess ( V )( x ) u, v i = α ( x ) h s ′ ( xπ ( x ) )( u − xπ ( x ) Dπ ( x ) u, v i /π ( x ) where h u, v i /π stands for P i u i v i π i . Since p = π ( p ) h Hess ( V )( p ) u, v i = α ( p ) s ′ (1) h ( I − Dπ ( p )) u, v i /p (20)for all u, v ∈ T ∆ , This proves that
HessV ( p ) is non degenerate ifand only if ( I − Dπ ( p )) = − DF π ( p ) is non degenerate and concludesthe proof of the first part.We now prove the second part. By the stable manifold theorem,there exists a (local) C manifold W sp tangent to E sp at p positivelyinvariant under Φ and such that for all x ∈ W sp lim t →∞ Φ t ( x ) = p. Clearly p is a global minimum of V restricted to W s . For otherwisethere would exists x ∈ W sp such that V ( p ) > V ( x ) > lim t →∞ V (Φ t ( x )) = V ( p ) . Since p is also a critical point ∇ V ( p ) = 0 . Let u ∈ E sp and let γ :] − , W sp be a C path with γ (0) = p, ˙ γ (0) = u. Set h ( t ) = V ( γ ( t )) . Then ˙ h (0) = 0 (because p is a critical point of V ) and h ′′ (0) = h HessV ( p ) u, u i is non negative because h ( t ) ≥ h (0) .
22n the other hand, by the spectral decomposition of
HessV ( p )we can write T ∆ = E sV ⊕ E uV with h HessV ( p ) u, u i > <
0) for all u ∈ E sV \ { } (respectively E uV \ { } ). Thus, E sp ∩ E uV = { } and, consequently, dim( E sp ) + dim( E uV ) ≤ dim( T ∆) . Sim-ilarly dim( E up ) + dim( E sV ) ≤ dim( T ∆) . This proves that dim( E up ) =dim( E uV ) = Ind ( p, V ) . (cid:4) Remark 6.2
This later proposition shows that in the neighborhoodof an hyperbolic equilibrium p , ˙ x = F ( x ) and ˙ x = −∇ V ( x ) are topo-logically conjugate. Indeed, part ( ii ) of the proposition implies thatthe linear flows { e tDF ( p ) } and { e tHess ( V )( p ) } are topologically con-jugate (see e.g Theorem 7.1 in [22]), and by Hartman-GrobmanTheorem (see again [22]), nonlinear flows are locally conjugate totheir linear parts in the neighborhood of hyperbolic equilibria. How-ever, note that while eigenvalues of Hess ( V )( p ) are reals there isno evidence that the same is true for DF ( p ) in general. The nextproposition proves that this is the case when L ( x ) is reversible withrespect to π ( x ) . Proposition 6.3
Let p ∈ Eq ( F ) ∩ ˙∆ . Assume that assumptions ofTheorem 5.3 hold and that L ( p ) is reversible with respect to π ( p ) = p. Then there exists a positive definite bilinear form g ( p ) on T ∆ suchthat for all u, v ∈ T ∆ g ( p )( DF ( p ) u, v ) = −h Hess ( V )( p ) u, v i In particular (i) DF ( p ) has real eigenvalues, (ii) p is hyperbolic for F if and only if it is a non degenerate criticalpoint of V. Proof.
Let p ∈ Eq ( F ) ∩ ˙∆ . Set L = L ( p ) and recall that L T : T ∆ T ∆ is defined by L T h = hL. Then, by Lemma 9.1 in theappendix, − L T is a definite positive operator for the scalar producton T ∆ defined by h u, v i /p = P i u i v i p i . Define now g ( p ) by g ( p )( u, v ) = −h ( L T ) − u, v i p . (21)23sing (19) and (20) it comes that for all u, v ∈ T ∆ g ( p )( DF ( p ) u, v ) = −h ( I − Dπ ( p )) u, v i /p = − [ α ( p ) s ′ (1)] − h Hess ( V )( p ) u, v i . Replacing g ( p ) by α ( p ) s ′ (1) g ( p ) proves the result. (cid:4) A useful consequence of this later proposition is that it is usu-ally much easier to verify non degeneracy of equilibria rather thanhyperbolicity. Here is an illustration:
Example 6.4 [Gibbs systems, 4]
Consider the symmetric Gibbsmodel analyzed in [11] (see remark 5.4 and example 5.10). We sup-pose that the symmetric matrix U = ( U ij ) is given and we treat U = ( U i ) i ∈ S and β as parameters. Let Ξ rev ( U ) denote the set ofmaps R + × ∆ T ∆ , ( β, x ) F β ( x ) = xL β ( x )such that L β verifies assumption 3.2, is C in x, and L β ( x ) is re-versible with respect to π β ( x ) where π β,i ( x ) = e − U i − β P j U ij x j Z ( x ) . Proposition 6.5
There exists an open and dense set G ⊂ R n suchthat for all U ∈ G and F ∈ Ξ rev ( U ) (i) The set { ( x, β ) ∈ ∆ × R +: : F β ( x ) = 0 } is a C ∞ one dimensionalsubmanifold, (ii) There exists an open dense set B ⊂ R + containing such thatfor all β ∈ B equilibria of F β are hyperbolic. Proof.
Let H : ˙∆ × R n × R + T ∆ be defined by H ( x, U , β ) = ∇ V U ,β ( x ) where V U ,β is given by (15). Since ∂H∂U ( x, U , β ) is theidentity map, H is a submersion. Hence, by Thom’s parametrizedtransversality Theorem (see [13], Chapter 3), there exists an openand dense set G ∈ R n such that for all U ∈ G , ( x, β ) H ( x, U , β )24s a submersion. This proves ( i ) . By the same theorem, for all β ∈ B with B open and dense in R + , x H ( x, U , β ) is a submersion,meaning that critical points of V U ,β are nondegenerate. By Propo-sition 6.3, equilibria of F β are hyperbolic. (cid:4) Remark 6.6
Other genericity results can be proved, if one fix U or β and treat U as a parameter. Compare to the proof of Theorem2.10 in [6] in an infinite dimensional setting. Consider a population game with C payoff function U : ∆ R n . Recall that the game is called a potential game, provided U i ( x ) = − ∂W∂x i ( x ) for all x ∈ ∆ and some potential W : R n R . Point x ∗ ∈ ∆ is called a Nash equilibrium of U if, given the pop-ulation state x ∗ , every agent has interest to play the mixed strategy x ∗ . That is ∀ i ∈ { , . . . , n } U i ( x ∗ ) ≤ h U ( x ∗ ) , x ∗ i (22)Let Supp ( x ∗ ) = { i ∈ { , . . . , n } : x ∗ i > } . It follows from (22) that ∀ i ∈ Supp ( x ∗ ) U i ( x ∗ ) = h U ( x ∗ ) , x ∗ i . We let NE ( U ) denote the set of Nash equilibria of U. For all β ≥ x ∈ ∆ we let π β ( x ) ∈ ∆ be defined as π β,i ( x ) = e βU i ( x ) P j e βU j ( x ) , i = 1 , . . . , n (23)and we let χ ( β, U ) (respectively, χ rev ( β, U )) denote the set of allvector fields having the form given by (1) where L ( x ) is C in x ,irreducible and admits π β ( x ) as invariant (respectively reversible)probability. Recall (see equation (9)) that F π β = − Id + π β . Eq ( F ) for F ∈ χ ( β, U ) in term of NE ( U )for large β, with a particular emphasis on potential games. Some ofthe results here are similar to the results obtained in [5] for n × Proposition 6.7
Let N be a neighborhood of NE ( U ) . There exists β ≥ such that for all β ≥ β and F ∈ χ ( β, U ) Eq ( F ) ⊂ N Proof.
Equilibria of F coincide with equilibria of F π β . Let x ( β ) besuch an equilibrium. Then for all i, j log( x i ( β )) − log( x j ( β )) β = U i ( x ( β )) − U j ( x ( β )) . Thus for every limit point x ∗ = lim β k →∞ x ( β k ) it follows that U i ( x ∗ ) = U j ( x ∗ )if i, j ∈ Supp ( x ∗ ) and U i ( x ∗ ) ≥ U j ( x ∗ )if i Supp ( x ∗ ) and j ∈ Supp ( x ∗ ) . (cid:4) Remark 6.8
Note that Proposition 6.7 only requires the continuityof U. We shall now prove some converse results.A Nash equilibrium x ∗ is called pure if Supp ( x ∗ ) has cardinal 1and mixed otherwise. It is called strict if inequality (22) is strict forall i Supp ( x ∗ ) . Theorem 6.9
Let x ∗ be a pure strict Nash equilibrium and N a(sufficiently small) neighborhood of x ∗ . Then, there exists β > such that for all β ≥ β and F ∈ χ ( β, U ) (i) Eq ( F ) ∩ N = { x ∗ β } (ii) Equilibrium x ∗ β is linearly stable for F π β . iii) Assume furthermore that the game is a potential game. Then x ∗ β is linearly stable for F under one of the following conditions: (a) L (hence F ) is C and x ∗ β is hyperbolic for F, or (b) F ∈ χ rev ( β, U ) . Proof.
Suppose without loss of generality that x ∗ = 1 and x ∗ i = 0for i = 1 . Set R ij = U j − U i . By assumption and continuity, there exists δ > , α > x ∈ B ( x ∗ , α ) = { x ∈ ∆ : k x − x ∗ k ≤ α } ,R i ( x ) ≥ δ for i > k R ij ( x ) k ≥ δ if R ij ( x ∗ ) = 0and k R ij ( x ) k ≤ δ if R ij ( x ∗ ) = 0 . Thus1 ≥ π β, ( x ) = (1 + X i> e − βR i ( x ) ) − ≥ (1 + ( n − e − βδ ) − . This implies that π β maps B ( x ∗ , α ) into itself for β large enough.By Brouwer’s Theorem, it then admits a fixed point x ∗ β . To proveuniqueness and assertion ( ii ) it suffices to prove that π β restricted to B ( x ∗ , α ) is a contraction. From the expression π β,i = ( P j e βR ij ) − , we get ∂π β,i ∂x m = − X j [ βe βR ij ( X k e βR ik ) − ∂R ij ∂x m ] := X j D ij = X j = i D ij . Let j = i. If R ij ( x ∗ ) = 0 | D ij | ≤ βe βR ij (1 + e βR ij ) − ≤ β min( e βR ij , e − βR ij ) ≤ βe − βδ . If R ij ( x ∗ ) = 0 . Then i = 1 and | D ij | ≤ βe βR ij ( e βR i ) − = βe β ( R ij − R i ) ≤ βe − βδ These inequalities show that k Dπ β ( x ) k < x ∈ B ( x ∗ , α ) and β large enough, proving uniqueness of the equilibrium as well as as-sertion ( ii ) . The last assertion follows from Propositions 6.1 and 6.3.27
A Nash equilibrium x ∗ is called fully mixed if Supp ( x ∗ ) = { , . . . , n } and partially mixed if 1 < card ( Supp ( x ∗ )) < n. A fully mixed Nash equilibrium is called non degenerate if for all u ∈ T ∆ [ ∀ w ∈ T ∆ h DU ( x ∗ ) u, w i = 0] ⇒ u = 0 . Let T ∆( x ∗ ) = { u ∈ T ∆ : u i = 0 for i Supp ( x ∗ ) } . A partially mixed equilibrium x ∗ is called non degenerate if for all u ∈ T ∆( x ∗ ) [ ∀ w ∈ T ∆( x ∗ ) h DU ( x ∗ ) u, w i = 0] ⇒ u = 0 , Lemma 6.10
Let x ∗ ∈ ∆ be a mixed equilibria. Assume that Supp ( x ∗ ) = { , . . . , r } for some < r ≤ n and set x ∗ = ( q , . . . , q r − , − r − X i =1 q i , , . . . , . Let, for i = 1 , . . . , r − ,h ri ( x , . . . , x r − , y , . . . , y n − r ) = U i ( x , . . . , x r − , − r − X i =1 x i − n − r X i =1 y i , y , . . . , y n − r ) − U r ( x , . . . , x r − , − r − X i =1 x i − n − r X i =1 y i , y , . . . , y n − r ) Then x ∗ is non degenerate if and only if the ( r − × ( r − matrix (cid:20) ∂h ri ∂x j (( q, (cid:21) i,j =1 ,...r − is invertible. roof. One has ∂h ri ∂x j ( q,
0) = ( ∂U i ∂x j ( x ∗ ) − ∂U i ∂x r ( x ∗ )) − ( ∂U r ∂x j ( x ∗ ) − ∂U r ∂x r ( x ∗ )) . Let v = ( v , . . . , v r − , − r − X i =1 v i , , . . . , ∈ T ∆( x ∗ )and w = ( w , . . . , w r − , − r − X i =1 w i , , . . . , ∈ T ∆( x ∗ ) . Then it is easily seen that r − X i =1 r − X j =1 ∂h ri ∂x j ( q ) v i w j = h DU ( x ∗ ) v, w i . This proves that x ∗ is non degenerate if and only if h ∂h ri ∂x j (( q, i i,j =1 ,...r − is invertible. (cid:4) Theorem 6.11
Let x ∗ be a non degenerate fully mixed Nash equi-librium for U and N a (sufficiently small) neighborhood of x ∗ . Then,there exists β > such that for all β ≥ β and F ∈ χ ( β, U ) Eq ( F ) ∩ N = { x ∗ β } . Assume furthermore that the game is a potential game with potential W. Then x ∗ β is hyperbolic for F and its unstable manifold (for F ) hasdimension Ind ( x ∗ , W | ∆ ) under one of the following conditions : (a) L (hence F ) is C and x ∗ β is hyperbolic for F, or (b) F ∈ χ rev ( β, U ) . Proof.
Set T = 1 /β. Equilibria of F π β are given by the set ofequations T (log( x i ) − log( x n )) = U i ( x ) − U n ( x ) , i = 1 , . . . , n − T (log( x i ) − log(1 − n − X i =1 x i )) = h ni ( x , . . . , x n − ) , i = 1 , . . . , n − . (24)Write x ∗ = ( q , . . . , q n − , − P n − i =1 q i ) . For T = 0 , q = ( q , . . . , q n − )is solution to (24). Hence, by the implicit function theorem (whichhypothesis is fulfilled by the non degeneracy of x ∗ and Lemma 6.10)there exists α > , a neighborhood O of q in ( R ∗ + ) n − and a C map T ∈ ] − α , α [ q ( T ) ∈ O such that ( T, q ( T )) is the uniquesolution to (24) in ] − α , α [ × O. This proves the first assertion ofthe theorem with β > /α and x ∗ β = ( q (1 /β ) , − P n − i =1 q i (1 /β ))In case, the game is a potential game with potential W, F isgradient-like with Lyapounov function V β given by (16). Since x ∗ is fully mixed, q i < ∞ so that k β HessV β ( x ∗ β ) − HessW ( x ∗ ) k → β → ∞ . In particular, for β large enough HessV β ( x ∗ β ) is nondegenerate, because x ∗ is non degenerate. The last assertion thenfollows from Propositions 6.1 and 6.3. (cid:4) Theorem 6.12
Let x ∗ be a strict and non degenerate partially mixedNash equilibrium for U which support has cardinal < r < n. Let N be a (sufficiently small) neighborhood of x ∗ . Then, there exists β > such that for all β ≥ β and F ∈ χ ( β, U ) Eq ( F ) ∩ N = { x ∗ β } . Assume furthermore that the game is a potential game with potential W and that one of the following conditions hold : (a) L (hence F ) is C and x ∗ β is hyperbolic for F, or (b) F ∈ χ rev ( β, U ) . Then x ∗ β is hyperbolic and k ≤ dim( E ux ∗ β ) ≤ min( n − r + k, r − . with k = Ind ( x ∗ , W | ∆( x ∗ ) ) and dim( E ux ∗ β ) stands for the dimensionof the unstable manifold (for F ). roof. Assume without loss of generality that
Supp ( x ∗ ) = { , . . . , r } and set x ∗ = ( q , . . . , q r − , − P r − i =1 q i , , . . . , . Write every ele-ment of ∆ as ( x , . . . , x r − , − P r − i =1 x i − P n − ri =1 y i , y , . . . y n − r ) andset x = ( x , . . . , x r − ) , y = ( y , . . . , y n − r ) . Thus, with β = 1 /T, equi-libria of F π β are given by the following system of equations: T (log( x i ) − log(1 − r − X i =1 x i − n − r X i =1 y i )) = h ri ( x, y ) , i = 1 , . . . r − T (log( y i ) − log(1 − r − X i =1 x i − n − r X i =1 y i )) = h ri + r ( x, y ) , i = 1 . . . n − r (26)where h ri is defined in Lemma 6.10. The triplet ( T = 0 , x = q, y = 0)is solution to (25). Thus by the non degeneracy hypothesis and theimplicit function theorem, there exists a smooth mapˆ x : O 7→ V , ( T, y ) ˆ x ( T, y )where O is a neighborhood of (0 ,
0) in R × R n − r and V a neighborhoodof q in R r − such that ( T, ˆ x ( T, y ) , y ) is solution to (25). Recall that0 < P r − i =1 q i < h ri + r ( q, < i = 1 , . . . , n − r (because x ∗ is strict). Thus, by choosing O small enough we can furthermoreensure that 0 < − r − X i =1 ˆ x i ( T, y ) − n − r X i =1 y i < h ri + r (ˆ x ( T, y ) , y ) ≤ − δ < , i = 1 . . . n − r (28)for all ( T, y ) ∈ O . Now replacing x by ˆ x ( T, y ) in (26) leads to y i = G i ( T, y ) , i = 1 . . . n − r where G i ( T, y ) = (1 − r − X i =1 ˆ x i ( T, y ) − n − r X i =1 y i ) exp ( 1 T h ri + r (ˆ x ( T, y ) , y )) . α small enough and T ≤ log(1 /α ) δ G ( T, · ) maps { y ∈ R n − r : 0 ≤ y i ≤ α } into itself. By Brouwer’sfixed point theorem, G ( T, · ) admits a fixed point ˆ y ( T ) . Furthermore, k D y G ( T, y ) k ≤ CT e − δ/T for some constant C making G ( T, · ) a con-traction. This implies that ˆ y ( T ) is unique. Finally define x ∗ β by x ∗ β,i = ˆ x i ( T, ˆ y ( T )) for 1 ≤ i < r and x ∗ β,i + r = ˆ y i ( T ) for 1 ≤ ≤ n − r. We now prove the last assertions. By assumption, T ∆( x ∗ ) admitsa decomposition T ∆( x ∗ ) = E + ⊕ E − with h Hess ( W )( x ∗ ) u, u i > <
0) for all u ∈ E + (respectively E − ) and u = 0 . Set T ∆ s ( x ∗ ) = { u ∈ T ∆ : u = . . . = u r = 0 } . Then T ∆ = E + ⊕ E − ⊕ T ∆ s ( x ∗ ) . Let now V β be the Lyapounov function given (16). Then for all u ∈ T ∆ Q β ( u ) := h β Hess ( V β )( x ∗ β ) u, u i = h HessW ( x ∗ β ) u, u i + 1 β X i x ∗ β,i u i . The construction of x ∗ β shows that β x ∗ β,i → i ≤ r and β x ∗ β,i → ∞ for i > r when β → ∞ . Thus, for β large enough, Q β is non degen-erate, definite positive on E + and T ∆ s ( x ∗ ) , and definite negative on E − . This implies that its index is bounded below by k = dim( E − ) andabove by min ( r − , n − r − k ) . This index equals the dimension ofthe stable manifold by Proposition 6.1. Under the reversibility as-sumption hyperbolicity follows from Proposition 6.3. (cid:4)
Recall that an irreducible rate matrix L is called reversible withrespect to π ∈ ˙∆ is π i L ij = π j L ji . In this case π is the (unique)invariant probability of L. Here we will consider gradient propertiesof (1) under the assumption that L ( x ) is reversible.A C k , k ≥ C k map g such that for each x ∈ ∆ g ( x ) : T ∆ × T ∆ R is a definite positive32ilinear form. Given a C map V : ˙∆ R we let grad g V denotethe gradient vector field of V with respect to g. That is g ( x )( grad g V ( x ) , u ) = h∇ V ( x ) , u i for all u ∈ T ∆ . Proposition 7.1
Assume that for all x ∈ ˙∆ L ( x ) is reversible withrespect to π ( x ) and assume that the map h : ˙∆ R n , defined by h ( x ) = xπ ( x ) is a α -quasigradient. Then there exists a metric g on ˙∆ such thatfor all x ∈ ˙∆ F ( x ) = − grad g V ( x ) . If L and α are C k then g is C k . Proof.
The proof is similar to the proof of Proposition 6.3. Let A ( x ) : T ∆ T ∆ be defined by A ( x ) h = − hL ( x ) . Then A ( x ) and L ( x ) are conjugate by the relation π ( x ) L ( x ) h = A ( x ) π ( x ) h and A ( x )is a definite positive operator for the scalar product on T ∆ definedby h u, v i /π ( x ) = P i u i v i π i ( x ) . Define now a Riemannian metric on T ∆ by g ( x )( u, v ) = h A ( x ) − u, v i π ( x ) . (29)Since F ( x ) = xL ( x ) = ( x − π ( x )) L ( x ) = A ( x )( − x + π ( x )) , we get g ( x )( F ( x ) , u ) = −h xπ ( x ) − , u i = −h xπ ( x ) , u i . If x xπ ( x ) is a quasi gradient, this makes F a gradient for the metric g ( x ) = α ( x ) g ( x ) . (cid:4) Example 7.2
Suppose that L ( x ) is reversible with respect to π, independent on x. Then x xπ is the gradient of the χ function V ( x ) = P i ( x i π i − π i . Hence F ( x ) = − grad g V ( x ) for some metric g . Under the weaker assumption that x s ( xπ ( x ) ) is a quasi-gradientfor some strictly increasing function s (see Theorem 5.3)) it is nolonger true that F is a gradient, but it can be approximated by agradient. The next Lemma is the key tool. Its proof is identical tothe proof of Proposition 6.3. 33 emma 7.3 Assume that assumptions of Theorem 5.3 hold and thatfor all x ∈ ˙∆ , L ( x ) is reversible with respect to π ( x ) . Then thereexists a metric g on ˙∆ such that for p ∈ Eq ( F ) ∩ ˙∆ and u, v ∈ T ∆ g ( p )( DF ( p ) u, v ) = −h Hess ( V )( p ) u, v i . If, furthermore, L and α (in equation 13) are C k then g is C k . Theorem 7.4
Assume that (a)
Assumptions of Theorem 5.3 hold with s, α and
L C k , k ≥ , (b) For all x ∈ ˙∆ L ( x ) is reversible with respect to π ( x ) , (c) Eq ( F ) ∩ ˙∆ is finite.Then for every neighborhood U of Eq ( F ) ∩ ˙∆ and every ε > thereexists a C k metric g on ˙∆ such that (i) − grad g V = F on ˙∆ \ U . (ii) k − grad g V − F k C , U ≤ ε where k G k C , U = sup x ∈U k G ( x ) k + k DG ( x ) k . Proof.
Let E = ˙∆ ∩ Eq ( F ) , v ( x ) = d ( x, E ) = min p ∈E k x − p k and let ψ : R + [0 ,
1] be a C ∞ function which is 0 on [0 , , , ∞ [ and such that 0 ≤ ψ ′ ≤ . Fix ε > λ ( x ) = ψ ( v ( x ) ε ) ,G = − grad g V where g is given by Lemma 7.3 and G ( x ) = (1 − λ ( x )) G ( x ) + λ ( x ) F ( x ) . Since for all p ∈ E , F ( p ) − G ( p ) = DF ( p ) − DG ( p ) = 0 there existsa constant C > k G ( x ) − F ( x ) k ≤ Cv ( x ) , k DG ( x ) − DF ( x ) k ≤ Cv ( x ) . Thus k G ( x ) − F ( x ) k = (1 − λ ( x )) k G ( x ) − F ( x ) k ≤ C (1 − λ ( x )) v ( x ) ≤ Cε k DG ( x ) − DF ( x ) k = k (1 − λ ( x ))( DG ( x ) − DF ( x ))+ h∇ λ ( x ) , G ( x ) − F ( x ) ik≤ C ((1 − λ ( x )) v ( x ) + 1 ε v ( x ) ) ≤ Cε.
This shows that G is a C approximation of F which coincides with F on { v ( x ) ≥ ε } and with G on { v ( x ) ≤ ε } . Furthermore, h∇ V ( x ) , G ( x ) i = − (1 − λ ( x )) g ( x )( G ( x ) , G ( x ))+ λ ( x ) h∇ V ( x ) , F ( x ) i ≤ x ∈ E . Now, for all x ∈ ˙∆ \ E T ∆ = ∇ V ( x ) ⊥ ⊕ R G ( x )and the splitting is smooth in x. Hence u ∈ T ∆ can be uniquelywritten as u = P x ( u ) + t x ( u ) G ( x ) with t x ( u ) ∈ R and P x ( u ) ∈∇ V ( x ) ⊥ . Let g be the metric on ˙∆ \ E defined by g ( x )( u, v ) = g ( P x ( u ) , P x ( v )) + t x ( u ) t x ( v ) g ( x )( G ( x ) , G ( x )) . Then g coincides with g on { < x < v ( x ) < ε } so that g canbe extended to a C metric on ˙∆ . By construction of G and g,G = − grad g V. (cid:4) Let C kpos (∆ , T ∆) denote the set of C k vector fields F : ∆ T ∆leaving ∆ positively invariant.Two elements F, G ∈ C kpos (∆ , T ∆) are said topologically equiva-lent if there exists a homeomorphism h : ∆ ∆ which takes orbits of F to orbits of G preserving their orientation. A set χ ⊂ C kpos (∆ , T ∆)is said structurally stable if all its elements are topologically equiva-lents.Let π : ∆ ˙∆ be a smooth function. Assume that π verifiesthe assumption of Theorem 5.3 and that F π has non degenerateequilibria. Let χ π, rev denote the convex set of vector fields having35he form given by (the right hand side of) (1), where for each x ∈ ∆ ,L ( x ) is irreducible and reversible with respect to π ( x ) . By Theorem5.3, Proposition 6.3 and Theorem 7.4 all the elements of χ π, rev havethe same strict Lyapounov function V, hyperbolic equilibria (givenby the critical points of V ) and are C close to − grad g V for somemetric g. We may then wonder wether χ π, rev is structurally stable.The following construction shows that this is not the case. Here ∆ stands for the 2-dimensional simplex in R . Let˜∆ = { ( y , y ) ∈ R : y , y ≥ , y + y ≤ } and : R R be the projection defined by ( x , x , x ) = ( x , x ) . Note that maps ∆ homeomorphically onto ˜∆ . Let ˜ W : R R be a smooth function. Assume that (a) −∇ ˜ W points inward ˜∆ on ∂ ˜∆; (b) The critical set crit ( ˜ W ) = { y ∈ ˜∆ : ∇ ˜ W ( y ) = 0 } consist of(finitely many) non degenerate points, (c) For all u ∈ R ∂ ˜ W∂y ( u, u ) = ∂ ˜ W∂y ( u, u ) . In particular, the diagonal D ( ˜∆) = { ( y , y ) ∈ ˜∆ : y = y } ispositively invariant under the dynamics˙ y = −∇ ˜ W ( y ) (30) (c) There is a saddle connection contained in D ( ˜∆) , meaning thatthere are two saddle points of ˜ W s , s ∈ D ( ˜∆) and some(hence every) point y ∈ ] s , s [ which α limit set under (30) is s and omega limit set is s . It is not hard to construct such a map.Let W : R R be defined by W = ˜ W ◦ . − strategies potential game associated to W. Payoffs are then defined by U i ( x ) = − ∂ ˜ W∂x i ( x , x ) , i = 1 , U ( x ) = 0 . Using the notation of section 6.1, record that F π β = − Id + π β where π β is defined by (23), and χ rev ( β, U ) is the set of vector fields givenby (1) with L ( x ) irreducible and reversible with respect to π β ( x ) . Proposition 8.1
For all β > sufficiently large, there exists F ∈ χ rev ( β, U ) (which can be chosen C close to F π β ) which is not equiv-alent to F π β . Proof of Proposition 8.1
By definition of Nash equilibria (see section 6.1) and condition ( a )above, Nash equilibria of U are fully mixed and coincide with criticalpoints of ˜ W : crit ( ˜ W ) = ( NE ( U )) . Lemma 8.2
For all ε > there exists β > such that for all β ≥ β and F ∈ χ rev ( β, U ) there is a one to one map p ∈ crit ( ˜ W ) p β ∈ Eq ( F ) , such that (i) k p − ( p β ) k ≤ ε, (ii) The unstable (respectively stable) manifold of p β has dimension Ind ( p, ˜ W , ) (resp. − Ind ( p, ˜ W ) ). In particular, s β and s β are saddle points. (iii) p ∈ D ( ˜∆) ⇔ ( p β ) ∈ D ( ˜∆) (iv) Under the dynamics induced by F π β , the interval [ s β , s β ] is in-variant and for some (hence all) q ∈ ] s β , s β [ the alpha limit(respectively omega limit) set of q equals s β (respectively s β ). roof. Assertions ( i ) and ( ii ) this follows from Propositions 6.7 and6.11.On − ( D ( ˜∆)) = { ( x , x , − x ) } equilibria of F π β are given bythe implicit equation T (log( x ) − log(1 − x )) = U ( x , x ) where T = 1 /β. Solutions for T = 0 coincide with − ( D ( ˜∆) ∩ crit ( ˜ W )) . For
T > iii ) then follows from theimplicit function theorem.By condition ( c ) , ∂ ˜ W∂x = ∂ ˜ W∂x on D ( ˜∆) . Thus U ( x ) = U ( x ) (hence F π β, ( x ) = F π β, ( x )) on − ( D ( ˜∆)) proving invariance of [ s β , s β ] ⊂ − ( D ( ˜∆)) . Assertion ( iv ) follows since, by ( iii ) , there are no equi-libria in ] s β , s β [ . (cid:4) We now construct F ∈ χ rev ( β, U ) . Let L ( x ) be the rate matrixdefined for i = j by L ij ( x ) = π β,j ( x ) if i, j
6∈ { , } L ( x ) = (1 + a ( x )) π β, ( x ) and L ( x ) = (1 + a ( x )) π β, ( x )where a : ∆ R + is a smooth function to be defined below. Thenequation (1) reads˙ x = ( x π β, ( x ) − x π β, ( x )) + ( x π β, ( x ) − x π β, ( x ))(1 + a ( x )) , ˙ x = ( x π β, ( x ) − x π β, ( x )) + ( x π β, ( x ) − x π β, ( x )) , ˙ x = − ˙ x − ˙ x . Thus, on x = x , ˙ x − ˙ x = [ x π β, ( x ) − x π β, ( x )] a ( x )= a ( x ) Z ( x ) ( x e βU ( x ) − x ) . The map x x e βU ( x ) − x vanishes at points s β , s β and has a con-stant sign over [ s β , s β ] (for otherwise there would exists an equilib-rium for F in ] s β , s β [ contradicting Lemma 8.2). Let p = ( s β + s β ) / B η be the Euclidean open ball with center p and radius η. Choose η small enough so that (i) B η ∩ [ s β , s β ] =] q , q [ with s β < q < q < s β where < stands forthe natural ordering on [ s β , s β ] . ii) x x π β, ( x ) − x π β, ( x ) has constant sign on B η . Let x a ( x ) be such a = 0 on ∆ \ B η , a > B η and 0 ≤ a ≤ η on∆ . Then, the alpha limit set of q equals s β , for both F and F π β butsince ˙ x − ˙ x doesn’t vanish on B η the trajectory through q exits B η at a point = q and, consequently, the omega limit set of q for F isdistinct from s β . This proves that F and F π β are not equivalent. The preceding construction shows that χ rev ( β, U ) is not structurallystable for an arbitrary potential game but this might be the casefor particular examples. Consider for example the Gibbs model de-scribed in Remark 5.4. For U ∈ R n and U = ( U ij ) symmetric, let χ rev ( β, U , U ) be the set of C vector field given by (1) with L ( x )irreducible and reversible with respect to the Gibbs measure (14). Question
For generic ( U , U ) and β large enough, is χ rev ( β, U , U )structurally stable ? Let L be an irreducible rate matrix and π ∈ ˙∆ denote the invariantprobability of L . That is the unique solution (in ∆) of πL = 0 . Forall f, g ∈ R n we let h f, g i = X i f i g i , h f, g i π = X i f i g i π i and h f, g i /π = X i f i g i π i . The
Dirichlet form of L is the map E : R n R + defined as E ( f ) = −h f, Lf i π = 12 X i,j ( f i − f j ) L ij π i . By irreducibility, E ( f ) > f is constant, and the spectral gap λ = sup {E ( f ) : h f, i π = 0 , h f, f i π = 1 }
39s positive. We let L ∗ be the irreducible rate matrix defined by L ∗ ij = π j L ji π i . Note that L ∗ admits π as invariant probability and that L ∗ is theadjoint of L for h , i π . We let L T : T ∆ T ∆ be defined by L T h = hL. Finally recall that for all f ∈ R n fπ stands for the vector defined by( fπ ) i = f i π i , i = 1 . . . n. Lemma 9.1
For all u, v ∈ T ∆ h L T u, v i /π = h L ∗ ( uπ ) , vπ ) i π In particular L T is invertible and L T is a definite negative operatorfor h , i π whenever L is reversible with respect to π. Proof.
The first assertion follows from elementary algebra. For thesecond, note that h L T u, u i /π = −E ( uπ ) . Thus, by irreducibility, h L T u, u i /π < u = 0 . (cid:4) Given f ∈ R n we write f ≥ f i ≥ i. We let 1 ∈ R n denote the vector which components are all equal to 1 . For all t ≥ P t = e tL . Since L is a rate matrix, ( P t ) is a Markov semigroupmeaning that P t f ≥ f ∈ R n with f ≥ P t . Lemma 9.2
Let I ⊂ R be an open interval and S : I R a C function such that S ′′ ( t ) ≥ α > . Let f ∈ R n be such that f i ∈ I forall i. Then ddt h S ( P t f ) , i π | t =0 ≤ − α E ( f ) . roof. For all u, v ∈ I S ( v ) − S ( u ) − S ′ ( u )( v − u ) ≥ α/ v − u ) . Hence for all i, jS ( f j ) − S (( P t f ) i ) − S ′ (( P t f ) i )( f j − ( P t f ) i ) ≥ α/ f j − ( P t f ) i ) . Applying P t to this inequality gives P t ( Sf ) i − S (( P t f ) i ) ≥ α/ P t ( f i − ( P t f ) i ) ) = α/ P t f i − ( P t f ) i )Hence P t ( Sf ) − S (( P t f )) ≥ α/ P t ( f − ( P t f )) ) = α/ P t f − ( P t f ) ) . Therefore, using the fact that h P t g, i π = h g, i π leads to h Sf − S ( P t f ) , i π ≥ α h f − ( P t f ) , i π . Dividing by t and letting t → (cid:4) Let S :]0 , ∞ [ R be a C function with positive second deriva-tive. Let H Sπ : ∆ R be the map defined by H Sπ ( x ) = X i π i S ( x i π i ) . Corollary 9.3
For all x ∈ ∆ h∇ H Sπ ( x ) , xL i ≤ − αλV ar π ( f ) where f i = x i π i Proof.
For x ∈ ∆ let x ( t ) = xe tL , f i = x i π i , f i ( t ) = x i ( t ) π i and P ∗ t g = e tL ∗ g. Note that P ∗ t ) is the adjoint of P t with respect to h , i π . For all g ∈ R n , h x ( t ) , g i = h x, P t g i = h f, P t g i π = h P ∗ t f, g i π sothat f ( t ) = P ∗ t f. Hence by the preceding lemma applied to L ∗ itfollows that h∇ H Sπ ( x ) , xL i = ddt h S ( P ∗ t f ) , i π | t =0 ≤ − α E ( f ) ≤ − αλV ar π ( f )where α = min i S ′′ ( x i π i ) > . (cid:4) We now prove the Lemma. Set S ( t ) = R t s ( u ) du. Then for all u ∈ T ∆ h∇ H Sπ ( x ) , u i = X i u i s ( x i π i )and the results follows from Corollary 9.3.41 eferences [1] M Bena¨ım, Vertex-reinforced random walks and a conjecture ofPemantle , Ann. Probab. (1997), no. 1, 361–392.[2] , Dynamics of stochastic approximation algorithms ,S´eminaire de Probabilit´es, XXXIII, Lecture Notes in Math.,vol. 1709, Springer, Berlin, 1999, pp. 1–68.[3] M. Bena¨ım, O. Benjamini, J. Chen, and Y. Lima,
A generalizedpolya’s urn with graph based interactions , Random Structuresand Algorithms (2013), To appear.[4] M Bena¨ım and M W. Hirsch,
Asymptotic pseudotrajectories andchain recurrent flows, with applications , J. Dynam. DifferentialEquations (1996), no. 1, 141–176. MR 1388167 (97d:58165)[5] , n × coordinative games , (2000), UnpublishedManuscript.[6] M. Bena¨ım and O. Raimond, Self interacting diffusions iii:Symmetric interactions , Annals of Probability (2005), no. 5,1716–1759.[7] M. Bena¨ım, O. Raimond, and B. Schapira,
Strongly reinforcedvertex-reinforced random walks on the complete graph , ALEA(2013), To appear.[8] M Bena¨ım and J. Weibull,
Deterministic approximation ofstochastic evolution in games , Econometrica (2003), no. 3,873–903.[9] J. Chen and C. Lucas, Generalized polya’s urn: convergence atlinearity , (2013), Preprint, arXiv:1306.5465.[10] R. Chill, A. Haraux, and M. Ali-Jendoubi,
Applications of thelojasiewicz simon, gradient inequality to gradient-like evolutionequations , Analysis and Applications (2009), no. 4, 351–372.[11] P. Dupuis and M. Fisher, On the construction of lyapunovfunctions for nonlinear markov processes via relative entropy ,(2011), Preprint, Lefschetz Center for Dynamical Systems.4212] M. Freildin and A.D. Wentzell,
Random perturbations of dy-namical sytems , Springer, 2012, 3ed.[13] M. W. Hirsch,
Differential topology , vol. 33, Springer, 1976.[14] J. Hofbauer and K. Sigmund,
Evolutionary games and popula-tion dynamics , Cambridge University Press, 1998.[15] J. Hofbauer, S. Sorin, and Y. Viossat,
Time average replicatorand best reply dynamics , Mathematics of Operation Research (2009), no. 2, 263–269.[16] T. G. Kurtz, Solutions of ordinary differential equations as lim-its of pure jump markov processes , Journal of Applied Proba-bility (1970), 49–58.[17] S. Lojasiewicz, Une propri´et´e topologique des sous-ensemblesanalytiques r´eels , Les ´Equations aux D´eriv´ees Partielles (1963),87–89, ´Editions du C.N.R.S, Paris.[18] B. Merlet and T. N. Nguyen,
Convergence to equilibrium fordiscretizations of gradient-like flows on riemannian manifolds. ,Differential Integral Equations (2013), no. 5-6, 571–602.[19] J. Palis and W. de Melo, Geometric theory of dynamical sys-tems , Springer, 1980.[20] R. Pemantle,
Vertex-reinforced random walk , Probab. TheoryRelated Fields (1992), 117–136.[21] , A survey of random processes with reinforcement ,Probab. Surv. (2007), 1–79. MR 2282181 (2007k:60230)[22] C. Robinson, Dynamical systems: Stability, symbolic dynamicsand chaos , CRC Press, Boca Raton, 1999, 2ed.[23] W.H. Sandholm,
Population games and evolutionary dynamics ,MIT Press, 2010.[24] ,
Population games and deterministic evolutionary dy-namics , Handbook of Game Theory and Economic Applications (2014), Young and S. Zamir, eds, Forthcoming.4325] R. van der Hofstad, M. Holmes, A. Kuznetsov , and W. Ruszel, Strongly reinforced polya urns with graph-based competition ,(2014), Preprint, arXiv:1406.0449.
Acknowledgments
This work was supported by the SNF grant 2000020 //