The Cover Time of a (Multiple) Markov Chain with Rational Transition Probabilities is Rational
aa r X i v : . [ m a t h . P R ] F e b The Cover Time of a (Multiple) Markov Chainwith Rational Transition Probabilities is Rational
John Sylvester
School of Computing Science, University of Glasgow, Glasgow, UK [email protected]
Abstract
The cover time of a Markov chain on a finite state space is the ex-pected time until all states are visited. We show that if the cover timeof a discrete-time Markov chain with rational transitions probabilities isbounded, then it is a rational number. The result is proved by relatingthe cover time of the original chain to the hitting time of a set in anotherhigher dimensional chain. We also extend this result to the setting where k ≥ Keywords:
Cover Time, Random Walk, Markov Chain, Rational Number
Let ( X t ) t ≥ be a discrete-time Markov chain with transition matrix P on a statespace Ω, see [5, 6]. For S ⊆ Ω, a subset of the state space, we let τ S := inf { t ≥ X t ∈ S } , be the first time that S is visited. If S = { s } is a singleton set we abuse notationslightly by taking τ s to mean τ { s } . For x ∈ Ω let E x [ τ S ] = E [ τ S | X = x ] bethe expected hitting time of S ⊆ S for a chain started from x . We say that aMarkov chain is irreducible if for every x, y ∈ Ω there exists some t ≥ P t ( x, y ) >
0, where P t ( x, y ) denotes the probability a chain started at x is atstate y after t ≥ P ( x, y ) = P ( x, y ). We let Q denote the setof rational numbers and say a chain is rational if all its transition probabilitiesare rational, i.e. P ( x, y ) ∈ Q for all x, y ∈ Ω. The following result is well known.1 roposition 1 (Folklore) . Let ( X t ) t ≥ be a discrete-time irreducible rationalMarkov chain on a finite state space Ω . Then E x [ τ y ] ∈ Q for any x, y ∈ Ω . Proposition 1 above follows from the fact E x [ τ y ] = 1 + P z ∼ x P x,z · E z [ τ y ]and thus can be solved as a set of linear equations. We shall need a strengthenedversion of this folklore result, which we state and prove in Section 2.Let τ cov denote the (random) first time all states are visited, that is τ cov := inf ( t ≥ t [ k =0 { X k } = Ω ) . For x ∈ Ω let E x [ τ cov ] be the cover time from x , that is, the expected time forthe chain to visit all states when started from x . Our main result establishesthe analogous result to Proposition 1 for the cover time. Theorem 2.
Let ( X t ) t ≥ be a discrete-time rational Markov chain on a finitestate space Ω . Then for any x ∈ Ω such that E x [ τ cov ] < ∞ we have E x [ τ cov ] ∈ Q . The finite state space assumption is necessary to ensure the cover time isbounded. However, our result does not require irreducibility, just that the covertime from the given start vertex is bounded.
Proposition 3 (Folklore) . Let ( X t ) t ≥ be a discrete-time irreducible Markovchain on a finite state space Ω . Then for any x, y ∈ Ω we have E x [ τ y ] < ∞ andconsequently E x [ τ cov ] < ∞ . The folklore result above shows that finite cover time condition in Theorem 2is implied for all start vertices by irreducibility, this gives the following corollary.
Corollary 4.
Let ( X t ) t ≥ be a discrete-time rational irreducible Markov chainon a finite state space Ω . Then E x [ τ cov ] ∈ Q for any x ∈ Ω . For an example of a non-irreducible Markov chain to which we can applyTheorem 2, one may consider a random walk on the graph consisting of a di-rected edge from x to y where y is a vertex in an n -vertex undirected clique. Inthis example the cover time from x is simply the cover time of an n -vertex cliqueplus one. This chain is clearly not irreducible and the cover time from any othervertex is unbounded/undefined as x cannot be reached. It is fairly easy to seethat rational transition probabilities are necessary, for a concrete example if onefixes any real number r ≥ P = − /r /r /r − /r ! , (1)has cover time r . This also gives the following corollary.2 orollary 5. The set of cover times attainable by finite discrete-time irreduciblerational Markov chains is ( Q ∩ [1 , ∞ )) ∪ { } . To prove Theorem 2 for a chain P on Ω we must introduce an auxiliaryMarkov chain Q on a larger state space V where | V | ≤ | Ω | · | Ω | . To be moreprecise, given a Markov chain P on Ω we define the auxiliary chain Q := Q ( P )to be the Markov Chain on the state space V := V (Ω) given by V = { ( x, S ) : x ∈ Ω , S ⊆ Ω , x ∈ S } , (2)and transition matrix given by Q (( x, S ) , ( y, S ∪ { y } )) = P ( x, y ) . (3)Observe that V ⊆ Ω × P (Ω) where P (Ω) = { S ⊆ Ω } is the power-set ofΩ. One may think of Q as a directed graph consisting of many ‘layers’, whereeach layer is a subset of Ω. These layers are linked by directed edges whichare crossed when a new state in Ω is ‘visited’ for the first time, where by firstvisited we mean that at some time t the current state is ( x t , S t ) ∈ V and x t = x i for any i < t . Thus, since a sequence x , x , . . . in the first component evolvesaccording to P by (3), each layer encodes which states of chain have been visitedso far by a trajectory in P . Figure 1 shows an example of the auxiliary chain Q of a simple chain P on three states.The next result equates the cover time in P to the hitting time of a specificset in the auxiliary chain Q ( P ). For clarity we use the notation E P · [ · ] tohighlight the specific chain, in this case P , in which the expectation is taken. Lemma 6.
Let P be a Markov chain on a state space Ω . Let Q := Q ( P ) be theassociated auxiliary chain on V := V (Ω) , and C = { ( x, Ω) | x ∈ Ω } ⊂ V . Thenfor any x ∈ Ω we have E P x [ τ cov ] = E Q ( x, { x } ) [ τ C ] . To prove Theorem 2 we encode the cover time of P as a hitting time in Q using Lemma 6 then deduce it is rational by Proposition 8, which is a strength-ening of Proposition 1 in the introduction.Similar constructions to Q were used by the author and co-authors in thestudy of the Choice and ε -TB random walks, which are walks where a controllercan influence which vertices are visited. In particular they were used to showthat there exist optimal strategies for covering a graph by these walks which aretime invariant in a certain sense [3] and to show the computational problem offinding optimal strategies to cover a graph by these walks is in PSPACE [4].We shall now introduce the notion of multiple Markov chains, which havebeen studied for their applications to parallelising algorithms driven by ran-dom walks, see [1] and subsequent papers citing it. For any k ≥
1, let X t =31 , { } ) (2 , { } ) / / (3 , { } ) / / (2 , { , } ) / / (1 , { , } )(3 , { , } ) / / / / (2 , { , } )(1 , { , } ) (3 , { , , } ) / / / (1 , { , , } )(2 , { , , } ) / C / / / / Figure 1: This figure shows an example of a simple Markov chain P on threestates (bottom right) and its associated auxillary chain Q ( P ), where the set C from Lemma 6 is shown in the red shaded ellipse. (cid:0) X (1) t , . . . , X ( k ) t (cid:1) be the k - multiple of the Markov chain P where each X ( i ) t is anindependent copy of the chain P run simultaneously on the same state space Ω.We denote the conditional expectation E ( x (1) ,...,x ( k ) ) [ · ] := E h · | X = ( x (1) , . . . , x ( k ) ) i , where for each 1 ≤ i ≤ k , X ( i )0 = x ( i ) ∈ Ω is the start state of the i th walk. Welet the random variable τ ( k ) cov = inf { t : S ti =0 { X (1) i , . . . , X ( k ) i } = V } be the firsttime every vertex of the graph has been visited by some walk X ( i ) t .We show that Theorem 2 also holds for k -multiple Markov chains. Theorem 7.
Let k ≥ and ( X t ) t ≥ be the k -multiple of a discrete-time ra-tional Markov chain on a finite state space Ω . Then for any x ∈ Ω k such that E x h τ ( k ) cov i < ∞ we have E x h τ ( k ) cov i ∈ Q . In this section we shall prove all claims in Section 1, including folklore resultsfor which the author could not find a suitable reference.4 .1 A Strengthening of Proposition 1
To prove the main theorem we use the following result, which is a strengtheningof Proposition 1 that does not require irreducibility.
Proposition 8.
Let P be a discrete-time rational Markov chain on a finite statespace Ω . For a non-empty set S ⊆ Ω let Ω( S ) = { x ∈ Ω | E x [ τ S ] < ∞} . Thenfor any S ⊆ Ω and x ∈ Ω( S ) we have E x [ τ S ] ∈ Q . To see why Proposition 8 implies Proposition 1, any P and S ⊆ Ω satis-fying Proposition 1 also satisfy Proposition 8. However, in addition P mustbe irreducible. Thus, by Proposition 3, all hitting times are bounded and soΩ( S ) = Ω.Before proving Proposition 8 we must state a definition and result from [2]. Definition 9.
A complex square matrix A = ( A ( i, j )) is said to weakly chaineddiagonally dominant (WCDD) if it satisfies the following:(i) all rows i satisfy | A ( i, i ) | ≥ P j = i | A ( i, j ) | , and(ii) for each row r , there exists a path in the graph of A from r to a row p which satisfies | A ( p, p ) | > P j = p | A ( p, j ) | . The following result shows that WCDD matrices are invertible.
Lemma 10 ([2, Lemma 3.2]) . A WCDD matrix is nonsingular.
We can now use Lemma 10 to prove Proposition 8.
Proof of Proposition 8.
To begin observe that Ω( S ) = ∅ since S ⊆ Ω( S ) and E s [ τ S ] = 0 for all s ∈ S . Observe that for any y ∈ Ω, if there exists some x ∈ Ω( S ) and t ≥ P t ( x, y ) >
0, then y ∈ Ω( S ) or else we contradictour assumption that E x [ τ S ] < ∞ . Denote ω := | Ω( S ) | .Now, by definition, each entry of the vector h := ( E x [ τ S ]) x ∈ Ω( S ) is boundedand h is uniquely determined by the following set of linear equations E x [ τ S ] = P y P xy · E y [ τ S ] if x S x ∈ S. This can be expressed as Ah = b for the ω × ω matrix A := ( I − M ) where I is the ω × ω identity matrix, M ( i, j ) = P ( i, j ) if i, j / ∈ S and 0 otherwise, and b is a 0-1 vector of length ω . Claim.
The matrix A is invertible. roof of Claim. We shall show that A = ( I − M ) is WCDD, that is it satisfiesProperties ((i)) and ((ii)) of Definition 9, the claim then follows by Lemma 10.To begin observe that M a sub-matrix of P where rows and columns corre-sponding to states not in Ω( S ) have been removed. Thus all entries of M arenon-negative and P j = i M ( i, j ) ≤ P j = i P ( i, j ) = 1 and so A = ( I − M ) satisfiesProperty ((i)).Property ((ii)) follows from the fact that there must be at least one row p of M such that P j M ( p, j ) <
1. To see this we recall that S = ∅ and S ⊆ Ω( S ) and consider two cases. Firstly, if S = Ω( S ) then for any s ∈ S wehave P j M ( p, j ) = 0. Otherwise, there must exist at least one pair x, s , where x ∈ Ω( S ) \ S and s ∈ S , such that P ( x, s ) >
0, however M ( x, s ) = 0 and so P j M ( x, j ) < P j P ( x, j ) = 1. ♦ Since P is rational, for each i, j ∈ Ω there exist natural numbers a i,j , d i,j ,where d i,j = 0, such that P ( i, j ) = a i,j /d i,j . As Ω( S ) is finite we can take D := Q i,j ∈ Ω( S ) d i,j as a common denominator for all { P ( i, j ) } i,j ∈ Ω( S ) and thusalso for all { A ( i, j ) } i,j ∈ Ω( S ) . By the Claim A − exists, and so h = A − b =det( A ) − C T b , where C is the matrix of cofactors. The determinant det( A ) = P σ ∈ S ω sgn( σ ) Q ωi =1 A ( i, σ ( i )) is a symmetric polynomial of degree ω whoseterms are distinct elements of A , thus it is also rational and can be placedover the common denominator D ω . Since each term C ( i, j ) of the Cofactormatrix C is given by det( A i,j ) where A i,j is the matrix A with the i th row and j th column deleted, it follows the entries of C are also rational and can be putover common denominator D ω − . Thus, since the entries of b are in { , } , theentries of h = det( A ) − C T b are rational. We now prove the key lemma which relates cover time in a Markov chain P tohitting times in the larger auxiliary Markov chain Q . Proof of Lemma 6.
We shall first explain how any trajectory ( X t ) t ≥ of theMarkov chain P on state space Ω can be coupled with a trajectory ( Y t ) t ≥ inthe auxiliary chain Q with state space V , where Q and V are given by (3) and(2) respectively. Given X = x let Y = ( x , { x } ) then, inductively, given atrajectory ( X i ) ti =0 = ( x i ) ti =0 we set ( Y i ) ti =0 = (cid:16)(cid:16) x i , S ij =0 { X j } (cid:17)(cid:17) ti =0 . Now by(3) the probability of the two trajectories is equal and given by P ( x , x ) · · · P ( x t − , x t ) = Q (( x , { x } ) , ( x , { x } ∪ { x } )) · · ·· Q x t − , t − [ j =0 { x j } , x t , t [ j =0 { x j } . (4)6hus given any trajectory ( X t ) t ≥ in P we can find a trajectory ( Y t ) t ≥ in Q with the same measure. To couple a given trajectory ( Y t ) t ≥ in Q to a trajectory( X t ) t ≥ in P is even simpler, given Y t = ( y t , S t ) we simply ‘forget’ the secondcomponent of and set X t = y t ∈ Ω. Again the measure is preserved by (4).To complete the proof we shall show, for any x ∈ Ω, the times τ cov and τ C in the coupled chains X t and Y t started from x and ( x, { x } ), respectively, areequidistributed.Suppose we take any trajectory ( X t ) Tt =0 of length T ≥ ∪ Tj =0 { X j } =Ω, then by the coupling above, we have Y T = (cid:0) x T , ∪ Tj =0 { X j } (cid:1) = ( x T , Ω) ∈ C .Since this holds for trajectory any T satisfying ∪ Tj =0 { X j } = Ω we can assumethat T is the first such time, that is we can take T = τ cov , it follows that τ C ≤ τ cov . Conversely let ( Y t ) Tt =0 be any trajectory in Q where Y = ( y , { y } ),for some y ∈ Ω and Y T ∈ C . Denote Y t = ( y t , S t ) where y t ∈ Ω and S t ∈ P (Ω)and recall that the only transitions supported by Q are from ( y, S ) to ( z, S ∪{ z } )where P ( y, z ) >
0. Then, since Y = ( y , { y } ), it follows that ∪ Tt =0 { y t } = Ωand thus, by the coupling above, ∪ Tt =0 { X t } = Ω. Again, since we can take T = τ C to be minimal we have τ cov ≤ τ C . This shows that for any pair ofcoupled trajectories with fixed start vertices x and ( x, { x } ) the times are τ cov and τ C are the same, the result follows by taking expectation.Having established Lemma 6 the proof of Theorem 2 is simple. Proof of Theorem 2.
Let Q := Q ( P ) be the auxiliary chain associated with P .By Lemma 6 we have E P x [ τ cov ] = E Q ( x, { x } ) [ τ C ] for any x ∈ Ω where C = { ( y, Ω) | y ∈ Ω } . By assumption we have E P x [ τ cov ] < ∞ and so x ∈ Ω( C ).It follows from Proposition 8 that E Q ( x, { x } ) [ τ C ] ∈ Q and so E P x [ τ cov ] ∈ Q asclaimed. For completeness we shall now prove Proposition 3 which states that the covertime of a finite irreducible Markov chain from any start vertex is finite.
Proof of Proposition 3.
Since the chain is irreducible, for every x, y ∈ Ω thereexists some fixed constants c ( x, y ) > T ( x, y ) < ∞ such that P T ( x,y ) ( x, y ) >c ( x, y ). Define c = min x,y ∈ Ω c ( x, y ) and T = max x,y ∈ Ω T ( x, y ). Observe that ifwe break a walk from any start vertex into sections of length T then for any y ∈ Ω the probability that y is hit within any section of length T is at least c .Thus, the time τ y to hit y is stochastically dominated by the number of walksections of length T until one contains y and so E x [ τ y ] ≤ T /c for any x, y ∈ Ω.Now we can fix a arbitrary ordering x , x , . . . , x | Ω | of the states, starting with x = x ∈ Ω, and we have E x [ τ cov ] ≤ P | Ω |− i =1 E x i (cid:2) τ x i +1 (cid:3) ≤ | Ω | · T /c < ∞ .7or completeness we also give a proof of Corollary 5. Proof of Corollary 5.
Corollary 4 ensures the cover time is rational and sincewe are in discrete time, the cover time must be grater than 1, unless it has onlyone state in which case it is 0. For the other direction, the example given by(1) shows that any r ∈ Q ∩ [1 , ∞ ) can be obtained as the cover time of a twostate rational irreducible Markov chain. Let P be an irreducible Markov chain on Ω and k ≥ P on the same statespace Ω we wish to run. Given P and k define the k -walk auxiliary chain M := M ( P , k ) to be the Markov Chain on state space W := W (Ω , k ) given by W = (cid:8) (( x , . . . , x k ) , S ) : x i ∈ Ω , S ⊆ Ω , x i ∈ S for all 1 ≤ i ≤ k (cid:9) ⊆ Ω k × P (Ω)with transition matrix given by M (( x , S ) , ( y , S ∪ { y (1) , . . . , y ( k ) } )) = P ( x (1) , y (1) ) · · · P ( x ( k ) , y ( k ) ) , (5)for any S ⊆ Ω and x , y ∈ Ω k where x = ( x (1) , . . . , x ( k ) ) and y = ( y (1) , . . . , y ( k ) ).Equipped with a definition of the axillary chain M which encodes k multiplewalks we can prove an analogue of Lemma 6, the proof is almost exactly thesame as Lemma 6 but we include an abridged version for completeness. Lemma 11.
Let P be an irreducible Markov chain on state space Ω , k ≥ be an integer. Let M := M ( P , k ) be the associated k -walk auxiliary chain withstate space W := W (Ω , k ) , and C = { ( x , Ω) | x ∈ Ω k } ⊂ W . Then for any x = ( x (1) , . . . , x ( k ) ) ∈ Ω k we have E Px h τ ( k ) cov i = E M ( x , { x (1) ,...,x ( k ) } ) [ τ C ] . Proof.
Given any trajectory ( X t ) t ≥ of a k -multiple of P we can define a tra-jectory ( Y t ) t ≥ in M ( P , k ) inductively. Given X = x = ( x (1)0 , . . . , x ( k )0 ) ∈ Ω k let Y = ( x , { x (1)0 , . . . , x ( k )0 } ) ∈ W . Then, given a trajectory ( X i ) ti =0 = ( x i ) ti =0 we set ( Y i ) ti =0 = (cid:16)(cid:16) x i , S ij =0 S kh =1 n X ( h ) j o(cid:17)(cid:17) ti =0 . Now by (5), t − Y i =0 k Y j =1 P ( x ( j ) i , x ( j ) i +1 ) = Q x , k [ h =1 n x ( h )0 o! , x , [ j =0 k [ h =1 n x ( h ) j o · · ·· Q x t − , t − [ j =0 k [ h =1 n x ( h ) j o , x t , t [ j =0 k [ h =1 n x ( h ) j o . k walkers in atrajectory ( X i ) Ti =0 then, since each of these states must have been added tothe set in the second coordinate of Y i , the coupled walk ( Y i ) Ti =0 must have hit C . For the other direction if we project a trajectory ( Y i ) Ti =0 in M starting at( y , { y (1) , . . . , y ( k ) } ) ∈ W onto the first k coordinates to give a k -multiple walk( X i ) Ti =0 , then if the walk ( Y i ) Ti =0 hit C then for every x ∈ Ω there is exists0 ≤ i ≤ T and j ∈ [ k ] such that X ( j ) i = x and thus Ω has been covered. Itfollows that τ ( k ) cov and τ C are equidistributedNow Theorem 7 follows easily from Lemmas 8 and 11. Proof of Theorem 7.
Let M be the k -walk auxiliary chain associated with P .Then E Px h τ ( k ) cov i = E M ( x , { x (1) ,...,x ( k ) } ) [ τ C ] for any x = ( x (1) , . . . , x ( k ) ) ∈ Ω k byLemma 11, where C = { ( y , Ω) | y ∈ Ω k } . By assumption we have E Px [ τ cov ] < ∞ and so x ∈ Ω( C ). It follows from Proposition 8 that E M ( x , { x (1) ,...,x ( k ) } ) [ τ C ] ∈ Q and so E Px [ τ cov ] ∈ Q as claimed. Acknowledgements
The author is currently supported by Engineering and Physical Sciences Re-search Council (ESPRC) grant number EP/T004878/1. This work was startedwhile the author was supported by ERC Starting Grant no. 679660 at the Uni-versity of Cambridge.
References [1] Noga Alon, Chen Avin, Michal Kouck´y, Gady Kozma, Zvi Lotker, andMark R. Tuttle. Many random walks are faster than one.
Combin. Probab.Comput. , 20(4):481–502, 2011.[2] Parizad Azimzadeh and Peter A. Forsyth. Weakly chained matrices, policyiteration, and impulse control.
SIAM J. Numer. Anal. , 54(3):1341–1364,2016.[3] Agelos Georgakopoulos, John Haslegrave, Thomas Sauerwald, and JohnSylvester. The power of two choices for random walks. arXiv:1911.05170 ,2019.[4] John Haslegrave, Thomas Sauerwald, and John Sylvester. Time dependentbiased random walks. arXiv:2006.02475 .95] David A. Levin and Yuval Peres.
Markov chains and mixing times . Amer-ican Mathematical Society, Providence, RI, 2017. Second edition of [MR2466937], With contributions by Elizabeth L. Wilmer, With a chapteron “Coupling from the past” by James G. Propp and David B. Wilson.[6] James R. Norris.
Markov chains , volume 2 of