Optimal steering to invariant distributions for networks flows
aa r X i v : . [ ee ss . S Y ] F e b OPTIMAL STEERING TO INVARIANT DISTRIBUTIONS FOR NETWORKS FLOWS 1
Optimal steering to invariant distributionsfor networks flows
Yongxin Chen,
Member, IEEE,
Tryphon T. Georgiou,
Fellow, IEEE, and Michele Pavon
Abstract —We derive novel results on the ergodic theory of irreducible, aperiodic Markov chains. We show how to optimally steer thenetwork flow to a stationary distribution over a finite or infinite time horizon. Optimality is with respect to an entropic distance betweendistributions on feasible paths. When the prior is reversible, it shown that solutions to this discrete time and space steering problem arereversible as well. A notion of temperature is defined for Boltzmann distributions on networks, and problems analogous to cooling (inthis case, for evolutions in discrete space and time) are discussed.
Index Terms —Controlled Markov chain, Schr¨odinger Bridge, Reversibility, Regularized Optimal Mass Transport ✦ NTRODUCTION
We consider an optimal steering problem for networksflows over a finite or infinite time horizon. Specifically, wederive discrete counterparts of our cooling results in [5].The goal is to steer the Markovian evolution to a steadystate with desirable properties while minimizing relativeentropy, or relative entropy rate, over admissible distribu-tions on paths. This relates to a special Markov DecisionProcess problem, cf. [9, Section 6] which is referred to as
Schr¨odinger’s Bridge Problem (SBP) or regularized optimal masstransport (OMT). A rich history on this circle of ideas origi-nates from two remarkable papers by Erwin Schr¨odinger in1931/32 [31], [32] who was interested in a large deviationproblem for a cloud of Brownian particles, and of earlierwork by Gaspar Monge in 1781 on the problem of optimallytransporting mass between sites [27].Important contributions on the existence question inSBP, left open by Schr¨odinger, were provided over timeby Fortet, Beurling, Jamison and F¨ollmer [2], [19], [20],[22]. It should be remarked that Fortet’s proof in 1940 isalgorithmic, establishing convergence of a (rather complex)iterative scheme. This predates by more than twenty yearsthe contribution of Sinkhorn [34] who established conver-gence in a special case of the discrete problem The latterproof is much simpler than in the continuous (time andspace) problem solved by Fortet. Thus, these algorithmsshould be called Fortet-IPF-Sinkhorn, where IPF stands forthe Iterative Proportional Fitting algorithm proposed in 1940without proof of convergence in [16]. These problems admita “static” and a “dynamic” formulation. While the static,discrete space problem was studied by many, starting fromSinkhorn, see [14], [29] and references therein, the dynamic,discrete problem was considered in [11], [12], [13], [21], [28].We provide in this paper a new fluid-dynamic derivationof the Schr¨odinger system on which the iteration is based.Convergence of the iterative scheme in a suitable projective • Y. Chen is with the School of Aerospace Engineering, Georgia Institute ofTechnology, Atlanta, GA 30332, USA, T.T. Georgiou is with the Depart-ment of Mechanical and Aerospace Engineering,University of California,Irvine, CA 92697, USA, M. Pavon is with the Department of Mathematics“Tullio Levi-Civita”,Universit`a di Padova, 35121 Padova, Italy. metric was also established in [21] for the discrete case andin [7] for the continuous case.This topic lies nowadays at the crossroads of many fieldsof science such as probability, statistical physics, optimalmass transport, machine learning, computer graphics, statis-tics, stochastic control, image processing, etc. Several surveypapers have appeared over time emphasizing different as-pects of the subject, see [8], [9], [10], [23], [29], [37].The paper is outlined as follows. In Section 2, we presentbackground on the discrete-time and space Schr¨odingerbridge problem while emphasizing its “fluid dynamic” for-mulation (the original proof of a key well-known result isdeferred to the Appendix). In Section 3, we introduce thecorresponding infinite-horizon steering problem by mini-mizing the entropy rate with respect to the prior measure.We then discuss existence for the one-step Schr´odingersystem leading to existence for the steering problem. Section4 is devoted to an interesting result linking optimality in thesteering problem to reversibility of the solution. The finalSection 5 applies all the previous results to cooling, wherethe goal is to optimally steer the Markov chain to a steadystate corresponding to a lower effective temperature.
ISCRETE - TIME DYNAMIC BRIDGES : BACK - GROUND
We begin by discussing a paradigm of great significancein network flows. It amounts to designing probabilistictransitions between nodes, and thereby probability lawson path spaces, so as to reconcile marginal distributionswith priors that reflect on the structure of the network andobjectives on transferance of resourses across the network.The basic formulation we discuss amounts to the so calledSchr¨odinger bridge problem in discrete-time. This dynamicformulation echoes the fluid dynamic formulation of theclassical (continuous time and space) Schr¨odinger bridgeproblem. Certain proofs that we present are mildly new.We consider a directed, strongly connected, aperiodicgraph G = ( X , E ) with vertex set X = { , , . . . , n } andedge set E ⊆ X × X , and we consider trajectories/paths onthis graph over the time set T = { , , . . . , N } . The family of OPTIMAL STEERING TO INVARIANT DISTRIBUTIONS FOR NETWORKS FLOWS feasible paths x = ( x , . . . , x N ) of length N , namely pathssuch that x i x i +1 ∈ E for i = 0 , , . . . , N − , is denotedby F P N ⊆ X N +1 . We seek a probability distribution P onthe space of paths FP N with prescribed initial and finalmarginals ν ( · ) and ν N ( · ) , respectively, and such that theresulting random evolution is closest to a “prior” measure M on FP N in a suitable sense.The prior law for our problem is a Markovian evolution µ t +1 ( x t +1 ) = X x t ∈X µ t ( x t ) m x t x t +1 ( t ) (1)with nonnegative distributions µ t ( · ) over X , t ∈ T , andweights m ij ( t ) ≥ for all indices i, j ∈ X and all times. Inaccordance with the topology of the graph, m ij ( t ) = 0 forall t whenever ij
6∈ E . Often, but not always, the matrix M ( t ) = [ m ij ( t )] ni,j =1 (2)does not depend on t . Here, m ij plays the same role asthat of the heat kernel in continuous-time. However, herein,the transition matrix M ( t ) may not represent a probability transition matrix in that the rows of M ( t ) do not necessarilysum up to one. Thus, the “total transported mass” is notnecessarily preserved, in that P x t ∈X µ t ( x t ) may be afunc-tion of time. A particular case of interest is when M is the adjacency matrix of the graph that encodes the topologicalstructure of the network.The evolution (1), together with measure µ ( · ) , whichwe assume positive on X , i.e., µ ( x ) > for all x ∈ X , (3)induces a measure M on FP N as follows. It assigns to apath x = ( x , x , . . . , x N ) ∈ FP N the value M ( x , x , . . . , x N ) = µ ( x ) m x x (0) · · · m x N − x N ( N − , (4)and gives rise to a flow of one-time marginals µ t ( x t ) = X x ℓ = t M ( x , x , . . . , x N ) , t ∈ T . Definition 1.
We denote by P ( ν , ν N ) the family of prob-ability distributions on FP N having the prescribedmarginals ν ( · ) and ν N ( · ) .We seek a distribution in this set which is closest tothe prior M in relative entropy (divergence, Kullback-Leiblerindex) defined by D ( P k Q ) := ( P x P ( x ) log P ( x ) M ( x ) , Supp( P ) ⊆ Supp( M ) , + ∞ , Supp( P ) Supp( M ) , Here, by definition, · log 0 = 0 . The value of D ( P k M ) mayturn out to be negative due to the different total masses inthe case when M is not a probability measure. The optimiza-tion problem, however, poses no challenge as the relativeentropy is (jointly) convex over this larger domain and isbounded below. This brings us to the so-called Schr¨odingerBridge Problem (SBP):
Problem 2.
Determine P ∗ [ ν , ν N ] := argmin { D ( P k M ) | P ∈ P ( ν , ν N ) } . (5) By analyzing the contribution to the entropy functionalof transitions along edges, in a sequential manner, we ariveat a formulation that parallels the fluid-dynamic formu-lation of the Schr¨odinger problem in continuous time ,considered in [23, Section 4] and [6].Suppose we restrict our search in Problem 2 to Marko-vian P ’s. In analogy to (2), we then have P ( x , x , . . . , x N ) = ν ( x ) π x x (0) · · · π x N − x N . (6)Also let p t be the one-time marginals of P . i.e. p t ( x t ) = X x ℓ = t P ( x , x , . . . , x N ) , t ∈ T . We finally have the update mechanism p t +1 ( x t +1 ) = X x t ∈X p t ( x t ) π x t x t +1 ( t ) (7)which, in vector form, is p t +1 = Π ′ ( t ) p t . (8)Here Π = [ p ij ( t )] ni,j =1 is the transition matrix and primedenotes transposition. Using (4)-(6) we obtain D ( P k M ) = D ( ν k µ )+ N − X t =0 X x t D ( π x t x t +1 ( t ) k m x t x t +1 ( t )) p t ( x t ) . Since D ( ν k µ ) is invariant over P ( ν , ν N ) , we can nowhave the following equivalent (fluid dynamic) formulation: Problem 3. min ( p,π ) ( N − X t =0 X x t D (cid:0) π x t x t +1 ( t ) k m x t x t +1 ( t )) p t ( x t ) (cid:1)) , subject to p t +1 ( x t +1 ) = X x t ∈X p t ( x t ) π x t x t +1 ( t ) , (9) X x t +1 π x t x t +1 ( t ) = 1 , ∀ x t ∈ X , and (10) p ( x ) = ν ( x ) , p N ( x N ) = ν N ( x N ) , (11)for t ∈ { , , . . . , N − } . (12)Here p = { p t ; t = 0 , , . . . , N } is a flow of prob-ability distributions on X (corresponds to the flowof densities ρ t in the continuous setting) and π = { π x t x t +1 ( t ); t = 0 , , . . . , N − } is a flow of transitionprobabilities (corresponds to the flow of the drifts). Therelative entropy D (cid:0) π x t x t +1 ( t ) k m x t x t +1 ( t )) (cid:1) between tran-sition laws, corresponds to the square of the modulus ofthe velocity field in the continuous setting [6], so that D (cid:0) π x t x t +1 ( t ) k m x t x t +1 ( t )) p t ( x t ) (cid:1) in fact corresponds to ki-netic energy.We do not impose explicitly the non-negativity con-straint on the p t and π x t x t +1 ( t ) as it will be automaticallysatisfied (due to the fact that the relative entropy growsunbounded at the boundary of the admissible distributions
1. The fluid dynamic reformulation of Schr¨odinger’s bridge problemin continuous space/time drew important connections with Monge-Kantorovich optimal mass transport (OMT) [1] –the latter seen as alimit when the stochastic excitation reduces to zero [23]. In particular, itpointed to a time-symmetric formulation of Schr¨odinger’s bridges [6].
HEN, GEORGIOU, AND PAVON 3 and transition probabilities). We have the following theo-rem:
Theorem 4.
Suppose there exists a pair of nonnegativefunctions ( ϕ, ˆ ϕ ) defined on [0 , T ] × X and satisfying thesystem ϕ ( t, x t ) = X x t +1 m x t x t +1 ( t ) ϕ ( t + 1 , x t +1 ) , (13) ˆ ϕ ( t + 1 , x t +1 ) = X x t m x t x t +1 ( t ) ˆ ϕ ( t, i ) , (14)for ≤ t ≤ ( T − , as well as the boundary conditions ϕ (0 , x ) · ˆ ϕ (0 , x ) = ν ) ( x ) , ϕ ( T, x T ) · ˆ ϕ ( T, x T ) = ν T ( x T ) , (15)for x t ∈ X and t ∈ { , T } , accordingly. Suppose more-over that ϕ ( t, i ) > , ∀ ≤ t ≤ T, ∀ i ∈ X . Then, theMarkov distribution P ∗ in P ( ν , ν N ) having transitionprobabilities π ∗ x t x t +1 ( t ) = m x t x t +1 ( t ) ϕ ( t + 1 , x t +1 ) ϕ ( t, x t ) (16)solves Problem 3. Remark 5.
Notice that if ( ϕ, ˆ ϕ ) satisfy (13)-(14)-(15), so doesthe pair ( cϕ, c ˆ ϕ ) for all c > . Hence, uniqueness for theSchr¨odinger system is always intended as uniqueness ofthe ray.The proof of Theorem 4 is deferred to Appendix A. Ourderivation there of the Schr¨odinger system appears originaland alternative to the two usual approaches. The first oneuses Lagrange multipliers for the constraints involving themarginals [21] very much as in Schr¨odinger’s original spirit.The second uses a “completion of the relative entropy”argument, see [28, Section IV]. Let M ( t ) = ( m ij ( t )) . Underthe assumption that the entries of the matrix product G := M (0) M (1) · · · M ( N − M ( N − (17)are all positive, there exists a (unique in the projectivegeometry sense) solution to the system (13)-(15) which canbe computed through a Fortet-IPF-Sinkhorn iteration [14],[16], [20], [21], [34]. PTIMAL S TEERING TO A S TEADY S TATE
Consider a given prior transition rate between the verticesof the graph, represented by a matrix M as in (2), whichat present we assume as being time-invariant. Suppose π is a desired stationary probability distribution that we wishto maintain over the state space X for the distribution ofresources, goods, packets, vehicles, etc. at steady state. Weaddress below the following basic question: How can wemodify M so that the new evolution is close to the prior inrelative entropy while at the same time it admits π as invari-ant distribution? We address the above question next andprovide a partial answer to the complementary question onwhat conditions ensure that any given π can be renderedthe stationary distribution with a transition law compatiblewith the topology of the graph. We consider the following problem inspired by [15]. Givena (prior) measure M ∈ FP N as in Section 2, correspondingto a constant transition matrix M , and a probability vector π on X , we are seeking a (row) stochastic matrix Π such that Π ′ π = π while the corresponding (time-invariant) measure P on X ×X × · · · is close to M in a suitable sense. Our interest herefocuses on the infinite horizon case where N is arbitrarilylarge.To this, we let P be the family of probability distributionson FP N , and consider the following problem. Problem 6. min P ∈P lim N →∞ N D (cid:0) P [0 ,N ] k M [0 ,N ] (cid:1) subject to Π ′ π = π, Π = , where is the vector with all entries equal to .By formula (18) in [28], which applies to general (not neces-sarily mass preserving) Markovian evolutions, we have thefollowing representation for the relative entropy betweenMarkovian measures D (cid:0) P [0 ,N ] k M [0 ,N ] (cid:1) = D ( π k m ) + N − X k =0 X i k D ( π i k i k +1 k m i k i k +1 ) π ( i k ) . Thus, Problem 6 reduces to the stationary
Schr¨odinger bridgeproblem:
Problem 7. min ( π ij ) X i,j ∈X D ( π ij k m ij ) π ( i ) , subject to X i ∈X π ij π ( i ) = π ( j ) , j ∈ X , X j ∈X π ij = 1 , i ∈ X . Note that in the above we have ignored enforcing thenonnegativity of the π ij . This problem can readily be shownto be equivalent to a standard one-step Schr¨odinger bridgeproblem for the joint distributions p ( i, j ) and m ( i, j ) attimes t ∈ { , } with the two marginals equal to π . Indeed,a straightforward calculation gives X ij log π ( i, j ) m ( i, j ) π ( i, j ) = X ij log π ij π ( i ) m ij m ( i ) π ij π ( i )= X i D ( π ij k m ij ) π ( i ) + D ( π k m ) . (18)Following Theorem 4, suppose there exist two vectors ( ϕ ( t, · ) , ˆ ϕ ( t, · )) , for t ∈ { , } , with nonnegative entries, satisfying the Schr¨odinger system OPTIMAL STEERING TO INVARIANT DISTRIBUTIONS FOR NETWORKS FLOWS ϕ (0 , i ) = X j m ij ϕ (1 , j ) , (19a) ˆ ϕ (1 , j ) = X i m ij ˆ ϕ (0 , i ) , (19b) ϕ (0 , i ) · ˆ ϕ (0 , i ) = π ( i ) , (19c) ϕ (1 , j ) · ˆ ϕ (1 , j ) = π ( j ) . (19d)Thence, we conclude that π ∗ ij = ϕ (1 , j ) ϕ (0 , i ) m ij (20)satisfies both constraints of Problem 7. The correspondingmeasure P ∗ ∈ P solves Problem 6. There is, however, a difficulty . The assumption on the matrix G in (17), whichguarantees existence for the Schr¨odinger system, becomeshere the fact that M must have all positive elements. Thelatter property is typically not satisfied since M must com-ply with the topology of the graph. There exists, fortunately,a reasonable condition on M which ensures existence ofsolutions for (19). Definition 8. [18] A square matrix A = ( a ij ) is called indecomposable if no permutation matrix P exists suchthat A = P (cid:20) A A A (cid:21) P ′ where A and A are nonvacuous square matrices. A is called fully indecomposable if there exist no pair ofpermutation matrices P and Q such that A = P (cid:20) A A A (cid:21) Q (21)where A and A are nonvacuous square matrices. Remark 9.
A square, indecomposable matrix A has a realpositive simple eigenvalue equal to its spectral radius [35].Let A = ( a ij ) be the adjacency matrix of the graph G =( X , E ) , namely a ij = (cid:26) , ( i, j ) ∈ E , , ( i, j )
6∈ E . Then A is indecomposable if and only if G = ( X , E ) isstrongly connected [18, p.608]. Proposition 10.
Suppose M is fully indecomposable and π has all positive components. Then there exists a solutionto (19) with ϕ (0 , · ) and ϕ (1 , · ) with positive componentswhich is unique in the sense of Remark 5. Proof 1.
Let ϕ = ϕ (0 , · ) and ϕ = ϕ (1 , · ) . Observe that(19a)-(19b)-(20) admit the matricial form ϕ = M ϕ , (22a) ˆ ϕ = M ′ ˆ ϕ , (22b) Π ∗ = Diag( ϕ ) − M Diag( ϕ ) . (22c)The proof can now be constructed along the lines of [26,Theorem 5]. Another interesting question is the following: Given thegraph G , what distributions π admit at least one stochasticmatrix compatible with the topology of the graph for whichthey are invariant? Clearly, if all self loops are present( ( i, i ) ∈ E , ∀ i ∈ X ), any distribution is invariant with respectto the identity matrix which is compatible. Without sucha strong assumption, a partial answer is provided by thefollowing result. Proposition 11.
Let π be a probability distribution sup-ported on all of X , i.e. π ( i ) > , ∀ i ∈ X . Assume that theadjacency matrix of G = ( X , E ) is fully indecomposable.Then, there exist stochastic matrices Π compatible withthe topology of the graph G such that Π ′ π = π. Proof 2.
Take, in Problem 7, M = A = [ a ij ] the adjacencymatrix of the graph G . By Theorem 4 and Proposition10, Problem 7 has a solution which admits π as invariantdistribution.One final interesting question is the following: Assume thatthe adjacency matrix of the graph G is fully indecompos-able. Let Q π be the non empty set of stochastic matrices Q compatible with the topology of G and such that Q ′ π = π .Clearly Q is a convex set. Can we characterize the matrixin Q which induces a maximum entropy rate measure onthe feasible paths? We already know the answer to thisquestion. Indeed, consider the formulation of Problem 7with the distribution of the edges of formula (18). Recallthat maximizing entropy (rate) is equivalent to minimizingrelative entropy (rate) from the uniform. If we take as priorthe uniform distribution on the edges, M is just a positivescalar times the adjacency matrix A of the graph. Thus, thesolution to Problem 7 with A as prior transition providesthe maximum entropy rate. EVERSIBILITY
In [5, Corollary 2], we have shown that, in the case of areversible prior (Boltzmann density) for a stochastic oscil-lator, the solution of the continuous countepart of Problem6 (minimizing the expected input power) is reversible. Weprove next that the same remarkable property holds here inthe discrete setting.
Theorem 12.
Let the transition matrix M of the prior measure M be time invariant. Assume that M is reversible withrespect to µ = µ , i.e. Diag( µ ) M = M ′ Diag( µ ) . (23)Then, then the solution Π ∗ of Problem 7 is also reversiblewith respect to π . Proof 3.
Reversibility of M with respect to µ is equivalentto the statement that Σ := Diag( µ ) M is symmetric, asnoted. Define the vectors ˆ ψ = Diag( µ ) − ˆ ϕ , ψ = Diag( µ ) ϕ . Then, the first two equations in system (22) supple-mented with the marginal conditions read ψ = Σ ϕ , (24a) ˆ ϕ = Σ ′ ˆ ψ = Σ ˆ ψ , (24b) ψ ◦ ˆ ψ = ϕ ◦ ˆ ϕ = π, (24c) HEN, GEORGIOU, AND PAVON 5 where the operation ◦ in (24c) represents componentwisebetween vectors. Due to the symmetry of Σ and theuniqueness in the sense of Remark 5, we have that ˆ ψ = cϕ and ψ = c ˆ ϕ for some positive scalar c . Itreadily follows that Diag( π )Π ∗ = Diag( π )Diag( ϕ ) − M Diag( ϕ )= Diag( ˆ ϕ ) M Diag( ϕ )= Diag( ˆ ψ )Diag( µ ) M Diag( ϕ )= Diag( ˆ ψ ) M ′ Diag( µ )Diag( ϕ )= Diag( ˆ ψ ) M ′ Diag( ϕ ) − Diag( ψ )Diag( ϕ )= Diag( ϕ ) M ′ Diag( ϕ ) − Diag( ˆ ϕ )Diag( ϕ )= Π ′ Diag( π ) . (cid:3) Returning now to the question raised at the end ofSubsection 3.2, in the case when A is symmetric, we cantake π A := A . OOLING
In several advanced applications such as atomic forcemicroscopy and microresonators, it is often important todampen stochastic excitations (due to vibrations, moleculardynamics, etc.) affecting experimental apparatuses. This isoften accomplished through feedback, steering the systemto a desired steady state corresponding to an effective tem-perature which is lower than that of the environment (e.g.,fluid), see for instance [4], [17], [24], [25], [33], [36]. Optimalasymptotic and fast cooling of stochastic oscillators wasstudied in [5]. We show next how the results of Section3 permit to derive corresponding results in the context ofdiscrete space and time considered herein.
Consider the setting of the previous section where thetransition matrix of the prior measure M is time invariant.Let us introduce a Boltzmann distribution π T on X π T ( i ) := Z ( T ) − exp (cid:20) − E i kT (cid:21) , Z ( T ) = X i exp (cid:20) − E i kT (cid:21) , (25)corresponding to the “energy function” E x , for x ∈ X , andlet us assume that π T is invariant for M = P ( T ) , that is, P ( T ) ′ π T = π T . One such class of transition matrices is given by P ( T, Q ) = ( p ij ( T, Q )) with p ij ( T, Q ) = q ij min (cid:16) exp (cid:16) E i − E j kT (cid:17) , (cid:17) , i = j, − P l,l = i q il min (cid:16) exp (cid:16) E i − E l kT (cid:17) , (cid:17) , i = j. , where Q = ( q ij ) is any symmetric transition matrix of anirreducible chain compatible with the topology of G . Let T eff < T be a desired lower temperature. Given any prior,such as the Ruelle-Bowen measure [11] or the invariant pathspace measure P ( T ) with transition P ( T ) and marginals
2. As is well-known, this (Metropolis) chain is actually reversiblewith respect to the Boltzmann distribution. π T and a sufficiently long time interval [0 , N ] , we canuse a standard Schr¨odinger bridge (see Section 2) to steeroptimally the network flow from any initial distribution ν to ν N = π T eff at time N . At time N , however, weneed to change the transition mechanism to keep the chainin the steady state π T eff . This is accomplished in the nextsubsection using the results of Section 3. Consider once again the case where the prior transition M = P ( T ) remains constant over time and has the Boltzmanndistribution (25) as initial, invariant measure. Consider theequivalent Problems 6 and 7. Theorem 13.
Assume that P ( T ) is fully indecomposable.Then, the solution to Problem 7 is given by Π ∗ = Diag( ϕ ) − P ( T )Diag( ϕ ) . (26)where ϕ = P ( T ) ϕ , (27a) ˆ ϕ = P ( T ) ′ ˆ ϕ , (27b) ϕ ◦ ˆ ϕ = ϕ ◦ ˆ ϕ = π T eff . (27c)As earlier, ◦ denotes componentwise multiplication ofvectors. Moreover, if P ( T ) is reversible with respect to π T , so is Π ∗ with respect to π T eff . Proof 4.
The result follows from Theorem 4, Proposition 10and Theorem 12.
ONCLUSION
The key point of this paper has been to explore analogues,in the setting of discrete space and time, for Markovianevolutions that match specified marginals and echo well-known results in the continuous time setting for Sch¨odingerbridges. Specifically, a key result is to show reversibilityof the transition rates under the assumption that this isa property of the prior. We consider both, transitioningbetween marginals over a specified time window as wellas the problem to secure a stationary distribution. Bothcases are accomplished by suitably adjusting the transitionrates from a given prior, while keeping the distance of thenew law closest to that corresponding to the prior in therelative entropy sense. Application of the results in thecontext regulating the flow of resources over a network iscontemplated. A notion of temperature when distributionsare expressed as Boltzmann distributions with respect to anenergy function is brought up, and the problem to transitionbetween distributions corresponding to different tempera-tures considered, in analogy to the problem of cooling incontinuous time and space. A PPENDIX P ROOF OF T HEOREM We form the Lagrangian for Problem 3: L ( p, π ; λ ) = N − X t =0 X x t D ( π x t x t +1 ( t ) k m x t x t +1 ( t )) p t ( x t )+ N − X t =0 X x t +1 λ t ( x t +1 ) " p t +1 ( x t +1 ) − X x t p t ( x t ) π x t x t +1 ( t ) OPTIMAL STEERING TO INVARIANT DISTRIBUTIONS FOR NETWORKS FLOWS
We now use discrete integration by parts: N − X t =0 X x t +1 λ t ( x t +1 ) p t +1 ( x t +1 )= N − X t =0 X x t λ t − ( x t ) p t ( x t )+ X x N λ N − ( x N ) ν N ( x N ) − X x λ − ( x ) ν ( x ) . Thus, the Lagrangian reads L ( p, π ; λ ) = N − X t =0 X x t D ( π x t x t +1 ( t ) k m x t x t +1 ( t )) p t ( x t )+ N − X t =0 X x t λ t − ( x t ) − X x t +1 λ t ( x t +1 ) π x t x t +1 ( t ) p t ( x t )+ X x N λ N − ( x N ) ν N ( x N ) − X x λ − ( x ) ν ( x ) . Observe that the last two terms are invariant over pairs ( p, π ) satisfying (9)-(11) and can therefore be discarded.Since we are minimizing over pairs satisfying the non-negativity requirement and with π satisfying (10), we canmultiply λ t − ( x t ) by P x t +1 π x t x t +1 ( t ) and subtract N = P N − t =0 P x t P x t +1 π x t x t +1 ( t ) p t ( x t ) to get L ( p, π ; λ ) = N − X t =0 X x t D ( π x t x t +1 ( t ) k m x t x t +1 ( t )) p t ( x t )+ N − X t =0 X x t X x t +1 π x t x t +1 ( t ) [ λ t − ( x t ) − λ t ( x t +1 )] p t ( x t ) − N − X t =0 X x t X x t +1 π x t x t +1 ( t ) p t ( x t ) . Next, we perform two-stage unconstrained minimiza-tion of the Lagrangian: For fixed flow p = { p t } (andfixed multipliers), we optimize the strictly convex function L ( p, · ; λ ) at each time t . We get the optimality conditions: h π ∗ x t x t +1 ( t ) − log m x t x t +1 ( t )+ λ t − ( x t ) − λ t ( x t +1 ) − p t ( x t ) = 0 , or π ∗ x t x t +1 ( t ) = m x t x t +1 ( t ) exp [ λ t ( x t +1 ) − λ t − ( x t )] . (28)Define the positive function ϕ ( t + 1 , x t +1 ) = exp [ λ t ( x t +1 )] . Then (28) becomes π ∗ x t x t +1 ( t ) = m x t x t +1 ( t ) ϕ ( t + 1 , x t +1 ) ϕ ( t, x t ) . Let us impose condition (10) on π ∗ . We get that π ∗ satisfiessuch condition if and only if ϕ satisfies the reverse-timerecursion ϕ ( t, x t ) = X x t +1 m x t x t +1 ( t ) ϕ ( t + 1 , x t +1 ) , ∀ x t ∈ X , t = 0 , , . . . , N − . Observe now that L ( p, π ∗ ; log ϕ ) = N − X t =0 X x t X x t +1 (cid:20) − m x t x t +1 ( t ) ϕ ( t + 1 , x t +1 ) ϕ ( t, x t ) (cid:21) p t ( x t )= − N − X t =0 X x t p t ( x t ) = − N, which is independent of p . Thus, we can choose p ∗ definedby p ∗ t +1 ( x t +1 ) = X x t p ∗ t ( x t ) π ∗ x t x t +1 ( t ) , p ( x ) = ν ( x ) , so as to satisfy constraint (9) and the first of (11). Define ˆ ϕ ( t, x t ) := p ∗ t ( x t ) ϕ ( t, x t ) . We get ˆ ϕ ( t + 1 , x t +1 ) = p ∗ t +1 ( x t +1 ) ϕ ( t + 1 , x t +1 )= P x t p ∗ t ( x t ) π ∗ x t x t +1 ( t ) ϕ ( t + 1 , x t +1 )= P x t p ∗ t ( x t ) m x t x t +1 ( t ) ϕ ( t +1 ,x t +1 ) ϕ ( t,x t ) ϕ ( t + 1 , x t +1 )= X x t m x t x t +1 ( t ) p ∗ t ( x t ) ϕ ( t, x t )= X x t m x t x t +1 ( t ) ˆ ϕ ( t, i ) , namely ˆ ϕ ( t, x t ) is space-time co-harmonic . A CKNOWLEDGMENTS
Supported in part by the NSF under grant 1807664, 1839441,1901599, 1942523, the AFOSR under grants FA9550-17-1-0435, and by the University of Padova Research ProjectCPDA 140897. R EFERENCES [1] J. Benamou and Y. Brenier, A computational fluid mechanics solu-tion to the Monge-Kantorovich mass transfer problem,
NumerischeMathematik , , no. 3, 2000, 375-393.[2] A. Beurling, An automorphism of product measures, Ann. Math. (1960), 189-200.[3] D. Bertsekas, Dynamic Programming and Optimal Control , vol 2. MA:Athena, 1995.[4] Y. Braiman, J. Barhen, and V. Protopopescu, Control of Friction atthe Nanoscale,
Phys. Rev. Lett. , (2003), 094301.[5] Y. Chen, T.T. Georgiou and M. Pavon, Fast cooling for a system ofstochastic oscillators, J. Math. Phys. , J. Optim. Theory and Applic. , SIAM Journal on Applied Mathematics , (6), 2375-2396, 2016.[8] Y. Chen, T.T. Georgiou and M. Pavon, Stochastic control liaisons:Richard Sinkhorn meets Gaspard Monge on a Schr¨odinger bridge,ArXiv e-prints, arXiv: 2005.10963, SIAM Review , to appear.[9] Y. Chen, T.T. Georgiou and M. Pavon, Optimal Trans-port in Systems and Control,
Annual Review of Control,Robotics, and Autonomous Systems , May 2021. HEN, GEORGIOU, AND PAVON 7 [10] Y. Chen, T.T. Georgiou and M. Pavon, Controlling Uncertainty:Schr¨odinger’s Inference Method and the Optimal Steering of Prob-ability Distributions, June 2020,
IEEE Control Systems Magazine , toappear.[11] Y. Chen, T.T. Georgiou, M. Pavon and A. Tannenbaum, Robusttransport over networks,
IEEE Trans. Aut. Control , , n.9, 4675-4682,2017.[12] Y. Chen, T.T. Georgiou, M. Pavon and A. Tannenbaum, Efficient-robust routing for single commodity network flows, IEEE Trans.Aut. Control , IEEE Trans. onControl of Network Systems , Advances in Neural Information Processing Systems ,2292-2300, 2013.[15] J.-C. Delvenne and A.-S. Libert, Centrality measures and thermo-dynamic formalism for complex networks,
Physical Review E , (4),046117, 2011.[16] W. E. Deming and F. F. Stephan, On a least squares adjustment ofa sampled frequency table when the expected marginal totals areknown, Ann. Math. Statist. , (4), (1940), 427-444.[17] M. Doi and S. F. Edwards, The Theory of Polymer Dynamics , OxfordUniversity Press, New York, 1988.[18] T. I. Fenner and G. Loizou, On fully indecomposable matrices,
J.Computer and System Sciences , , 1971, 607-622.[19] H. F¨ollmer, Random fields and diffusion processes, in: `Ecole d’ `Et`ede Probabilit`es de Saint-Flour XV-XVII , edited by P. L. Hennequin,Lecture Notes in Mathematics, Springer-Verlag, New York, 1988,vol.1362,102-203.[20] R. Fortet, R´esolution d’un syst`eme d’equations de M. Schr¨odinger, J. Math. Pure Appl.
IX (1940), 83-105.[21] T. T. Georgiou and M. Pavon, Positive contraction mappings forclassical and quantum Schr¨odinger systems,
J. Math. Phys. , ,033301 (2015).[22] B. Jamison, The Markov processes of Schr¨odinger, Z. Wahrschein-lichkeitstheorie verw. Gebiete (1975), 323-331.[23] C. L´eonard, A survey of the Schroedinger problem and some ofits connections with optimal transport, Discrete Contin. Dyn. Syst.A , 2014, (4): 1533-1574.[24] S. Liang, D. Medich, D. M. Czajkowsky, S. Sheng, J. Yuan, and Z.Shao, Ultramicroscopy , (2000), p.119.[25] F. Marquardt and S. M. Girvin, Optomechanics, Physics 2 , (2009).[26] A. W. Marshall and I. Olkin, Scaling of matrices to achieve speci-fied row and column sums, Numerische Mathematik , , 1968, 83-90.[27] G. Monge, M´emoire sur la th´eorie des d´eblais et des remblais, His-toire de l’Academie royale des sciences avec les m´emoires de math´ematiqueet de physique tir´es des registres de cette Acad´emie , De l’ImprimerieRoyale, 1781, 666-704.[28] M. Pavon and F. Ticozzi, Discrete-time classical and quantumMarkovian evolutions: Maximum entropy problems on path space,
J. Math. Phys. , , 042104-042125 (2010).[29] G. Peyr´e and M. Cuturi, Computational Optimal Transport, Foun-dations and Trends in Machine Learning , , no. 5-6,1-257, 2019.[30] M. L. Puterman, Markov Decision Processes , Wiley, 1994.[31] E. Schr¨odinger, ¨Uber die Umkehrung der Naturgesetze,
Sitzungs-berichte der Preuss Akad. Wissen. Berlin, Phys. Math. Klasse , , 144-153, 1931.[32] E. Schr¨odinger, Sur la th´eorie relativiste de l’´electron etl’interpretation de la m´ecanique quantique, Ann. Inst. H. Poincar´e , 269 (1932).[33] K. C. Schwab and M. L. Roukes, Phys. Today
58, No. 7, 36 (2005).[34] R. Sinkhorn, A relationship between arbitrary positive matricesand doubly stochastic matrices,
Ann. Math. Statist. , (2)(1964), 876-879.[35] R. S. Varga, Matrix Iterative Analysis , Prentice-Hall, 1962.[36] A. Vinante, M. Bignotto, M. Bonaldi et al. , Feedback Cooling of theNormal Modes of a Massive Electromechanical System to Submil-likelvin Temperature,
Physical Review Letters (2008), 033601.[37] A. Wakolbinger, Schroedinger bridges from 1931 to 1991, in
Proc.of the 4th Latin American Congress in Probability and MathematicalStatistics , Mexico City 1990,
Contribuciones en probabilidad y estadis-tica matematica ,3