J. Laurie Snell

Finite Markov chains

Markov chains are one of the most useful classes of stochastic processes, being • simple, flexible and supported by many elegant theoretical results • valuable for building intuition about random dynamic models • central to quantitative modeling in their own right You will find them in many of the workhorse models of economics and finance. In this lecture we review some of the theory of Markov chains. We will also introduce some of the high quality routines for working with Markov chains available in QuantEcon.jl. Prerequisite knowledge is basic probability and linear algebra.

Random walks and electric networks

Probability theory, like much of mathematics, is indebted to physics as a source of problems and intuition for solving these problems. Unfortunately, the level of abstraction of current mathematics often makes it difficult for anyone but an expert to appreciate this fact. In this work we will look at the interplay of physics and mathematics in terms of an example where the mathematics involved is at the college level. The example is the relation between elementary electric network theory and random walks. Central to the work will be Polya’s beautiful theorem that a random walker on an infinite street network in d-dimensional space is bound to return to the starting point when d = 2, but has a positive probability of escaping to infinity without returning to the starting point when d ≥ 3. Our goal will be to interpret this theorem as a statement about electric networks, and then to prove the theorem using techniques from classical electrical theory. The techniques referred to go back to Lord Rayleigh, who introduced them in connection with an investigation of musical instruments. The analog of Polya’s theorem in this connection is that wind instruments are possible in our three-dimensional world, but are not possible in Flatland (Abbott [1]). The connection between random walks and electric networks has been recognized for some time (see Kakutani [12], Kemeny, Snell, and

Mathematical models in the social sciences

As the need for more substantial mathematical training has increased among social science students, the lack of any adequate textbook between the very elementary and the very advanced levels has become crutial. The authors, long-time experts in this field, have answered the need with this volume, and the MIT Press has repsonded by bringing it into renewed circulation.Mathematical Models in the Social Sciences investigates and teaches the formation and analysis of mathematical models with detailed interpretations of the results. These models are self-contained, with the necessary mathematics included in each chapter. A vast range of topics in the social sciences and a wide variety of mathematical techniques are covered by the models. Ample opportunity is also provided for the students to form their own models. Republication of this book provides social science and mathematics students with a text that is the analogue of mathematical methods textbooks used in the study of the physical sciences and engineering. Prerequisites are kept to a minimum; a course in finite mathematics and a semester of calculus are all that is necessary.The chapters cover these main topics (and employ the mathematical approach parenthetically indicated): methodology; preference rankings (an axiomatic approach); ecology (two dynamic models); market stability (a dynamic model); a Markov chain model in sociology; stabilization of money flow (an application of discrete potential theory); branching processes; organization theory (applications of graph theory); and optimal scheduling (a problem in dynamic programming).

Potentials for denumerable Markov chains

A unified treatment of potentials for general Markov chains is presented. Basic properties of potentials associated with transient chains are reviewed; these behave like Newtonian potentials. Potentials for recurrent chains are developed; they behave like logarithmic potentials. The discrete analog of the Newtonian potential classical theory is obtained by applying these results to random walls in 3 dimensions. Since the results are applicable to Markov chains in general, discrete versions of the more general theory are obtained. (L.N.N.)

On the relation between Markov random fields and social networks

Holland and Leinhardt (1977a) introduced a continuous time Markov chain to model changes in a social network. Their model was further studied by Wasserman (1977). Holland and Leinhardt (1977b) have also proposed a general class of probability measures on random networks. The purpose of this paper is to show that these probability measures may be viewed as Gibbs measures induced by a nearest neighbor potential. As such they have a Markov field property which is a natural generalization of Markov chains to spatial situations. The dynamic models are shown to fit into a general class of Markov processes suggested by the study of interacting particle systems. A related “voter model” is discussed.

Finite Continuous Time Markov Chains

In a recent book by the authors (see footnote on page 101) a systematic method was developed for computing the basic descriptive quantities of finite Markov chains. Matrix expressions were obtained for these quantities in terms of certain basic matrices, easily obtainable from the transition matrix.In this note, corresponding expressions are given for the basic quantities for finite continuous time chains.

Game-Theoretic Solution of Baccarat

The famous gambling game of baccarat has been the subject of many mathematical studies. There is, however, considerable variation in the solutions offered and it is now clear that these variations have come from trying to solve a game of strategy without a precise meaning for the solution of such a game. The theory of games has made this precise; it is now interesting to carry out the solution of baccarat in terms of the modern concepts of game theory and to compare the solution obtained with the earlier attempts based upon the more vague ideas previously available. Baccarat is usually played by three men, one banker playing against a pair of players. But there is a popular variant of the game, known as chemin de fer, in which the banker plays against a single player. This is the version commonly discussed in the literature, and we too will take this two-person version of baccarat as our subject.

Markov processes in learning theory

Consideration is given mathematical problems arising in two learning theories—one developed by Bush and Mosteller, the other developed by Estes. The theory of Bush and Mosteller leads to a class of Markov processes which have been studied in considerable detail (see [1] and [7]). The Estes model can be treated as a Markov chain, i.e., a Markov process with a finite number of states. For an important class of special cases, it is shown that the Bush-Mosteller model is, in a sense, a limiting form of the Estes model. The limiting probability distributions are derived for the cases treated in both models.

Properties of Markov Chains

During all of our discussion of Markov chains, we shall wish to confine ourselves to stochastic processes defined on a sequence space. We have shown that an arbitrary stochastic process may be considered as a process on a suitable Ω in which the outcome functions f n are coordinate functions. We see, therefore, that in a sense no generality is lost by discussing Markov chains in terms of sequence space.

NOTES ON DISCRETE POTENTIAL THEORY

A concrete example of a null chain for which the limits are not abel- summable is presented. It is the combination of two rooted trees, having 0 and 1 as roots, respectively. (L.N.N.)