Isaac M. Sonin

The Elimination algorithm for the problem of optimal stopping

Abstract. We present a new algorithm for solving the optimal stopping problem. The algorithm is based on the idea of elimination of states where stopping is nonoptimal and the corresponding correction of transition probabilities. The formal justification of this method is given by one of two presented theorems. The other theorem describes the situation when an aggregation of states is possible in the optimal stopping problem.

The Optimal Stopping of a Markov Chain and Recursive Solution of Poisson and Bellman Equations

We discuss a modified version of the Elimination algorithm proposed earlier by the author to solve recursively a problem of optimal stopping of a Markov chain in discrete time and finite or countable state space. This algorithm and the idea behind it are applied to solve recursively the discrete versions of the Poisson and Bellman equations. To Albert N. Shiryaeyev-one of the pioneers of Optimal Stopping, with appreciation and gratitude, on the occasion of his 70th birthday. 20 Aug

Notes on equivalent stationary policies in Markov Decision Processes with total rewards

We construct examples of Markov Decision Processes for which, for a given initial state and for a given nonstationary transient policy, there is no equivalent (randomized) stationary policy, i.e. there is no stationary policy which occupation measure is equal to the occupation measure of a given policy. We also investigate the relation between the existence of equivalent stationary policies in special models and the existence of equivalent strategies in various classes of nonstationary policies in general models.

Optimal stopping of Markov chains and three abstract optimization problems

There is a well-known connection between the three problems related to the optimal stopping of Markov chains and the equality of three corresponding indices: the classical Gittins index (GI) in the ratio maximization problem, the Kathehakis–Veinott index in a restart problem and Whittle index in a family of retirement problems. In Sonin [Statist. Probab. Lett. 78 (12,1) (2008), pp. 1526–1533], these three problems and these three indices were generalized in such a way that it became possible to use the state elimination algorithm [Sonin, Math. Meth of Oper. Res. (1999), pp. 111–123] to calculate this common generalized GI α. The main goal of this note is to demonstrate that the equality of these (generalized) indices is a special case of a more general relation between three simple abstract optimization problems.

The Decomposition-Separation Theorem for Finite Nonhomogeneous Markov Chains and Related Problems

Let M be a finite set, P be a stochastic matrix and U = f(Zn)g be the family of all finite Markov chains (MC) (Zn) defined by M;P, and all possible initial distributions. The behavior of a MC (Zn) is a classical result of Probability Theory derived in the 30s by A. Kolmogorov and W. Doeblin. If a stochastic matrix P is replaced by a sequence of stochastic matrices (Pn) and transitions at moment n are defined by Pn, then U becomes a family of nonhomogeneous MCs. There are numerous results concerning the behavior of such MCs given some specific properties of the sequence (Pn): But what if there are no assumptions about sequence (Pn)? Is it possible to say something about the behavior of the family U ? The surprising answer to this question is Yes. Such behavior is described by a theorem which we call a Decomposition- Separation (DS) Theorem, and which was initiated by a small paper of A. N. Kolmogorov (1936) and formulated and proved in a few stages in a series of papers including: D. Blackwell (1945), H. Cohn (1971, 1989) and I. Sonin (1987, 1991, 1996).

Growth rate, internal rates of return and turnpikes in an investment model

SummaryThis paper describes the relationship between the models growth rate, the set of vectors of equilibrium growth and the set of internal rates of return of the investment matrix. This matrix specifies the renewable and reproducible scale-neutral investment possibilities. An explicit description of quasioptimal strategies and turnpikes is given.

Increasing the reliability of a machine reduces the period of its work

The comparison of optimal strategies in a simple stochastic replacement model for two types of machines with identical cost characteristics when one of them is more reliable than the other is conducted. It is proven that the scheduled replacement period for the more reliable type is always less than for the less reliable one. An example is presented when even the expected period of use of a more reliable machine is less than the expected period for the less reliable one. Some related problems are briefly discussed.

On Optimal Stopping of Random Sequences Modulated by Markov Chain

This paper has two main goals: first, to describe a new class of optimal stopping (OS) problems for which the solutions can be found either in an explicit form, or in a finite number of steps, and second, to demonstrate the potential of the state elimination algorithm developed by one of the authors earlier, for the problem of OS of a finite or countable Markov chain.

The Existence and Uniqueness of Nash Equilibrium Point in an m-player Game “Shoot Later, Shoot First!”

We consider the following “silent duel” of m players with a possible economic interpretation. Each player has one “bullet”, which she can shoot at any time during the time interval [0,1]. The probability that the i-th player hits the “target” at moment t is given by an increasing accuracy function fi(t). The winner is the player who hits the target first. Under natural assumptions on the functions fi(t) we prove the existence and uniqueness of a Nash equilibrium point in this game, and we provide an explicit construction of this equilibrium. This construction allows us to obtain exact solutions for many specific examples. Some of them are presented.

Continue, quit, restart probability model

We discuss a new applied probability model: there is a system whose evolution is described by a Markov chain (MC) with known transition matrix on a discrete state space and at each moment of a discrete time a decision maker can apply one of three possible actions: continue, quit, and restart MC in one of a finite number of fixed “restarting” points. Such a model is a generalization of a model due to Katehakis and Veinott (Math. Oper. Res. 12:262, 1987), where a restart to a unique point was allowed without any fee and quit action was absent. Both models are related to Gittins index and to another index defined in a Whittle family of stopping retirement problems. We propose a transparent recursive finite algorithm to solve our model by performing O(n3) operations.