Flora M. Spieksma

Contraction Conditions for Average and α-Discount Optimality in Countable State Markov Games with Unbounded Rewards

The goal of this paper is to provide a theory of N-person Markov games with unbounded cost, for a countable state space and compact action spaces. We investigate both the finite and infinite horizon problems. For the latter, we consider the discounted cost as well as the expected average cost. We present conditions for the infinite horizon problems for which equilibrium policies exist for all players within the stationary policies, and show that the costs in equilibrium policies exist for all players within the stationary policies, and show that the costs in equilibrium satisfy the optimality equations. Similar results are obtained for the finite horizon costs, for which equilibrium policies are shown to exist for all players within the Markov policies. As special case of N-person games, we investigate the zero-sum 2 players game, for which we establish the convergence of the value iteration algorithm. We conclude by studying an application of a zero-sum Markov game in a queueing model.

On the relation between recurrence and ergodicity properties in denumerable Markov decision chains

This paper studies two properties of the set of Markov chains induced by the deterministic policies in a Markov decision chain. These properties are called µ-uniform geometric ergodicity and µ-uniform geometric recurrence. µ-uniform ergodicity generalises a quasi-compactness condition. It can be interpreted as a strong version of stability, as it implies that the Markov chains generated by the deterministic stationary policies are uniformly stable. µ-uniform geometric recurrence can be shown to be equivalent to the simultaneous Doeblin condition, If µ is bounded. Both properties imply the existence of deterministic average and sensitive optimal policies. The second Key theorem in this paper shows the equivalence of µ-uniform geometric ergodicity and weak µ-uniform geometric recurrence under appropriate continuity conditions. In the literature numerous recurrence conditions have been used. The first Key theorem derives the relation between several of these conditions, which interestingly turn out to be equivalent in most cases.

Strengthening ergodicity to geometric ergodicity for markov chains

In this paper we find conditions under which ergodic Markov chains are also geometrically ergodic: that is, converge to their limits geometrically quickly. We show that if the increment distributions of the chain have uniform exponential tails in an appropriate sense, then the stronger geometric ergodicity hold, whilst if the stationary measure π of the chain has suitably exponential tails then again geometric ergodicity holds but under further auxiliary conditions. We give examples to show that, in particular, π may have geometric tails but the chain need not be geometrically ergodic. We conclude with a number of examples from queueing and network theory covered by the results, indicating the use of the results when there is a known Foster-Lyapunov function and also when the hitting times of finite sets are merely

ON DEVIATION MATRICES FOR BIRTH–DEATH PROCESSES

We study deviation matrices of birth–death processes. This is relevant to the control of multidimensional queueing systems. We give an algorithm for computing deviation matrices for birth–death processes. As an application, we compute them explicitly for the M/M/s/N and M/M/s/∞ queues.

On structural properties of the value function for an unbounded jump Markov process with an application to a processor sharing retrial queue

The derivation of structural properties for unbounded jump Markov processes cannot be done using standard mathematical tools, since the analysis is hindered due to the fact that the system is not uniformizable. We present a promising technique, a smoothed rate truncation method, to overcome the limitations of standard techniques and allow for the derivation of structural properties. We introduce this technique by application to a processor sharing queue with impatient customers that can retry if they renege. We are interested in structural properties of the value function of the system as a function of the arrival rate.

Kolmogorov forward equation and explosiveness in countable state Markov processes

In countable state non-explosive minimal Markov processes the Kolmogorov forward equations hold under sufficiently weak conditions. However, a precise description of the functions that one may integrate with respect to these equations seems to be absent in the literature. This problem arises for instance when studying the Poisson equation, as well as the average cost optimality equation in a Markov decision process.We will show that the class of non-negative functions for which an associated transformed Markov process is non-explosive do have this desirable property. This characterisation easily allows to construct counter-examples of functions for which the functional form of the Kolmogorov forward equations does not hold.Another approach of the problem is to study the transition operator as a transition semi-group on Banach space. The domain of the generator is a collection of functions that can be integrated with respect to the Kolmogorov forward equations. We focus on Banach spaces equipped with a weighted supremum norm, and we identify subsets of the domain of the generator.

COUNTABLE STATE MARKOV PROCESSES: NON-EXPLOSIVENESS AND MOMENT FUNCTION

The existence of a moment function satisfying a drift function condition is well known to guarantee non-explosiveness of the associated minimal Markov process (cf. [1,9]), under standard technical conditions. Surprisingly, the reverse is true as well for a countable space Markov process. We prove this result by showing that recurrence of an associated jump process, that we call the α-jump process, is equivalent to non-explosiveness. Non-explosiveness corresponds in a natural way to the validity of the Kolmogorov integral relation for the function identically equal to 1. In particular, we show that positive recurrence of the α-jump chain implies that all bounded functions satisfy the Kolmogorov integral relation. We present a drift function criterion characterizing positive recurrence of this α-jump chain. Suppose that to a drift function V there corresponds another drift function W , which is a moment with respect to V . Via a transformation argument, the above relations hold for the transformed process with respect to V . Transferring the results back to the original process, allows to characterize the V -bounded functions that satisfy the Kolmogorov forward equation.

On the uniqueness of solutions to the Poisson equations for average cost Markov chains with unbounded cost functions

AbstractWe consider the Poisson equations for denumerable Markov chains with unbounded cost functions. Solutions to the Poisson equations exist in the Banach space of bounded real-valued functions with respect to a weighted supremum norm such that the Markov chain is geometrically ergodic. Under minor additional assumptions the solution is also unique. We give a novel probabilistic proof of this fact using relations between ergodicity and recurrence. The expressions involved in the Poisson equations have many solutions in general. However, the solution that has a finite norm with respect to the weighted supremum norm is the unique solution to the Poisson equations. We illustrate how to determine this solution by considering three queueing examples: a multi-server queue, two independent single server queues, and a priority queue with dependence between the queues.

Countable state Markov decision processes with unbounded jump rates and discounted cost:optimality equation and approximations

This paper considers Markov decision processes (MDPs) with unbounded rates, as a function of state. We are especially interested in studying structural properties of optimal policies and the value function. A common method to derive such properties is by value iteration applied to the uniformised MDP. However, due to the unboundedness of the rates, uniformisation is not possible, and so value iteration cannot be applied in the way we need. To circumvent this, one can perturb the MDP. Then we need two results for the perturbed sequence of MDPs: 1. there exists a unique solution to the discounted cost optimality equation for each perturbation as well as for the original MDP; 2. if the perturbed sequence of MDPs converges in a suitable manner then the associated optimal policies and the value function should converge as well. We can model both the MDP and perturbed MDPs as a collection of parametrised Markov processes. Then both of the results above are essentially implied by certain continuity properties of the process as a function of the parameter. In this paper we deduce tight verifiable conditions that imply the necessary continuity properties. The most important of these conditions are drift conditions that are strongly related to nonexplosiveness.

Aspects of the Cooperative Card Game Hanabi

We examine the cooperative card game Hanabi. Players can only see the cards of the other players, but not their own. Using hints partial information can be revealed. We show some combinatorial properties, and develop AI (Artificial Intelligence) players that use rule-based and Monte Carlo methods.