Lodewijk C. M. Kallenberg

Variance-penalized Markov decision processes

We consider a Markov decision process with both the expected limiting average, and the discounted total return criteria, appropriately modified to include a penalty for the variability in the stream of rewards. In both cases we formulate appropriate nonlinear programs in the space of state-action frequencies averaged, or discounted whose optimal solutions are shown to be related to the optimal policies in the corresponding “variance-penalized MDP.” The analysis of one of the discounted cases is facilitated by the introduction of a “Cartesian product of two independent MDPs.”

Constrained Undiscounted Stochastic Dynamic Programming

In this paper we investigate the computation of optimal policies in constrained discrete stochastic dynamic programming with the average reward as utility function. The state-space and action-sets are assumed to be finite. Constraints which are linear functions of the state-action frequencies are allowed. In the general multichain case, an optimal policy will be a randomized nonstationary policy. An algorithm to compute such an optimal policy is presented. Furthermore, sufficient conditions for optimal policies to be stationary are derived. There are many applications for constrained undiscounted stochastic dynamic programming, e.g., in multiple objective Markovian decision models.

Survey of Linear Programming for Standard and Nonstandard Markovian Control Problems. Part I: Theory

This paper gives an overview of linear programming methods for solving standard and nonstandard Markovian control problems. Standard problems are problems with the usual criteria such as expected total (discounted) rewards and average expected rewards; we also discuss a particular class of stochastic games. In nonstandard problems there are additional considerations as side constraints, multiple criteria or mean-variance tradeoffs. In a second companion paper efficient linear programing algorithms are discussed for some applications.

Finite State and Action MDPS

In this chapter we study Markov decision processes (MDPs) with finite state and action spaces. This is the classical theory developed since the end of the fifties. We consider finite and infinite horizon models. For the finite horizon model the utility function of the total expected reward is commonly used. For the infinite horizon the utility function is less obvious. We consider several criteria: total discounted expected reward, average expected reward and more sensitive optimality criteria including the Blackwell optimality criterion. We end with a variety of other subjects.

On finding optimal policies for Markov decision chains: a unifying framework for mean-variance-tradeoffs

In a photoresist material comprising a support having formed thereon a layer of a photopolymerizable composition comprising a film forming polymer, a monomer having at least one addition polymerizable unsaturated bond, and a photopolymerization initiator, the adhesive property and the light-sensitivity of the layer of the photopolymerizable composition are improved by incorporating a rosin tackifier in the layer.

Transient policies in discrete dynamic programming: Linear programming including suboptimality tests and additional constraints

This paper investigates the computation of transient-optimal policies in discrete dynamic programming. The model, is quite general: it may contain transient as well as nontransient policies. and the transition matrices are not necessarily substochastic.A functional equation for the so-called transient-value-vector is derived and the concept of superharmonicity is introduced. This concept provides the linear program to compute the transientvalue-vector and a transient-optimal policy.We also discuss the elimination of suboptimal actions, the solution of problems with additional constraints, and the computation of an efficient policy for a multiple objective dynamic programming problem.

Sensitivity Analysis in Discounted Markovian Decision Problems

At discrete time points t = 1, 2 . . . . a system is observed by a decision maker in one of the states of a finite statespace E = {1, 2 . . . . . N}. If, at time point t, the system is observed in state i, the decision maker controls the system by choosing an action from a finite action setA(i), which is independent of t. If the decision maker chooses action a in state i, then the following happens independently of the history of the process:

A Note on M. N. Katehakis' and Y.-R. Chen's Computation of the Gittins Index

In a recent paper Katehakis and Chen propose a sequence of linear programs for the computation of the Gittins indices. If there are N projects and project v has Kv states, then Σv=1NKv linear programs have to be solved. In this note it is shown that instead of the Kv linear programs for project v also one parametric linear program with the same dimensions can be solved.

On the value function in constrained control of Markov chains

It is known that the value function in an unconstrained Markov decision process with finitely many states and actions is a piecewise rational function in the discount factor a, and that the value function can be expressed as a Laurent series expansion about α = 1 for α close enough to 1. We show in this paper that this property also holds for the value function of Markov decision processes with additional constraints. More precisely, we show by a constructive proof that there are numbers O = αo <α1 <... < αm−1 < αm = 1 such that for everyj = 1, 2, ...,m − 1 either the problem is not feasible for all discount factors α in the open interval (αj−1, αj) or the value function is a rational function in a in the closed interval [αj−1, αj]. As a consequence, if the constrained problem is feasible in the neighborhood of α = 1, then the value function has a Laurent series expansion about α = 1. Our proof technique for the constrained case provides also a new proof for the unconstrained case.

Maintenance and Markov Decision Models

In this chapter we first give an introduction to Markov decision theory. We state the main optimality criteria and solution approaches. Next we sketch how it can be applied in maintenance theory. In particular, we deal with the civil infrastructure sector and show what kind of results it brings. Finally we also indicate which problems arise in applications. Keywords: Markov decision process; maintenance; replacement; policy optimization; aging; civil structures