Karl Hinderer

Cash management in a randomly varying environment

We study a cash management system, in which the distribution of the cash flow Xn in period n=1,2,… depends on the state In of a randomly varying environment. Sufficient conditions are found for the optimality of a simple transfer rule, generalizing and partially improving the well-known results for the classical case with i.i.d. cash flows. These and further structural results obtained for the cash balance are shown to reduce the computational effort drastically in determining an optimal transfer rule. In addition, structural and computational results w.r.t. the environment are derived. Finally, some examples are given for economic and statistical environments and their interactions with the cash flow process.

Algorithms for Countable State Markov Decision Models with an Absorbing Set

We consider a countable state Markov decision process with bounded reward function and an absorbing set. At first we generalize known properties and derive new properties of the critical discount factor, which is roughly defined as the maximal discount factor under which for V, the maximal expected infinite-stage discounted reward, there is guaranteed existence, boundedness, and computability by the successive approximation method. The emphasis of the paper is on algorithms for computing V exactly (recursion in state space and policy iteration) or approximately (value iteration combined with an extrapolation and finite state approximation). Our extrapolation method is motivated by and based on the Perron--Frobenius theory for nonlinear operators. As a by-product we obtain an efficient algorithm for determining the distribution of the entrance time of a Markov chain into an absorbing set. Further results concern asymptotically

Lipschitz Continuity of Value Functions in Markovian Decision Processes

\epsilon

The critical discount factor for finite Markovian decision processes with an absorbing set

-optimal policies and a new turnpike theorem. Some of the results need tightness of the transition law, which turns out to be equivalent to compactness of a nonlinear operator, which is crucial for our study.

On polychotomous search problems

We present tools and guidelines for investigating Lipschitz continuity of the value functions in MDP’s, using the Hausdorff metric and the Kantorovich metric for measuring the influence of the constraint set and the transition law, respectively. The methods are explained by examples. Additional topics include an application to the the discretization algorithm of Bertsekas (1975).

An Improvement of J. F. Shapiro’s Turnpike Theorem for the Horizon of Finite Stage Discrete Dynamic Programs

Abstract. This paper deals with a Markovian decision process with an absorbing set J0. We are interested in the largest number β*≥1, called the critical discount factor, such that for all discount factors β smaller than β* the limit V of the N-stage value function VN for N →∞ exists and is finite for each choice of the one-stage reward function. Several representations of β* are given. The equality of 1/β* with the maximal Perron/Frobenius eigenvalue of the MDP links our problem and our results to topics studied intensively (mostly for β=1) in the literature. We derive in a unified way a large number of conditions, some of which are known, which are equivalent either to β1. In particular, the latter is equivalent to transience of the MDP. A few of our findings are extended with the aid of results in Rieder (1976) to models with standard Borel state and action space. We also complement an algorithm of policy iteration type, due to Mandl/Seneta (1969), for the computation of β*. Finally we determine β* explicitly in two models with stochastically monotone transition law.

Increasing and Lipschitz continuous minimizers in one-dimensional linear-convex systems without constraints: The continuous and the discrete case

Abstract The paper gives a unified treatment of those stochastic sequential search problems where in each state the searcher may decompose according to a given scheme the current search region in a finite number of parts upon which one obtains the error-free information in which of the parts the object is hidden. Many examples are given to show the generality of the model. The main results are: (a) a rigorous proof of the dynamic programming approach, (b) the reduction of the state space for problems with translation invariant data, (c) a dynamic programming proof for the computationally very attractive property of monotonicity and Lipschitz continuity of optimal search rules for two important special models, one of which has been investigated by Hassin (1984) using methods from information theory.

On the k-armed Bernoulli bandit: monotonicity of the total reward under an arbitrary prior distribution

The potential usefulness of the well-known turnpike theorem in [12] is for practical purposes rather limited, since numerical evaluation of the estimate given in [12] for the turnpike horizon N* gives poor results, unless the discount factor β is rather small. We contribute to the problem in the following directions: (i) Considerably improved estimates for N* are derived, including cases where β ≧ 1. (ii) For dynamic programs with absorbing sets further improvement is derived. (iii) Some numerical examples are given, including a rather detailed analysis of Howard’s toymaker example for arbitrary values of β.

The integrated distribution function and its application for some inventory problems with linear demand pattern, arbitrary demand distribution and without fixed ordering costs

We consider a stochastic control model with linear transition law and arbitrary convex cost functions, a far-reaching generalization of the familiar linear quadratic model. Firstly conditions are given under which the continuous state version has minimizersfn at each stagen which are increasing and in addition either right continuous or continuous or Lipschitz continuous with explicitly given Lipschitz constant. For the computationally important discrete version we verify some analogous properties under stronger assumptions.

Approximate Solution of Markov Renewal Programs with Finite Time Horizon

We investigate monotonicity properties of the success probabilities and the total reward when the number of previously observed successes and failures change. Using a well-known Bayesian approach and dynamic programming we give conditions in terms of the covariances of the posterior distributions and in terms of the support of the prior distribution. Special order relations for the number of successes and failures allow a simple and unified treatment of different cases. The results extend some of the investigations of Hengartner/Kalin/Theodorescu [1].