Manfred Schäl

Average Optimality in Dynamic Programming with General State Space

A Markovian decision model with general state space, compact action space, and the average cost as criterion is considered. The existence of an optimal policy is shown via an optimality inequality in terms of the minimal average cost g and a relative value function w. The existence of some w is usually shown via relative compactness in a space of real-valued functions on the state space. Here it shall be shown that one can instead do with pointwise relative compactness in the set of real numbers if one makes use of a generalized lower limit of functions. An application to an inventory model is given.

On dynamic programming: Compactness of the space of policies

The compactness of the set of policies in a dynamic programming decision model, which guarantees the existence of an optimal policy, is proven by reducing the problem to the compactness of the set of probability measures which are induced by the policies. When studying the set of probability measures, use is made of the weak topology and the so-called ws[infinity]-topology. A definition and a discussion of the latter topology is given in this paper, where we pay attention to criteria for relative compactness.

On the Second Optimality Equation for Semi-Markov Decision Models

For a semi-Markov decision model with average return, the validity of the second optimality equation is shown in the nonmodified form where the actions run through the set of all admissible actions rather than through the set of maximum points conserving actions for the first optimal equation. As a consequence the existence of a strongly optimal stationary policy is shown. The results seem to be known only for finite state finite action models whereas here countable state compact action models with unbounded rewards are considered.

On Discrete-Time Dynamic Programming in Insurance: Exponential Utility and Minimizing the Ruin Probability

This paper studies an insurance model where the risk process can be controlled by reinsurance and by investment in a financial market. The performance criterion is either the expected exponential utility of the terminal surplus or the ruin probability. It is shown that the problems can be imbedded in the framework of discrete-time stochastic dynamic programming but with some special features. A short introduction to control theory with infinite state space is provided which avoids the measure-theoretic apparatus by use of the so-called structure assumption. Moreover, in order to treat models without discount factor, a weak contraction property is derived. Explicit conditions are obtained for the optimality of employing no reinsurance.

On piecewise deterministic Markov control processes: Control of jumps and of risk processes in insurance

Abstract Dynamic programming for piecewise deterministic Markov processes is studied where only the jumps but not the deterministic flow can be controlled. Then one can dispense with relaxed controls. There exists an optimal stationary policy of feedback form. Further, a piecewise deterministic Markov model for the control of dividend pay-out and reinsurance is introduced. This model can be transformed to a model with uncontrolled flow. It is shown that a classical solution to the Bellman equation exists and that a non-relaxed optimal policy of feedback form can be obtained via the Bellman equation. Lipschitz continuity of the one-dimensional vector field defining the controlled flow will be replaced by strict positivity.

Piecewise Deterministic Markov Control Processes with Feedback Controls and Unbounded Costs

The control of piecewise-deterministic processes is studied where only local boundedness of the data is assumed. Moreover the discount rate may be zero. The value function is shown to be solution to the Bellman equation in a weak sense; however the solution concept is strong enough to generate optimal policies. Continuity and compactness conditions are given for the existence of nonrelaxed optimal feedback controls.

Stationary Policies in Dynamic Programming Models Under Compactness Assumptions

The present work deals with the usual stationary decision model of dynamic programming. The imposed convergence condition on the expected total rewards is so general that both the negative (unbounded) case and the positive (unbounded) case are included. However, the gambling model studied by Dubins and Savage is not covered by the present model. In addition to the convergence condition, a continuity and compactness condition is imposed. The main result states that the supremum of the expected total rewards under all stationary policies is equal to the supremum under all (possibly randomized and non-Markovian) policies.

Control of ruin probabilities by discrete-time investments

The control problem of controlling ruin probabilities by investments in a financial market is studied. The insurance business is described by the usual Cramer-Lundberg-type model and the risk driver of the financial market is a compound Poisson process. Conditions for investments to be profitable are derived by means of discrete-time dynamic programming. Moreover Lundberg bounds are established for the controlled model.

Markov Decision Processes in Finance and Dynamic Options

In this paper a discrete-time Markovian model for a financial market is chosen. The fundamental theorem of asset pricing relates the existence of a martingale measure to the no-arbitrage condition. It is explained how to prove the theorem by stochastic dynamic programming via portfolio optimization. The approach singles out certain martingale measures with additional interesting properties. Furthermore, it is shown how to use dynamic programming to study the smallest initial wealth x * that allows for super-hedging a contingent claim by some dynamic portfolio. There, a joint property of the set of policies in a Markov decision model and the set of martingale measures is exploited. The approach extends to dynamic options which are introduced here and are generalizations of American options.

Martingale Measures and Hedging for Discrete-Time Financial Markets

The price of stocks is modelled by a discrete-time, square-integrable, vector-valued process X. No further boundedness condition on X is imposed. Contingent claims H are described by square-integrable random variables. One looks for values v of the initial wealth v that allow for super-hedging H by some portfolio plan. In several cases, the smallest value v is known to coincide with the maximal expectation of H under equivalent martingale measures. Here, within an L2-framework, another sufficient condition is provided which can be looked upon as a stronger form of the no-arbitrage condition. The mathematical tool and one of the main contributions is an optional decomposition theorem for a process which is a supermartingale under any equivalent martingale measure. The upper price process for a contingent claim is shown to be a typical example for such a process. Moreover it is shown that in a Markovian model one can restrict attention to Markovian portfolio plans and to Markovian martingale measures.