Yoshinobu Kadota

Discounted Markov decision processes with utility constraints

We consider utility-constrained Markov decision processes. The expected utility of the total discounted reward is maximized subject to multiple expected utility constraints. By introducing a corresponding Lagrange function, a saddle-point theorem of the utility constrained optimization is derived. The existence of a constrained optimal policy is characterized by optimal action sets specified with a parametric utility.

Stopped decision processes in conjunction with general utility

A combined model of Markov decision process and the stopping problem, called Stopped Decision Processes, with a general utility is considered in the paper. This model is important for the various applications including Bandit problems and piecewise deterministic Markov processes, etc. Our aim is to formulate the general utility-treatment of processes with a countable state space and give a functional characterization from points of seeking an optimal pair, that is, a policy and a stopping time. And the further results concerning an optimality equation of our model are given. If we restrict the problem as the choice of a policy for a given fixed stopping time, these results are consistent with our previous paper (1998). Especially, in the case of the exponential utility functions, the optimal pair can be derived concretely using the idea of the onestep look ahead policy. Also, a simple numerical example is given to illustrate the results.

Notes on variance in randomized reward Markov decision processes

Abstract In Markov decision processes with randomized reward function, countable states and finite actions, this paper studies total variance of average rewards. The variance is represented explicitly in a different form from the deterministic reward case. By parallel arguments to the case, we show the existence of a stationary deterministic policy which is mean-variance optimal. The recurrent condition used in the existence theorem is examined.

Simultaneous recurrent conditions on countable state Markov chains

Abstract This paper considers four simultaneous recurrent conditions. They are imposed on a family of Markov chains with a countable state space. Several relationships among them will be obtained. These facts mean that they are sufficient for the existence and the continuity of the deviation matrices of the Markov chains.