Arie Hordijk

Constrained Undiscounted Stochastic Dynamic Programming

In this paper we investigate the computation of optimal policies in constrained discrete stochastic dynamic programming with the average reward as utility function. The state-space and action-sets are assumed to be finite. Constraints which are linear functions of the state-action frequencies are allowed. In the general multichain case, an optimal policy will be a randomized nonstationary policy. An algorithm to compute such an optimal policy is presented. Furthermore, sufficient conditions for optimal policies to be stationary are derived. There are many applications for constrained undiscounted stochastic dynamic programming, e.g., in multiple objective Markovian decision models.

Multimodularity, Convexity, and Optimization Properties

In this paper we investigate the properties of multimodular functions. In doing so we give elementary proofs for properties already established by Hajek and we generalize some of his results. In particular, we extend the relation between convexity and multimodularity to some convex subsets of z m . We also obtain general optimization results for average costs related to a sequence of multimodular functions rather than to a single function. Under this general context, we show that the expected average cost problem is optimized by using regular sequences. We finally illustrate the usefulness of this theory in admission control into a D/D/1 queue with fixed batch arrivals, with no state information. We show that the regular policy minimizes the average queue length for the case of an infinite queue, but not for the case of a finite queue. When further adding a constraint on the losses, it is shown that a regular policy is also optimal for the finite queue case.

On the Optimality of the Generalized Shortest Queue Policy

Consider a queueing model in which arriving customers have to choose between m parallel servers, each with its own queue. We prove for general arrival streams that the policy which assigns to the shortest queue is stochastically optimal for models with finite buffers and batch arrivals.

Balanced sequences and optimal routing

The objective pursued in this paper is two-fold. The first part addresses the following combinatorial problem: is it possible to construct an infinite sequence over n letters where each letter is distributed as “evenly” as possible and appears with a given rate? The second objective of the paper is to use this construction in the framework of optimal routing in queuing networks. We show under rather general assumptions that the optimal deterministic routing in stochastic event graphs is such a sequence.

Constrained admission control to a queueing system

We consider an exponential queue with arrival and service rates depending on the number of jobs present in the queue. The queueing system is controlled by restricting arrivals. Typically, a good policy should provide a proper balance between throughput and congestion. A mathematical model for computing such a policy is a Markov decision chain with rewards and a constrained cost function. We give general conditions on the reward and cost function which guarantee the existence of an optimal threshold or thinning policy. An efficient algorithm for computing an optimal policy is constructed.

Discrete-event control of stochastic networks : multimodularity and regularity

Preface.- Part I: Theoretical Foundations: Multimodularity, Convexity and Optimization Balanced Sequences Stochastic Event Graphs.- Part II: Admission and Routing Control: Admission Control in Stochastic Event Graphs Applications in Queuing Networks Optimal Routing Optimal Routing in two Deterministic Queues.- Part III: Several Extensions: Networks with no Buffers Vacancies, Service Allocation and Polling Monotonicity of Feedback Control.- Part IV: Comparisons: Comparison of Queues with Discrete-time Arrival Processes Simplex Convexity Orders and Bounds for Multimodular Functions Regular Ordering.- References.- Index.

On the Assignment of Customers to Parallel Queues

This paper considers routing to parallel queues in which each queue has its own single server, and service times are exponential with nonidentical parameters. We give conditions on the cost function such that the optimal policy assigns customers to a faster queue when that server has a shorter queue. The queues may have finite buffers, and the arrival process can be controlled and can depend on the state and routing policy. Hence our results on the structure of the optimal policy are also true when the assigning control is in the ”last” node of a network of service centers. Using dynamic programming we show that our optimality results are true in distribution. Published as Probability in the Engineering and Informational Sciences 6:495–511, 1992.

Average, sensitive and Blackwell optimal policies in denumerable Markov decisionchains with unbounded rewards

In this paper we consider a discrete-time Markov decision chain with a denumerable state space and compact action sets and we assume that for all states the rewards and transition probabilities depend continuously on the actions. The first objective of this paper is to develop an analysis for average optimality without assuming a special Markov chain structure. In doing so, we present a set of conditions guaranteeing average optimality, which are automatically fulfilled in the finite state and action model. The second objective is to study simultaneously average and discount optimality as Veinott Veinott, A. F., Jr. 1969. On discrete dynamic programming with sensitive discount optimality criteria. Ann. Math. Statist.40 1635--1660. did for the finite state and action model. We investigate the concepts of n-discount and Blackwell optimality in the denumerable state space, using a Laurent series expansion for the discounted rewards. Under the same condition as for average optimality, we establish solutions to the n-discount optimality equations for every n.

Taylor series expansions for stationary Markov chains

We study Taylor series expansions of stationary characteristics of general-state-space Markov chains. The elements of the Taylor series are explicitly calculated and a lower bound for the radius of convergence of the Taylor series is established. The analysis provided in this paper applies to the case where the stationary characteristic is given through an unbounded sample performance function such as the second moment of the stationary waiting time in a queueing system.

Contraction Conditions for Average and α-Discount Optimality in Countable State Markov Games with Unbounded Rewards

The goal of this paper is to provide a theory of N-person Markov games with unbounded cost, for a countable state space and compact action spaces. We investigate both the finite and infinite horizon problems. For the latter, we consider the discounted cost as well as the expected average cost. We present conditions for the infinite horizon problems for which equilibrium policies exist for all players within the stationary policies, and show that the costs in equilibrium policies exist for all players within the stationary policies, and show that the costs in equilibrium satisfy the optimality equations. Similar results are obtained for the finite horizon costs, for which equilibrium policies are shown to exist for all players within the Markov policies. As special case of N-person games, we investigate the zero-sum 2 players game, for which we establish the convergence of the value iteration algorithm. We conclude by studying an application of a zero-sum Markov game in a queueing model.