Alexei B. Piunovskiy

Optimal Control of Random Sequences in Problems with Constraints

Preface. Introduction. 1. Methods of Stochastic Optimal Control. 2. Optimal Control Problems with Constraints. 3. Solvability of the Main Constrained Problem and Some Extensions. 4. Linear-Quadratic Systems. 5. Some Applications. Conclusion. Appendix. References. List of Symbols. List of the Main Statements.

Constrained Markovian decision processes: the dynamic programming approach

We consider semicontinuous controlled Markov models in discrete time with total expected losses. Only control strategies which meet a set of given constraint inequalities are admissible. One has to build an optimal admissible strategy. The main result consists in the constructive development of optimal strategy with the help of the dynamic programming method. The model studied covers the case of a finite horizon and the case of a homogeneous discounted model with different discount factors.

Optimal choice of the buffer size in the Internet routers

We study an optimal choice of the buffer size in the Internet routers. The objective is to determine the minimum value of the buffer size required in order to fully utilize the link capacity. The reare some empirical rules for the choice of the buffer size. The most known rule of thumb states that the buffer length should be set to the bandwidth delay product of the network, i.e., the product between the average round trip time in the network and the capacity of the bottleneck link. Several recent works have suggested that as a consequence of the traffic aggregation, the buffer size should be set to smaller values. In this paper we propose an analytical framework for the optimal choice of the router buffer size. We formulate this problem as a multi-criteria optimization problem, in which the Lagrange function corresponds to a linear combination of the average sending rate and average delay in the queue. The solution to this optimization problem provides further evidence that indeed the buffer size should be reduced in the presence of traffic aggregation. Furthermore, our result states that the minimum required buffer is smaller than what previous studies suggested. Our analytical results are confirmed by simulations performed with the NS simulator.

Discounted Continuous-Time Markov Decision Processes with Constraints: Unbounded Transition and Loss Rates

This paper deals with denumerable continuous-time Markov decision processes (MDP) with constraints. The optimality criterion to be minimized is expected discounted loss, while several constraints of the same type are imposed. The transition rates may be unbounded, the loss rates are allowed to be unbounded as well (from above and from below), and the policies may be history-dependent and randomized. Based on Kolmogorovs forward equation and Dynkins formula, we remind the reader about the Bellman equation, introduce and study occupation measures, reformulate the optimization problem as a (primary) linear program, provide the form of optimal policies for a constrained optimization problem here, and establish the duality between the convex analytic approach and dynamic programming. Finally, a series of examples is given to illustrate all of our results.

Bicriteria Optimization of a Queue with a Controlled Input Stream

Motivated by a certain situation in communication networks, we will investigate the M/M/1/∞ queuing system, where one can change the input stream. Performance criteria coincide with the long-run average throughput of the system and the queue length. We will present a rigorous mathematical study of the constrained version of the multicriteria optimization problem for jump Markov processes. Subsequently, it will be shown that the stationary control strategies of the threshold type form a sufficient class in the initial bicriteria problem considered.

An explicit optimal isolation policy for a deterministic epidemic model

Optimal policies involving the isolation of infectives are derived for a deterministic epidemic model with non-standard infection rate function. Denoting by y the number of infective individuals and x the number of susceptible individuals in the population, we replace the classical Kermack-McKendrick infection rate function @bxy (for some constant @b) with @bxy/(x+y). This modified model has been studied by various previous authors, but not in the context of control policies. We show that the optimal isolation policy is to intervene with maximal effort when y=

Convergence of trajectories and optimal buffer sizing for MIMD congestion control

We study the interaction between the MIMD (Multiplicative Increase Multiplicative Decrease) congestion control and a bottleneck router with Drop Tail buffer. We consider the problem in the framework of deterministic hybrid models. We study conditions under which the system trajectories converge to limiting cycles with a single jump. Following that, we consider the problem of the optimal buffer sizing in the framework of multi-criteria optimization in which the Lagrange function corresponds to a linear combination of the average throughput and the average delay in the queue. As case studies, we consider the Slow Start phase of TCP New Reno and Scalable TCP for high speed networks.

Multicriteria impulsive control of jump Markov processes

Abstract.Impulsive control consideres so called interventions meaning immediate change of the state of the system; between intervention, the continuous time jump Markov process is uncontrollable, with “natural” jump intensities. Multicriteria control problem for such model is considered, and the constrained version is investigated with the help of the Lagrange multipliers technique. All the theory is illustrated by an example of the optimal control of epidemic with carriers. The fluid model approach to the epidemic considered, is presented, too.

Random walk, birth-and-death process and their fluid approximations: absorbing case

Fluid models are used to study functionals of the underlying random processes. Instead of analysing the trajectories, we investigate algebraic equations of the dynamic programming type which turn out to be discrete analogs of the corresponding differential equations. This analysis makes it possible to estimate the accuracy of approximation. Since the algebraic equations are the same for random walks and continuous time birth-and-death processes, we study the two cases in parallel. Several illustrative examples are also presented.

The expected total cost criterion for Markov decision processes under constraints: a convex analytic approach

This paper deals with discrete-time Markov decision processes (MDPs) under constraints where all the objectives have the same form of expected total cost over the infinite time horizon. The existence of an optimal control policy is discussed by using the convex analytic approach. We work under the assumptions that the state and action spaces are general Borel spaces, and that the model is nonnegative, semicontinuous, and there exists an admissible solution with finite cost for the associated linear program. It is worth noting that, in contrast to the classical results in the literature, our hypotheses do not require the MDP to be transient or absorbing. Our first result ensures the existence of an optimal solution to the linear program given by an occupation measure of the process generated by a randomized stationary policy. Moreover, it is shown that this randomized stationary policy provides an optimal solution to this Markov control problem. As a consequence, these results imply that the set of randomized stationary policies is a sufficient set for this optimal control problem. Finally, our last main result states that all optimal solutions of the linear program coincide on a special set with an optimal occupation measure generated by a randomized stationary policy. Several examples are presented to illustrate some theoretical issues and the possible applications of the results developed in the paper.