Naci Saldi

Asymptotic Optimality and Rates of Convergence of Quantized Stationary Policies in Stochastic Control

We consider the discrete approximation of stationary policies for a discrete-time Markov decision process with Polish state and action spaces under total, discounted, and average cost criteria. Deterministic stationary quantizer policies are introduced and shown to be able to approximate optimal deterministic stationary policies with arbitrary precision under mild technical conditions, thus demonstrating that one can search for ε-optimal policies within the class of quantized control policies. We also derive explicit bounds on the approximation error in terms of the quantization rate.

Convex Analysis in Decentralized Stochastic Control, Strategic Measures, and Optimal Solutions

This paper is concerned with the properties of the sets of strategic measures induced by admissible team policies in decentralized stochastic control and the convexity properties in dynamic team problems. To facilitate a convex analytical approach, strategic measures for team problems are introduced. Properties such as convexity, and compactness and Borel measurability under weak convergence topology are studied, and sufficient conditions for each of these properties are presented. These lead to existence of and structural results for optimal policies. It will be shown that the set of strategic measures for teams which are not classical is in general non-convex, but the extreme points of a relaxed set consist of deterministic team policies, which lead to their optimality for a given team problem under an expected cost criterion. Externally provided independent common randomness for static teams or private randomness for dynamic teams do not improve the team performance. The problem of when a sequential team problem is convex is studied and necessary and sufficient conditions for problems which include teams with a non-classical information structure are presented. Implications of this analysis in identifying probability and information structure dependent convexity properties are presented.

Finite Model Approximations and Asymptotic Optimality of Quantized Policies in Decentralized Stochastic Control

In this paper, we consider finite model approximations of a large class of static and dynamic team problems where these models are constructed through uniform quantization of the observation and action spaces of the agents. The strategies obtained from these finite models are shown to approximate the optimal cost with arbitrary precision under mild technical assumptions. In particular, quantized team policies are asymptotically optimal. This result is then applied to the Gaussian relay channel problem. This result also applies to Witsenhausens counterexample, which we had studied individually earlier.

Randomized quantization and optimal design with a marginal constraint

We consider the problem of optimal randomized vector quantization under a constraint on the outputs distribution. The problem is formalized by introducing a general representation of randomized quantization via probability measures over the space of joint distributions on the source and reproduction alphabets. Using this representation and results from optimal transport theory, we show the existence of an optimal (minimum distortion) randomized quantizer having a fixed output distribution under various conditions. For sources with densities and the mean square distortion measure, we show that this optimum can be attained by randomizing quantizers having convex code cells. We also consider a relaxed version of the problem where the output marginal must belong to some neighborhood (in the weak topology) of a fixed probability measure. We demonstrate that finitely randomized quantizers form an optimal class for the relaxed problem.

On the Asymptotic Optimality of Finite Approximations to Markov Decision Processes with Borel Spaces

Calculating optimal policies is known to be computationally difficult for Markov decision processes with Borel state and action spaces and for partially observed Markov decision processes even with finite state and action spaces. This paper studies finite-state approximations of discrete time Markov decision processes with Borel state and action spaces, for both discounted and average costs criteria. The stationary policies thus obtained are shown to approximate the optimal stationary policy with arbitrary precision under mild technical conditions. For compact-state MDPs, we obtain explicit rates of convergence bounds quantifying how the approximation improves as the size of the approximating finite state space increases. Using information theoretic arguments, the order optimality of the obtained rates of convergence is established for a large class of problems. We also show that, as a pre-processing setup, action space can taken to be finite with sufficiently large number points for the finite-state approximation problem; thereby, well known algorithms, such as value or policy iteration, Q-learning, etc., can be used to calculate near optimal policies.

Randomized Quantization and Source Coding With Constrained Output Distribution

This paper studies fixed-rate randomized vector quantization under the constraint that the quantizers output has a given fixed probability distribution. A general representation of randomized quantizers that includes the common models in the literature is introduced via appropriate mixtures of joint probability measures on the product of the source and reproduction alphabets. Using this representation and results from optimal transport theory, the existence of an optimal (minimum distortion) randomized quantizer having a given output distribution is shown under various conditions. For sources with densities and the mean square distortion measure, it is shown that this optimum can be attained by randomizing quantizers having convex codecells. For stationary and memoryless source and output distributions, a rate-distortion theorem is proved, providing a single-letter expression for the optimum distortion in the limit of large blocklengths.

Asymptotic optimality of quantized policies in stochastic control under weak continuity conditions

Quantization is an increasingly important operation both because of applications in networked control and the computational benefits of working with finite state spaces. In this paper, we consider quantized approximations of stationary policies for a discrete-time Markov decision process with discounted and average costs and weakly continuous transition probability kernels. We show that deterministic stationary quantizer policies approximate optimal deterministic stationary policies with arbitrary precision under mild technical conditions. We thus extend recent and older results in the literature which consider more stringent continuity conditions for the transition kernels, such as setwise continuity, which limit the applicability of such results. In particular, the weaker continuity requirements allow for the study of partially observable Markov decision processes under practical conditions.

Finite state approximations of Markov decision processes with general state and action spaces

General state space valued optimal stochastic control problems are often computationally intractable. On the other hand, for finite state-action models, there exist powerful computational and simulation tools for computing optimal strategies. With this motivation, we consider finite state and action space approximations of discrete time Markov decision processes with discounted and average costs and compact state and action spaces. Stationary policies obtained from finite state approximations of the original model are shown to approximate the optimal stationary policy with arbitrary precision under mild technical conditions. These results complement recent work that studied the finite action approximation of discrete time Markov decision process with discounted and average costs.

Approximation of stationary control policies by quantized control in Markov decision processes

We consider the problem of approximating optimal stationary control policies by quantized control. Stationary quantizer policies are introduced and it is shown that such policies are “-optimal among stationary policies under mild technical conditions. Quantitative bounds on the approximation error in terms of the rate of the approximating quantizers are also derived. Thus, one can search for”-optimal policies within quantized control policies. These pave the way for applications in optimal design of networked control systems where controller actions need to be quantized, as well as for a new computational method for the generation of approximately optimal Markov decision policies in general (Borel) state and action spaces for both discounted cost and average cost infinite horizon optimal control problems.

Convex analysis in decentralized stochastic control and strategic measures

This paper is concerned with the properties of the sets of strategic measures induced by admissible team policies in decentralized stochastic control and the convexity properties in dynamic team problems. To facilitate a convex analytical approach, strategic measures for team problems are introduced. Properties such as convexity, and compactness and Borel measurability under weak convergence topology are studied, and sufficient conditions for each of these properties are presented. These lead to existence of and structural results for optimal policies. It will be shown that the set of strategic measures for teams which are not classical is in general non-convex, but the extreme points of a relaxed set consist of deterministic team policies, which lead to their optimality for a given team problem under an expected cost criterion. Externally provided independent common randomness for static teams or private randomness for dynamic teams do not improve the team performance. The problem of when a sequential team problem is convex is studied and necessary and sufficient conditions for problems which include teams with a non-classical information structure are presented. Implications of this analysis in identifying probability and information structure dependent convexity properties are presented.