Zhongjing Ma

Stochastic Control of Network Systems II: NETCAD Optimal Control & the HJB Equation

The centralized optimal stochastic control of call admission (CAC) and routing (RC) problems is analysed for NETCAD systems as formulated in (Z. Ma et al., 2006). In the principal result an HJB equation is derived for the optimal stochastic control of NETCAD systems; this equation is equivalent to a hierarchy of HJB PDEs linked by integral coefficients. In the main example this system of equations is solved for a very simple network with a Poisson call request process and an exponential connection holding time process

Stochastic Control of Network Systems I: NETCAD State Space Structure & Dynamics

A stochastic systems framework is established for the formulation of call admission control (CAC) and routing control (RC) problems in loss networks. The state process of the underlying system is a piecewise deterministic Markov process (PDMP) evolving deterministically between random event instants at which times the state jumps to another state value. The random events in the system correspond to the arrival of call requests or the departure of connections. The system state process is hybrid since it possesses positive integer and positive real valued components. The resulting NETCAD stochastic state space systems framework permits the formulation and analysis of centralized optimal stochastic control with respect to specified utility functions, see (Z. Ma et al., 2006)

Control of Admission and Routing in Loss Networks: Hybrid Dynamic Programming Equations

Call admission and routing control decisions in stochastic loss (circuit-switched) networks with semi Markovian, multi-class, call arrival and general connection time processes are formulated as optimal stochastic control problems. The resulting so-called Hybrid Dynamic Programming equation systems take the form of vectors of partial differential equations with each component associated to a distinct distribution of routed calls over the network (i.e. distinct occupation states). This framework reduces to that of a Markov Decision Process when the traffic is Poisson and the associated computational limitations are approximately those of linear programs. Examples are provided of (i) network state space constructions and controlled state transition processes, (ii) a new closed form solution for a simple network, and (iii) the analysis and illustrative numerical results for a three link network.

Distributed control of loss network systems: Independent subnetwork behaviour in infinite networks

Call admission and routing control of loss (circuit- switched) networks can be formulated as optimal stochastic control (OSC) problems in case of a class of integral cost functions. The resulting Hamilton-Jacobi-Bellman (HJB) equation for such OSC problems consists of a collection of coupled first order PDEs linked by sets of integral coefficients. Unfortunately, the implementation of optimal control laws even for medium size systems is not feasible since the computational complexity of the HJB equations increases exponentially with network size. In this paper, we study admission control problems for a specific class of radial network systems composed of a group of weakly coupled subnetwork systems. We delineate the so-called (asymptotic) subnetwork (stochastic state) independence property as the network size goes to infinity. In particular this implies that the acceptance of incoming and outgoing call requests are asymptotically independent of all other state processes of the mass (i.e. overall) system. Under the general class of Markovian feedback control laws, we show the basic property of asymptotic sustainability of independent subnetwork behaviour holds as the network size goes to infinity. Based upon this class of network system models, distributed OSC problems may be formulated whereby each subnetwork system implements a local optimal control. This methodology leads to an application of the Nash certainty equivalence (NCE) principle itself proven quite useful within the LQG framework in M. Huang, et al., (Dec. 2006).

Stochastic Hybrid Netcad Systems for Modeling Call Admission and Routing Control in Networks

Abstract In this paper a stochastic hybrid systems framework is established for the formulation of call admission control (CAC) and routing control (RC) problems in networks. The hybrid state process of the underlying system is a piecewise deterministic Markovian process (PDMP) evolving deterministically between random event instants at which times the state jumps to another state value. The random events in the system correspond to the arrival of call requests or the departure of connections. The resulting NETCAD stochastic state space systems framework permits the formulation and analysis of centralized optimal stochastic control with respect to specified utility functions [3,8,10,11].

Distributed control for radial loss network systems via the ash Certainty Equivalence (mean field) principle

The computational intractability of the dynamic programming (DP) equations associated with optimal admission and routing in stochastic loss networks of any non-trivial size (Ma et al., 2006, 2008) leads to the consideration of suboptimal distributed game theoretic formulations of the problem. This work presents a formulation of loss network admission control problems in terms of a class of systems composed of a large population of weakly coupled competitive individual agent networks. The resulting distributed dynamic stochastic game problem is solved and analyzed by application of the so-called point process Nash certainty equivalence (PPNCE) principle; this is an extension to the network point process context of the NCE principle originally formulated in the LQG framework by M. Huang et al., (2006, 2007). This methodology has close connections with the mean field models studied by Lasry and Lions (2006, 2007) and the notion of oblivious equilibrium proposed by Weintraub, Benkard, and Van Roy (2005, 2007) via a mean field approximation.

Large population games in radial loss networks: Computationally tractable equilibria for distributed network admission control

The computational intractability of the dynamic programming (DP) equations associated with optimal admission and routing in stochastic loss networks of any non-trivial size (Ma et al, 2006, 2008) leads one to consider suboptimal distributed game theoretic formulations of the problem. The special class of radial networks with a central core of infinite capacity is considered, and it is shown (under adequate assumptions) that an associated distributed admission control problem becomes tractable asymptotically, as the size of radial network grows to infinity. This is achieved by following a methodology first explored by M. Huang et. al. (2003, 2006–2008) in the context of large scale dynamic games for sets of weakly coupled linear stochastic control systems. At the established Nash equilibrium, each agent reacts optimally with respect to the average trajectory of the mass of all other agents; this trajectory is approximated by a deterministic infinite population limit (associated with the mean field or ensemble statistics of the random agents) which is the solution of a particular fixed point problem. This framework has connections with the mean field models studied by Lasry and Lions (2006, 2007) and close connections with the notion of oblivious equilibrium proposed by Weintraub, Benkard, and Van Roy (2005, 2008) via a mean field approximation.