Sean P. Meyn

Stability of Markovian processes. III: Foster-Lyapunov criteria for continuous-time processes

In Part I we developed stability concepts for discrete chains, together with Foster-Lyapunov criteria for them to hold. Part II was devoted to developing related stability concepts for continuous-time processes. In this paper we develop criteria for these forms of stability for continuous-parameter Markovian processes on general state spaces, based on Foster-Lyapunov inequalities for the extended generator. Such test function criteria are found for non-explosivity, non-evanescence, Harris recurrence, and positive Harris recurrence. These results are proved by systematic application of Dynkins formula. We also strengthen known ergodic theorems, and especially exponential ergodic results, for continuous-time processes. In particular we are able to show that the test function approach provides a criterion for f-norm convergence, and bounding constants for such convergence in the exponential ergodic case. We apply the criteria to several specific processes, including linear stochastic systems under non-linear feedback, work-modulated queues, general release storage

The O.D. E. Method for Convergence of Stochastic Approximation and Reinforcement Learning

It is shown here that stability of the stochastic approximation algorithm is implied by the asymptotic stability of the origin for an associated ODE. This in turn implies convergence of the algorithm. Several specific classes of algorithms are considered as applications. It is found that the results provide (i) a simpler derivation of known results for reinforcement learning algorithms; (ii) a proof for the first time that a class of asynchronous stochastic approximation algorithms are convergent without using any a priori assumption of stability; (iii) a proof for the first time that asynchronous adaptive critic and Q-learning algorithms are convergent for the average cost optimal control problem.

Stability of Markovian processes II: continuous-time processes and sampled chains

In this paper we extend the results of Meyn and Tweedie (1992b) from discrete-time parameter to continuous-parameter Markovian processes Φ evolving on a topological space. We consider a number of stability concepts for such processes in terms of the topology of the space, and prove connections between these and standard probabilistic recurrence concepts. We show that these structural results hold for a major class of processes (processes with continuous components) in a manner analogous to discrete-time results, and that complex operations research models such as storage models with state-dependent release rules, or diffusion models such as those with hypoelliptic generators, have this property

Stability and convergence of moments for multiclass queueing networks via fluid limit models

The subject of this paper is open multiclass queueing networks, which are common models of communication networks, and complex manufacturing systems such as wafer fabrication facilities. We provide sufficient conditions for the existence of bounds on long-run average moments of the queue lengths at the various stations, and we bound the rate of convergence of the mean queue length to its steady-state value. Our work provides a solid foundation for performance analysis either by analytical methods or by simulation. These results are applied to several examples including re-entrant lines, generalized Jackson networks, and a general polling model as found in computer networks applications. >

Duality and linear programs for stability and performance analysis of queuing networks and scheduling policies

We consider the problems of performance analysis and stability/instability determination of queuing networks and scheduling policies. We exhibit a strong duality relationship between the performance of a system and its stability analysis via mean drift. We obtain a variety of linear programs (LPs) to conduct such stability and performance analyses. A certain LP, called the performance LP, bounds the performance of all stationary nonidling scheduling policies. If it is bounded, then its dual, called the drift LP, has a feasible solution which is a copositive matrix. The quadratic form associated with this copositive matrix has a negative drift, showing that all stationary nonidling scheduling policies result in a geometrically converging exponential moment. These results carry over to fluid models, allowing the study of networks with nonexponential distributions. If a modification of the performance LP, called the monotone LP, is bounded, then the system is stable. Finally, there is a another modification of the performance LP, called the finite time LP. It provides transient bounds on the performance of the system from any initial condition.

Characterization and computation of optimal distributions for channel coding

This paper concerns the structure of capacity-achieving input distributions for stochastic channel models, and a renewed look at their computational aspects. The following conclusions are obtained under general assumptions on the channel statistics. i) The capacity-achieving input distribution is binary for low signal-to-noise ratio (SNR). The proof is obtained on comparing the optimization equations that determine channel capacity with a linear program over the space of probability measures. ii) Simple discrete approximations can nearly reach capacity even in cases where the optimal distribution is known to be absolutely continuous with respect to Lebesgue measure. iii) A new class of algorithms is introduced based on the cutting-plane method to iteratively construct discrete distributions, along with upper and lower bounds on channel capacity. It is shown that the bounds converge to the true channel capacity, and that the distributions converge weakly to a capacity-achieving distribution.

Ancillary Service to the Grid Through Control of Fans in Commercial Building HVAC Systems

The thermal storage potential in commercial buildings is an enormous resource for providing various ancillary services to the grid. In this paper, we show how fans in Heating, Ventilation, and Air Conditioning (HVAC) systems of commercial buildings alone can provide substantial frequency regulation service, with little change in their indoor environments. A feedforward architecture is proposed to control the fan power consumption to track a regulation signal. The proposed control scheme is then tested through simulations based on a calibrated high fidelity non-linear model of a building. Model parameters are identified from data collected in Pugh Hall, a commercial building located on the University of Florida campus. For the HVAC system under consideration, numerical experiments demonstrate how up to 15% of the rated fan power can be deployed for regulation purpose while having little effect on the building indoor temperature. The regulation signal that can be successfully tracked is constrained in the frequency band [1/τ0,1/τ1], where τ0 ≈ 3 minutes and τ1 ≈ 8 seconds. Our results indicate that fans in existing commercial buildings in the U.S. can provide about 70% of the current national regulation reserve requirements in the aforementioned frequency band. A unique advantage of the proposed control scheme is that assessing the value of the ancillary service provided is trivial, which is in stark contrast to many demand-response programs.

A sensor-utility-network method for estimation of occupancy in buildings

We introduce the sensor-utility-network (SUN) method for occupancy estimation in buildings. Based on inputs from a variety of sensor measurements, along with historical data regarding building utilization, the SUN estimator produces occupancy estimates through the solution of a receding-horizon convex optimization problem. State-of-the-art on-line occupancy algorithms rely on indirect measurements, such as CO2 levels, or people counting sensors which are subject to significant errors and cost. The newly proposed method was evaluated via experiments in an office building environment. Estimation accuracy is shown to improve significantly when all available data is incorporated in the estimator. In particular, it is found that the average estimation error at the building level is reduced from 70% to 11% using the SUN estimator, when compared to the naive approach that relies solely on flow measurements.

Sequencing and Routing in Multiclass Queueing Networks Part I: Feedback Regulation

This paper establishes new criteria for stability and for instability of multiclass network models under a given stationary policy. It also extends previous results on the approximation of the solution to the average cost optimality equations through an associated fluid model: It is shown that an optimized network possesses a fluid limit model which is itself optimal with respect to a total cost criterion. A general framework for constructing control algorithms for multiclass queueing networks is proposed based on these general results. Network sequencing and routing problems are considered as special cases. The following aspects of the resulting feedback regulation policies are developed in the paper: The policies are stabilizing and are, in fact, geometrically ergodic for a Markovian model. Numerical examples are given. In each case it is shown that the feedback regulation policy closely resembles the average-cost optimal policy. A method is proposed for reducing variance in simulation for a network controlled using a feedback regulation policy.

Building thermal model reduction via aggregation of states

This paper proposes an aggregation-based model reduction method for thermal models of large buildings. Using an electric analogy, the baseline thermal model is represented as an RC-network. The proposed model reduction methodology is used to obtain a simpler (with fewer states) multi-scale representation of this network. The methodology preserves the electrical analogy and retains the physical intuition during the model reduction process. The theoretical results are illustrated with the aid of examples.