Frederick J. Beutler

Optimal policies for controlled Markov chains with a constraint

Abstract The time average reward for a discrete-time controlled Markov process subject to a time-average cost constraint is maximized over the class of al causal policies. Each epoch, a reward depending on the state and action, is earned, and a similarly constituted cost is assessed; the time average of the former is maximized, subject to a hard limit on the time average of the latter. It is assumed that the state space is finite, and the action space compact metric. An accessibility hypothesis makes it possible to utilize a Lagrange multiplier formulation involving the dynamic programming equation, thus reducing the optimization problem to an unconstrained optimization parametrized by the multiplier. The parametrized dynamic programming equation possesses compactness and convergence properties that lead to the following: If the constraint can be satisfied by any causal policy, the supremum over time-average rewards respective to all causal policies is attained by either a simple or a mixed policy; the latter is equivalent to choosing independently at each epoch between two specified simple policies by the throw of a biased coin.

The theory of stationary point processes.

An axiomatic formulation is presented for point processes which may be interpreted as ordered sequences of points randomly located on the real line. Such concepts as forward recurrence times and number of points in intervals are defined and related in set-theoretic Note that for α∈A,Gα may not coverGα as a convex subgroup and so we cannot use Theorem 1.1 to prove this result. Moreover, all that we know about theGα/Gα is that each is an extension of a trivially ordered subgroup by a subgroup ofR. ItB is a plenary subset ofA, then there exists av-isomorphism μ ofG intoV(B, Gβ/Gβ), but whether or not μ is ano-isomorphism is not known.

The Spectral Analysis of Impulse Processes

An expression for the spectral density of the impulse process s(t) = {if236-1} is derived under the assumption that \αn\}} is a stationary process, and that \tn\}} is a stationary point process independent of \αn\}}. The spectral density appears as an infinite series in terms of the correlation of \αn\}} and the interval statistics of \tn\}}. The same result was obtained by Leneman by a different argument under considerably more restrictive conditions of validity. Various models of impulse processes are discussed relative to random sampling of random processes. Random and systematic loss of samples, separate read-in and read-out jitters, and correlated random scaling errors can all be represented by appropriate assumptions on \αn\}} and \tn\}}. Finally, closed form expressions are calculated for the spectral density of s(t) and the sampled process under combinations of the sampling errors mentioned in the preceding paragraph.

Sampling theorems and bases in a Hilbert space

A unified approach to sampling theorems for (wide sense) stationary random processes rests upon Hilbert space concepts. New results in sampling theory are obtained along the following lines: recovery of the process x(t) from nonperiodic samples, or when any finite number of samples are deleted; conditions for obtaining x (t) when only the past is sampled; a criterion for restoring x(t) from a finite number of consecutive samples; and a minimum mean square error estimate of x(t) based on any (possibly nonperiodie) set of samples. In each case, the proofs apply not only to the recovery of x(t), but are extended to show that (almost) arbitrary linear operations on x (t) can be reproduced by linear combinations of the samples. Further generality is attained by use of the spectral distribution function F(. ) of x(t), without assuming F(.) absolutely continuous.

Random sampling of random processes: Stationary point processes

This is the first of a series of papers treating randomly sampled random processes. Spectral analysis of the resulting samples pre-supposes knowledge of the statistics of tn, the random point process whose variates represent the sampling times. We introduce a class of stationary point processes, whose stationarity (as characterized by any of several equivalent criteria) leads to wide-sense stationary sampling trains when applied to wide-sense stationary processes. Of greatest importance are the nth forward [backward] recurrence times (distances from t to the nth point thereafter [preceding]), whose distribution functions prove more useful to the computation of covariances than interval statistics, and which possess remarkable properties that facilitate the analysis. The moments of the number of points in an interval are evaluated by weighted sums of recurrence time distribution functions, the moments being finite if and only if the associated sum converges. If the first moment is finite, these distribution functions are absolutely continuous, and obey some convexity relations. Certain formulas relate recurrence statistics to interval length statistics, and conversely; further, the latter are also suitable for a direct evaluation of moments of points in intervals. Our point process requires neither independent nor identically distributed interval lengths. It embraces most of the common sampling schemes (e.g., periodic, Poisson, jittered), as well as some new models. Of particular interest are point processes obtained from others by a random deletion of points (skip processes), as for instance a jittered cyclically periodic process with (random or systematic) skipping. Computation of the statistics for several point processes yields new results of interest not only for their own sake, but also of use for spectral analyses appearing in other papers of this series.

Routing in queueing networks under imperfect information: stochastic dominance and thresholds

Optimal policies for routing between two servers under imperfect information is treated by discrete time dynamic programming. It is proved that certain inequalities involving stochastic ordering of information measures can be propagated inductively from one epoch to the next. Convexity conditions on the instantaneous costs insure proper initiation and inductive continuation of these properties. Consequently, an inductive procedure shows that threshold routing policies are optimal, and that the total cost is convex and monotone. Two examples are provided. The first deals with a tandem queue having inputs to both work stations, and only inferential information available on the state of the second station. It is first shown that the optimal control is bang-bang, and then that the hypotheses dictating a threshold policy resulting in convex and monotone costs are satisfied. The second example considers optimal routing for customers arriving in a renewal stream, when the routing decision is between two parallel...

The operator theory of the pseudo-inverse I. Bounded operators

Operator theory of pseudo-inverse generalized to encompass linear bounded operators on Hilbert spaces

The Operator Pseudoinverse in Control and Systems Identification

Publisher Summary This chapter discusses the pseudoinverse of a linear transformation between Hilbert spaces. It presents the basic theory and shows its use for applications to systems identification and the quadratic regulator problem. It also presents an exposition of the basic theory of the pseudoinverse for densely defined linear closed operators with arbitrary range. The theory includes the case of operators that do not have closed range. The chapter discusses the Gauss-Markov theorem on statistical estimation that is shown to be proved under the hypothesis that both the quantity to be estimated and the observations are elements of Hilbert spaces. This theorem applies only to nonsingular covariance operators and reduces to the classical Gauss-Markov theorem when the spaces are finite dimensional. In one classical form of the quadratic regulator problem, it is required to find the minimum energy input that will move a system from some initial state to the origin at a designated time. The chapter presents a reformulation that generalizes this problem to admit a greater variety of linear constraints, possibly including some of which are incompatible and/or unattainable by the system. The solution always exists as a pseudoinverse and reduces to the classical result if the system is capable of meeting the constraint.

The Operator Theory of the Pseudo-Inverse II. Unbounded Operators with Arbitrary Range*

Extension of pseudo-inverse operator theory to Hilbert space operators that are unbounded and have arbitrary range

On the statistics of random pulse processes

Statistics are obtained for pulse trains in which the pulse shapes as well as the time base are random. The general expression derived for the mean and spectral density of the pulse train require neither independence of intervals between time base points nor independence of the pulses. The spectral density appears as an infinite series that can be summed to closed form in many applications (e.g., pulse duration modulation with skipped and jittered samples). If the time base is a Poisson point process and the pulse shapes are independent, stronger results become available; we are then able to calculate joint characteristic functions for the pulse process, thus providing a more complete statistical description. Examples are given, illustrating use of the above results for pulse duration modulation (with arbitrary pulse shapes) and telephone traffic.