James C. Spall

Multivariate stochastic approximation using a simultaneous perturbation gradient approximation

The problem of finding a root of the multivariate gradient equation that arises in function minimization is considered. When only noisy measurements of the function are available, a stochastic approximation (SA) algorithm for the general Kiefer-Wolfowitz type is appropriate for estimating the root. The paper presents an SA algorithm that is based on a simultaneous perturbation gradient approximation instead of the standard finite-difference approximation of Keifer-Wolfowitz type procedures. Theory and numerical experience indicate that the algorithm can be significantly more efficient than the standard algorithms in large-dimensional problems. >

Implementation of the simultaneous perturbation algorithm for stochastic optimization

The need for solving multivariate optimization problems is pervasive in engineering and the physical and social sciences. The simultaneous perturbation stochastic approximation (SPSA) algorithm has recently attracted considerable attention for challenging optimization problems where it is difficult or impossible to directly obtain a gradient of the objective function with respect to the parameters being optimized. SPSA is based on an easily implemented and highly efficient gradient approximation that relies on measurements of the objective function, not on measurements of the gradient of the objective function. The gradient approximation is based on only two function measurements (regardless of the dimension of the gradient vector). This contrasts with standard finite-difference approaches, which require a number of function measurements proportional to the dimension of the gradient vector. This paper presents a simple step-by-step guide to implementation of SPSA in generic optimization problems and offers some practical suggestions for choosing certain algorithm coefficients.

Model-free control of nonlinear stochastic systems with discrete-time measurements

Consider the problem of developing a controller for general (nonlinear and stochastic) systems where the equations governing the system are unknown. Using discrete-time measurement, this paper presents an approach for estimating a controller without building or assuming a model for the system. Such an approach has potential advantages in accommodating complex systems with possibly time-varying dynamics. The controller is constructed through use of a function approximator, such as a neural network or polynomial. This paper considers the use of the simultaneous perturbation stochastic approximation algorithm which requires only system measurements. A convergence result for stochastic approximation algorithms with time-varying objective functions and feedback is established. It is shown that this algorithm can greatly enhance the efficiency over more standard stochastic approximation algorithms based on finite-difference gradient approximations.

A one-measurement form of simultaneous perturbation stochastic approximation

The simultaneous perturbation stochastic approximation (SPSA) algorithm has proven very effective for difficult multivariate optimization problems where it is not possible to obtain direct gradient information. As discussed to date, SPSA is based on a highly efficient gradient approximation requiring only two measurements of the loss function independent of the number of parameters being estimated. This note presents a form of SPSA that requires only one function measurement (for any dimension). Theory is presented that identifies the class of problems for which this one-measurement form will be asymptotically superior to the standard two-measurement form.

A Stochastic Approximation Technique for Generating Maximum Likelihood Parameter Estimates

This paper shows how stochastic approximation (SA) can be used to construct maximum likelihood estimates of system parameters. The procedure described here relies on a derivative approximation other than the usual finite-difference approximation associated with a Kiefer-Wolfowitz SA procedure. This alternative derivative approximation requires fewer, by a factor equal to the dimension of the parameter vector being estimated, computations than the standard finite-difference approximation. Numerical evidence presented in the paper indicates that this SA procedure is, relative to a Kiefer-Wolfowitz procedure, most efficient when considering large-scale systems.

TRAFFIC-RESPONSIVE SIGNAL TIMING FOR SYSTEM-WIDE TRAFFIC CONTROL

Abstract A long-standing problem in traffic engineering is to optimize the flow of vehicles through a given road network. Improving the timing of the traffic signals at intersections in the network is generally the most powerful and cost-effective means of achieving this goal. However, because of the many complex aspects of a traffic system—human behavioral considerations, vehicle flow interactions within the network, weather effects, traffic accidents, long-term (e.g. seasonal) variation, etc.—it has been notoriously difficult to determine the optimal signal timing. This is especially the case on a system-wide (multiple intersection) basis. Much of this difficulty has stemmed from the need to build extremely complex models of the traffic dynamics as a component of the control strategy. This paper presents a fundamentally different approach for optimal signal timing that eliminates the need for such complex models. The approach is based on a neural network (or other function approximator) serving as the basis for the control law, with the weight estimation occurring in closed-loop mode via the simultaneous perturbation stochastic approximation (SPSA) algorithm. The neural network function uses current traffic information to solve the current (instantaneous) traffic problem on a system-wide basis through an optimal signal timing strategy. The approach is illustrated by a realistic simulation of a nine-intersection network within the central business district of Manhattan, New York.

Nonlinear adaptive control using neural networks: estimation with a smoothed form of simultaneous perturbation gradient approximation

This is a condensed version of a full length interdisciplinary paper spanning the fields of control statistics, neural networks and optimisation. In this paper only the following two topics are considered: 1) consider the problem of developing adaptive controllers for general dynamic systems with unknown governing equations and develop a solution for an important class of such problems; and 2) introduce a modification to simultaneous perturbation stochastic approximation that is based on smoothing gradient approximations across iterations and illustrate this modification on the neural net based control problem.

A stochastic approximation algorithm for large-dimensional systems in the Kiefer-Wolfowitz setting

The author considers the problem of finding a root of the multivariate gradient equation that arises in function maximization. When only noisy measurements of the function are available, a stochastic approximation (SA) algorithm of the general type due to Kiefer and Wolfowitz (1952) is appropriate for estimating the root. An SA algorithm is presented that is based on a simultaneous-perturbation gradient approximation instead of the standard finite-difference approximation of Kiefer-Wolfowitz type procedures. Theory and numerical experience indicate that the algorithm can be significantly more efficient than the standard finite-difference-based algorithms in large-dimensional problems.<<ETX>>

Feedback and Weighting Mechanisms for Improving Jacobian Estimates in the Adaptive Simultaneous Perturbation Algorithm

It is known that a stochastic approximation (SA) analogue of the deterministic Newton-Raphson algorithm provides an asymptotically optimal or near-optimal form of stochastic search. However, directly determining the required Jacobian matrix (or Hessian matrix for optimization) has often been difficult or impossible in practice. This paper presents a general adaptive SA algorithm that is based on a simple method for estimating the Jacobian matrix while concurrently estimating the primary parameters of interest. Relative to prior methods for adaptively estimating the Jacobian matrix, the paper introduces two enhancements that generally improve the quality of the estimates for underlying Jacobian (Hessian) matrices, thereby improving the quality of the estimates for the primary parameters of interest. The first enhancement rests on a feedback process that uses previous Jacobian estimates to reduce the error in the current estimate. The second enhancement is based on an optimal weighting of per-iteration Jacobian estimates. From the use of simultaneous perturbations, the algorithm requires only a small number of loss function or gradient measurements per iteration - independent of the problem dimension - to adaptively estimate the Jacobian matrix and parameters of primary interest.

Asymptotic Sampling Distribution for Polynomial Chaos Representation from Data: A Maximum Entropy and Fisher Information Approach

A procedure is presented for characterizing the asymptotic sampling distribution of the estimators of the polynomial chaos (PC) coefficients of physical process modeled as non-stationary, non-Gaussian second-order random process by using a collection of observations. These observations made over a denumerable subset of the indexing set of the process are considered to form a set of realizations of a random vector, y, representing a finite-dimensional model of the random process. The estimators of the PC coefficients of y are next deduced by relying on its reduced order representation obtained by employing Karhunen-Loeve decomposition and subsequent use of the maximum-entropy principle, Metropolis-Hastings Markov chain Monte Carlo algorithm and the Rosenblatt transformation. These estimators are found to be maximum likelihood estimators as well as consistent and asymptotically efficient estimators. The computation of the covariance matrix of the associated asymptotic normal distribution of the estimators of these PC coefficients requires evaluation of Fisher information matrix that is evaluated analytically and also estimated by using a sampling technique for the accompanied numerical illustration