H. Prasad

Stochastic Recursive Algorithms for Optimization

The first € price and the £ and

Stochastic Approximation Algorithms

price are net prices, subject to local VAT. Prices indicated with * include VAT for books; the €(D) includes 7% for Germany, the €(A) includes 10% for Austria. Prices indicated with ** include VAT for electronic products; 19% for Germany, 20% for Austria. All prices exclusive of carriage charges. Prices and other details are subject to change without notice. All errors and omissions excepted. S. Bhatnagar, H.L. Prasad, L.A. Prashanth Stochastic Recursive Algorithms for Optimization

Kiefer-Wolfowitz Algorithm

Stochastic approximation algorithms have been one of the main focus areas of research on solution methods for stochastic optimization problems. The Robbins-Monro algorithm [17] is a basic stochastic approximation scheme that has been found to be applicable in a variety of settings that involve finding the roots of a function under noisy observations. We first review in this chapter the Robbins-Monro algorithm and its convergence. In cases where one is interested in optimizing the steady-state system performance, i.e., the objective is a long-run average cost function, multi-timescale variants of the Robbins-Monro algorithm have been found useful. We also review multi-timescale stochastic approximation in this chapter since many of the schemes presented in the later chapters shall involve such algorithms.

General-sum stochastic games

In this chapter, we review the Finite Difference Stochastic Approximation (FDSA) algorithm, also known as Kiefer-Wolfowitz (K-W) algorithm, and some of its variants for finding a local minimum of an objective function. The K-W scheme is a version of the Robbins-Monro stochastic approximation algorithm and incorporates balanced two-sided estimates of the gradient using two objective function measurements for a scalar parameter. When the parameter is an N-dimensional vector, the number of function measurements using this algorithm scales up to 2N. A one-sided variant of this algorithm in the latter case requires N + 1 function measurements. We present the original K-W scheme, first for the case of a scalar parameter, and subsequently for a vector parameter of arbitrary dimension. Variants including the one-sided version are then presented. We only consider here the case when the objective function is a simple expectation over noisy cost samples and not when it has a long-run average form. The latter form of the cost objective would require multi-timescale stochastic approximation, the general case of which was discussed in Chapter 3. Stochastic algorithms for the long-run average cost objectives will be considered in later chapters.

Simultaneous perturbation methods for adaptive labor staffing in service systems

Unlike zero-sum stochastic games, a difficult problem in general-sum stochastic games is to obtain verifiable conditions for Nash equilibria. We show in this paper that by splitting an associated non-linear optimization problem into several sub-problems, characterization of Nash equilibria in a general-sum discounted stochastic games is possible. Using the aforementioned sub-problems, we in fact derive a set of necessary and sufficient verifiable conditions (termed KKT-SP conditions) for a strategy-pair to result in Nash equilibrium. Also, we show that any algorithm which tracks the zero of the gradient of the Lagrangian of every sub-problem provides a Nash strategy-pair.

Algorithms for Constrained Optimization

We consider the problem of optimizing the workforce of a service system. Adapting the staffing levels in such systems is non-trivial due to large variations in workload and the large number of system parameters do not allow for a brute force search. Further, because these parameters change on a weekly basis, the optimization should not take longer than a few hours. Our aim is to find the optimum staffing levels from a discrete high-dimensional parameter set, that minimizes the long run average of the single-stage cost function, while adhering to the constraints relating to queue stability and service-level agreement (SLA) compliance. The single-stage cost function balances the conflicting objectives of utilizing workers better and attaining the target SLAs. We formulate this problem as a constrained parameterized Markov cost process parameterized by the (discrete) staffing levels. We propose novel simultaneous perturbation stochastic approximation (SPSA)-based algorithms for solving the above problem. The algorithms include both first-order as well as second-order methods and incorporate SPSA-based gradient/Hessian estimates for primal descent, while performing dual ascent for the Lagrange multipliers. Both algorithms are online and update the staffing levels in an incremental fashion. Further, they involve a certain generalized smooth projection operator, which is essential to project the continuous-valued worker parameter tuned by our algorithms onto the discrete set. The smoothness is necessary to ensure that the underlying transition dynamics of the constrained Markov cost process is itself smooth (as a function of the continuous-valued parameter): a critical requirement to prove the convergence of both algorithms. We validate our algorithms via performance simulations based on data from five real-life service systems. For the sake of comparison, we also implement a scatter search based algorithm using state-of-the-art optimization tool-kit OptQuest. From the experiments, we observe that both our algorithms converge empirically and consistently outperform OptQuest in most of the settings considered. This finding coupled with the computational advantage of our algorithms make them amenable for adaptive labor staffing in real-life service systems.

A constrained optimization perspective on actor–critic algorithms and application to network routing

This chapter develops algorithms for parameter optimization under multiple functional (inequality) constraints. Both the objective as well as the constraint functions depend on the parameter and are suitable long-run averages. The Lagrangian relaxation technique is used together with multi-timescale stochastic approximation and algorithms based on gradient and Newton SPSA/SF ideas where the afore-mentioned parameter is updated on a faster timescale as compared to the Lagrange parameters are presented.

Smoothed Functional Gradient Schemes

We propose a novel actor-critic algorithm with guaranteed convergence to an optimal policy for a discounted reward Markov decision process. The actor incorporates a descent direction that is motivated by the solution of a certain non-linear optimization problem. We also discuss an extension to incorporate function approximation and demonstrate the practicality of our algorithms on a network routing application.

Deterministic Algorithms for Local Search

We studied SPSA-based gradient estimation techniques in the previous chapter. Along similar lines, we present in this chapter, smoothed functional (SF)-based estimators of the gradient. While SF is also based on simultaneously perturbing the parameter vector, unlike SPSA, for the purpose of perturbation, one uses a smoothing function that possesses certain properties. An alternate view of the SF approach is that the gradient is convolved with a smoothing function, which in turn could possibly help in finding the global minimum of the smoothed objective. We discuss SF-based algorithms where the smoothing is done using Gaussian and Cauchy density functions. The regular SF algorithms require only one measurement of the objective function. We also provide the two-measurement variants of all the algorithms presented.

Newton-Based Simultaneous Perturbation Stochastic Approximation

Most algorithms for stochastic optimization can be viewed as noisy versions of well-known incremental update deterministic optimization algorithms. Hence, we review in this chapter, some of the well-known algorithms for deterministic optimization. We shall study the noisy versions of these algorithms in later chapters.