Q. Zhang

Discrete-time singularly perturbed Markov chains: aggregation, occupation measures, and switching diffusion limit

This work is devoted to asymptotic properties of singularly perturbed Markov chains in discrete time. The motivation stems from applications in discrete-time control and optimization problems, manufacturing and production planning, stochastic networks, and communication systems, in which finite-state Markov chains are used to model large-scale and complex systems. To reduce the complexity of the underlying system, the states in each recurrent class are aggregated into a single state. Although the aggregated process may not be Markovian, its continuous-time interpolation converges to a continuous-time Markov chain whose generator is a function determined by the invariant measures of the recurrent states. Sequences of occupation measures are defined. A mean square estimate on a sequence of unscaled occupation measures is obtained. Furthermore, it is proved that a suitably scaled sequence of occupation measures converges to a switching diffusion.

Optimal filtering of discrete-time hybrid systems

This paper is concerned with discrete-time hybrid filtering of linear non-Gaussian systems coupled by a hidden switching process. An optimal control approach is used to derive a finite-dimensional recursive filter which is optimal in the sense of the most probable trajectory estimate. Models with unknown switching distributions are considered. Extensions to nonlinear hybrid systems are given. Numerical examples are considered and computational experiments are reported. These examples demonstrate that our filtering scheme outperforms popular filtering schemes available in the literature.

Stochastic Optimization Methods for Buying-Low-and-Selling-High Strategies

Abstract This article is concerned with a numerical method using stochastic approximation approach for an optimal trading (buy and sell) strategy. The underlying asset price is governed by a mean-reverting stochastic process. The objective is to buy and sell the asset so as to maximize an overall expected return. One of the advantages of our approach is that the underlying asset is model free. Only mean reversion is required. Slippage cost is imposed on each transaction. Convergence of the algorithms is provided. Numerical examples are reported to demonstrate the results.

Stock Liquidation via Stochastic Approximation Using NASDAQ Daily and Intra-Day Data

By focusing on computational aspects, this work is concerned with numerical methods for stock selling decision using stochastic approximation methods. Concentrating on the class of decisions depending on threshold values, an optimal stopping problem is converted to a parametric stochastic optimization problem. The algorithms are model free and are easily implementable on-line. Convergence of the algorithms is established, second moment bound of estimation error is obtained, and escape probability from a neighborhood of the true parameter is also derived. Numerical examples using both daily closing prices and intra-day data are provided to demonstrate the performance of the algorithms.

Constrained Stochastic Estimation Algorithms for a Class of Hybrid Stock Market Models

This paper is concerned with a class of hybrid stock market models, in which both the return rate and the volatility depend on a hidden, continuous-time Markov chain with a finite state space. One of the crucial issues is to estimate the generator of the underlying Markov chain. We develop a stochastic optimization procedure for this task, prove its convergence, and establish the rate of convergence. Numerical tests are carried out via simulation as well as using real market data. In addition, we demonstrate how to use the estimated generator in making stock liquidation decisions.

Controlled Markov chains with weak and strong interactions: asymptotic optimality and applications to manufacturing

This paper deals with the asymptotic optimality of a stochastic dynamic system driven by a singularly perturbed Markov chain with finite state space. The states of the Markov chain belong to several groups such that transitions among the states within each group occur much more frequently than transitions among the states in different groups. Aggregating the states of the Markov chain leads to a limit control problem, which is obtained by replacing the states in each group by the corresponding average distribution. The limit control problem is simpler to solve as compared with the original one. A nearly-optimal solution for the original problem is constructed by using the optimal solution to the limit problem. To demonstrate, the suggested approach of asymptotic optimal control is applied to examples of manufacturing systems of production planning.

Periodic maintenance and repair rate control in stochastic manufacturing systems

In this paper, we consider a periodic preventive maintenance, repair, and production model of a flexible manufacturing system with failure-prone machines, where the control variables are the repair rate and production rate. We use periodic preventive maintenance to reduce the machine failure rates and improve the productivity of the system. One of the distinct features of the model is that the repair rate is adjustable. Our objective is to choose a control process that minimizes the total cost of inventory/shortage, production, repair, and maintenance. Under suitable conditions, we show that the value function is locally Lipschitz and satisfies an Hamilton-Jacobi-Bellman equation. A sufficient condition for optimal control is obtained. Since analytic solutions are rarely available, we design an algorithm to approximate the optimal control problem. To demonstrate the performance of the numerical method, an example is presented.

CONTROL OF SINGULARLY PERTURBED MARKOV CHAINS: A NUMERICAL STUDY

This work is devoted to numerical studies of nearly optimal controls of systems driven by singularly perturbed Markov chains. Our approach is based on the ideas of hierarchical controls applicable to many large-scale systems. A discrete-time linear quadratic control problem is examined. Its corresponding limit system is derived. The associated asymptotic properties and near optimality are demonstrated by numerical examples. Numerical experiments for a continuous-time hybrid linear quadratic regulator with Gaussian disturbances and a discrete-time Markov decision process are also presented. The numerical results have not only supported our theoretical findings but also provided insights for further applications.

A class of hybrid market models: simulation, identification, and estimation

Concerns the modeling of stock market using hybrid processes. By hybrid processes, we mean such processes that involve continuous dynamics and discrete events. The theory developed is based on the hybrid geometric Brownian motion (HGBM). We use both simulation and real market data to demonstrate that the hybrid model better describes the market and is more suitable for applications. Once the generator of the Markov chain is specified, the system is determined. To identify or to estimate the generator, we propose and develop several procedures including nonlinear regression model and stochastic optimization type of procedures.

Approximating the Optimal Threshold Levels under Robustness Cost Criteria for Stochastic Manufacturing Systems

Abstract A stochastic manufacturing system under a long run average cost arid minimax (robust control) criteria is considered in this work. In addition to minimizing the production cost, we also take the maintenance effor : into consideration. Furthermore, as an unknown bounded process, the demand can be either deterministic or random. In lieu of finding the optimal control of the system, our attention is focused on the threshold type of control polices. The underlying problem is then converted to an optimization problem. Our objective is to find the optimal threshold levels under worst case demand fluctuation. By means of stochastic approximation methods, we construct recursive algorithms to locate the optimal threshold levels. Under suitable conditions, the algorithm is shown to be convergent. Examples are considered; the numerical experimental results are also provided.