Qiming Zhao

Near optimal output feedback control of nonlinear discrete-time systems based on reinforcement neural network learning

In this paper, the output feedback based finite-horizon near optimal regulation of nonlinear affine discrete-time systems with unknown system dynamics is considered by using neural networks (NNs) to approximate Hamilton-Jacobi-Bellman (HJB) equation solution. First, a NN-based Luenberger observer is proposed to reconstruct both the system states and the control coefficient matrix. Next, reinforcement learning methodology with actor-critic structure is utilized to approximate the time-varying solution, referred to as the value function, of the HJB equation by using a NN. To properly satisfy the terminal constraint, a new error term is defined and incorporated in the NN update law so that the terminal constraint error is also minimized over time. The NN with constant weights and time-dependent activation function is employed to approximate the time-varying value function which is subsequently utilized to generate the finite-horizon near optimal control policy due to NN reconstruction errors. The proposed scheme functions in a forward-in-time manner without offline training phase. Lyapunov analysis is used to investigate the stability of the overall closed-loop system. Simulation results are given to show the effectiveness and feasibility of the proposed method.

Neural Network-Based Finite-Horizon Optimal Control of Uncertain Affine Nonlinear Discrete-Time Systems

In this paper, the finite-horizon optimal control design for nonlinear discrete-time systems in affine form is presented. In contrast with the traditional approximate dynamic programming methodology, which requires at least partial knowledge of the system dynamics, in this paper, the complete system dynamics are relaxed utilizing a neural network (NN)-based identifier to learn the control coefficient matrix. The identifier is then used together with the actor-critic-based scheme to learn the time-varying solution, referred to as the value function, of the Hamilton-Jacobi-Bellman (HJB) equation in an online and forward-in-time manner. Since the solution of HJB is time-varying, NNs with constant weights and time-varying activation functions are considered. To properly satisfy the terminal constraint, an additional error term is incorporated in the novel update law such that the terminal constraint error is also minimized over time. Policy and/or value iterations are not needed and the NN weights are updated once a sampling instant. The uniform ultimate boundedness of the closed-loop system is verified by standard Lyapunov stability theory under nonautonomous analysis. Numerical examples are provided to illustrate the effectiveness of the proposed method.

Finite-Horizon Near-Optimal Output Feedback Neural Network Control of Quantized Nonlinear Discrete-Time Systems With Input Constraint

The output feedback-based near-optimal regulation of uncertain and quantized nonlinear discrete-time systems in affine form with control constraint over finite horizon is addressed in this paper. First, the effect of input constraint is handled using a nonquadratic cost functional. Next, a neural network (NN)-based Luenberger observer is proposed to reconstruct both the system states and the control coefficient matrix so that a separate identifier is not needed. Then, approximate dynamic programming-based actor-critic framework is utilized to approximate the time-varying solution of the Hamilton-Jacobi-Bellman using NNs with constant weights and time-dependent activation functions. A new error term is defined and incorporated in the NN update law so that the terminal constraint error is also minimized over time. Finally, a novel dynamic quantizer for the control inputs with adaptive step size is designed to eliminate the quantization error overtime, thus overcoming the drawback of the traditional uniform quantizer. The proposed scheme functions in a forward-in-time manner without offline training phase. Lyapunov analysis is used to investigate the stability. Simulation results are given to show the effectiveness and feasibility of the proposed method.

Finite-horizon neural network-based optimal control design for affine nonlinear continuous-time systems

In this paper, the finite-horizon optimal control design for affine nonlinear continuous-time systems in the presence of known system dynamics is presented. A neural network (NN) is utilized to learn the time-varying solution of the Hamilton-Jacobi-Bellman (HJB) equation in an online and forward in time manner. To handle the time varying nature of the value function, the NN with constant weights and time-varying activation function is considered. The update law for tuning the NN weights is derived based on normalized gradient descent approach. To satisfy the terminal constraint and ensure stability, additional terms, one corresponding to the terminal constraint, and the other to stabilize the nonlinear system are added to the novel updating law. A uniformly ultimately boundedness of the non-autonomous closed-loop system is verified by using standard Lyapunov theory. The effectiveness of the proposed method is verified by simulation results.

Finite-horizon optimal control design for uncertain linear discrete-time systems

In this paper, the finite-horizon optimal adaptive control design for linear discrete-time systems with unknown system dynamics by using adaptive dynamic programming (ADP) is presented. In the presence of full state feedback, the terminal state constraint is incorporated in solving the optimal feedback control via the Bellman equation. The optimal regulation of the uncertain linear system is solved in a forward-in-time and online manner without using value and/or policy iterations. Due to the nature of finite horizon, the stability of the closed-loop system is involved but verified by using Lyapunov theory. The effectiveness of the proposed method is verified by simulation results.

Neural network-based finite-horizon approximately optimal control of uncertain affine nonlinear continuous-time systems

This paper develops a novel neural network (NN) based finite-horizon approximate optimal control of nonlinear continuous-time systems in affine form when the system dynamics are complete unknown. First an online NN identifier is proposed to learn the dynamics of the nonlinear continuous-time system. Subsequently, a second NN is utilized to learn the time-varying solution, or referred to as value function, of the Hamilton-Jacobi-Bellman (HJB) equation in an online and forward in time manner. Then, by using the estimated time-varying value function from the second NN and control coefficient matrix from the NN identifier, an approximate optimal control input is computed. To handle time varying value function, a NN with constant weights and time-varying activation function is considered and a suitable NN update law is derived based on normalized gradient descent approach. Further, in order to satisfy terminal constraint and ensure stability within the fixed final time, two extra terms, one corresponding to terminal constraint, and the other to stabilize the nonlinear system are added to the novel update law of the second NN. No initial stabilizing control is required. A uniformly ultimately boundedness of the closed-loop system is verified by using standard Lyapunov theory.

Adaptive dynamic programming-based state quantized networked control system without value and/or policy iterations

In this paper, the Bellman equation is used to solve the stochastic optimal control of unknown linear discrete-time system with communication imperfections including random delays, packet losses and quantization. A dynamic quantizer for the sensor measurements is proposed which essentially provides system states to the controller. To eliminate the effect of the quantization error, the dynamics of the quantization error bound and an update law for tuning its range are derived. Subsequently, by using adaptive dynamic programming technique, the infinite horizon optimal regulation of the uncertain NCS is solved in a forward-in-time manner without using value and/or policy iterations by using Q-function and reinforcement learning. The asymptotic stability of the closed-loop system is verified by standard Lyapunov stability theory. Finally, the effectiveness of the proposed method is verified by simulation results.

Finite-horizon optimal adaptive neural network control of uncertain nonlinear discrete-time systems

In this paper, finite-horizon optimal control design for affine nonlinear discrete-time systems with totally unknown system dynamics is presented. First, a novel neural network (NN)-based identifier is utilized to learn the control coefficient matrix. This identifier is used together with the action-critic-based scheme to learn the time-varying solution, or referred to as value function, of the Hamilton-Jacobi-Bellman (HJB) equation in an online and forward in time manner. To handle the time varying nature of the value function, NNs with constant weights and time-varying activation functions are considered. To satisfy the terminal constraint, an additional term is added to the novel updating law. The uniformly ultimately boundedness of the closed-loop system is demonstrated by using standard Lyapunov theory. The effectiveness of the proposed method is verified by simulation results.

Fixed final-time near optimal regulation of nonlinear discrete-time systems in affine form using output feedback

In this paper, the fixed final-time near optimal output regulation of affine nonlinear discrete-time systems with unknown system dynamics is considered. First, a neural network (NN)-based observer is proposed to reconstruct both the system state vector and control coefficient matrix. Next, actor-critic structure is utilized to approximate the time-varying solution of the Hamilton-Jacobi-Bellman (HJB) equation or value function. To satisfy the terminal constraint, a new error term is defined and incorporated in the NN update law so that the terminal constraint error is also minimized over time. A NN with constant weights and time-dependent activation function is employed to approximate the time-varying value function which subsequently is utilized to generate the fixed final time near optimal control policy due to NN reconstruction errors. The proposed scheme functions in a forward-in-time manner without offline training phase. The effectiveness of the proposed method is verified via simulation.

Optimal adaptive controller scheme for uncertain quantized linear discrete-time system

In this paper, the Bellman equation is used to solve the optimal adaptive control of quantized linear discrete-time system with unknown dynamics. To mitigate the effect of the quantization errors, the dynamics of the quantization error bound and an update law for tuning the range of the dynamic quantizer are derived. Subsequently, by using adaptive dynamic programming technique, the infinite horizon optimal regulation problem of the uncertain quantized linear discrete-time system is solved in a forward-in-time manner without using value and/or policy iterations. The asymptotic stability of the closed-loop system is verified by standard Lyapunov stability approach in the presence of state and input quantizers. The effectiveness of the proposed method is verified via simulation.