Qichao Zhang

Data-Based Adaptive Critic Designs for Nonlinear Robust Optimal Control With Uncertain Dynamics

In this paper, the infinite-horizon robust optimal control problem for a class of continuous-time uncertain nonlinear systems is investigated by using data-based adaptive critic designs. The neural network identification scheme is combined with the traditional adaptive critic technique, in order to design the nonlinear robust optimal control under uncertain environment. First, the robust optimal controller of the original uncertain system with a specified cost function is established by adding a feedback gain to the optimal controller of the nominal system. Then, a neural network identifier is employed to reconstruct the unknown dynamics of the nominal system with stability analysis. Hence, the data-based adaptive critic designs can be developed to solve the Hamilton-Jacobi-Bellman equation corresponding to the transformed optimal control problem. The uniform ultimate boundedness of the closed-loop system is also proved by using the Lyapunov approach. Finally, two simulation examples are presented to illustrate the effectiveness of the developed control strategy.

Experience Replay for Optimal Control of Nonzero-Sum Game Systems With Unknown Dynamics

In this paper, an approximate online equilibrium solution is developed for an N-player nonzero-sum (NZS) game systems with completely unknown dynamics. First, a model identifier based on a three-layer neural network (NN) is established to reconstruct the unknown NZS games systems. Moreover, the identifier weight vector is updated based on experience replay technique which can relax the traditional persistence of excitation condition to a simplified condition on recorded data. Then, the single-network adaptive dynamic programming (ADP) with experience replay algorithm is proposed for each player to solve the coupled nonlinear Hamilton-Jacobi (HJ) equations, where only the critic NN weight vectors are required to tune for each player. The feedback Nash equilibrium is provided by the solution of the coupled HJ equations. Based on the experience replay technique, a novel critic NN weights tuning law is proposed to guarantee the stability of the closed-loop system and the convergence of the value functions. Furthermore, a Lyapunov-based stability analysis shows that the uniform ultimate boundedness of the closed-loop system is achieved. Finally, two simulation examples are given to verify the effectiveness of the proposed control scheme.

Event-based input-constrained nonlinear H∞ state feedback with adaptive critic and neural implementation

Abstract In this paper, the continuous-time input-constrained nonlinear H ∞ state feedback control under event-based environment is investigated with adaptive critic designs and neural network implementation. The nonlinear H ∞ control issue is regarded as a two-player zero-sum game that requires solving the Hamilton–Jacobi–Isaacs equation and the adaptive critic learning (ACL) method is adopted toward the event-based constrained optimal regulation. The novelty lies in that the event-based design framework is combined with the ACL technique, thereby carrying out the input-constrained nonlinear H ∞ state feedback via adopting a non-quadratic utility function. The event-based optimal control law and the time-based worst-case disturbance law are derived approximately, by training an artificial neural network called a critic and eventually learning the optimal weight vector. Under the action of the event-based state feedback controller, the closed-loop system is constructed with uniformly ultimately bounded stability analysis. Simulation studies are included to verify the theoretical results as well as to illustrate the event-based H ∞ control performance.

Event-Based Robust Control for Uncertain Nonlinear Systems Using Adaptive Dynamic Programming

In this paper, the robust control problem for a class of continuous-time nonlinear system with unmatched uncertainties is investigated using an event-based control method. First, the robust control problem is transformed into a corresponding optimal control problem with an augmented control and an appropriate cost function. Under the event-based mechanism, we prove that the solution of the optimal control problem can asymptotically stabilize the uncertain system with an adaptive triggering condition. That is, the designed event-based controller is robust to the original uncertain system. Note that the event-based controller is updated only when the triggering condition is satisfied, which can save the communication resources between the plant and the controller. Then, a single network adaptive dynamic programming structure with experience replay technique is constructed to approach the optimal control policies. The stability of the closed-loop system with the event-based control policy and the augmented control policy is analyzed using the Lyapunov approach. Furthermore, we prove that the minimal intersample time is bounded by a nonzero positive constant, which excludes Zeno behavior during the learning process. Finally, two simulation examples are provided to demonstrate the effectiveness of the proposed control scheme.In this paper, the robust control problem for a class of continuous-time nonlinear system with unmatched uncertainties is investigated using an event-based control method. First, the robust control problem is transformed into a corresponding optimal control problem with an augmented control and an appropriate cost function. Under the event-based mechanism, we prove that the solution of the optimal control problem can asymptotically stabilize the uncertain system with an adaptive triggering condition. That is, the designed event-based controller is robust to the original uncertain system. Note that the event-based controller is updated only when the triggering condition is satisfied, which can save the communication resources between the plant and the controller. Then, a single network adaptive dynamic programming structure with experience replay technique is constructed to approach the optimal control policies. The stability of the closed-loop system with the event-based control policy and the augmented control policy is analyzed using the Lyapunov approach. Furthermore, we prove that the minimal intersample time is bounded by a nonzero positive constant, which excludes Zeno behavior during the learning process. Finally, two simulation examples are provided to demonstrate the effectiveness of the proposed control scheme.

Data-driven adaptive dynamic programming for continuous-time fully cooperative games with partially constrained inputs

In this paper, the fully cooperative game with partially constrained inputs in the continuous-time Markov decision process environment is investigated using a novel data-driven adaptive dynamic programming method. First, the model-based policy iteration algorithm with one iteration loop is proposed, where the knowledge of system dynamics is required. Then, it is proved that the iteration sequences of value functions and control policies can converge to the optimal ones. In order to relax the exact knowledge of the system dynamics, a model-free iterative equation is derived based on the model-based algorithm and the integral reinforcement learning. Furthermore, a data-driven adaptive dynamic programming is developed to solve the model-free equation using generated system data. From the theoretical analysis, we prove that this model-free iterative equation is equivalent to the model-based iterative equations, which means that the data-driven algorithm can approach the optimal value function and control policies. For the implementation purpose, three neural networks are constructed to approximate the solution of the model-free iteration equation using the off-policy learning scheme after the available system data is collected in the online measurement phase. Finally, two examples are provided to demonstrate the effectiveness of the proposed scheme.

Multi-task learning for dangerous object detection in autonomous driving☆

Abstract Recently, autonomous driving has been extensively studied and has shown considerable promise. Vision-based dangerous object detection is a crucial technology of autonomous driving. In previous work, dangerous object detection is generally formulated as a typical object detection problem and a distance-based danger assessment problem, separately. These two problems are usually dealt with using two independent models. In fact, vision-based object detection and distance prediction present prominent visual relationship. The objects with different distance to the camera have different attributes (pose, size and definition), which are very worthy to be exploited for dangerous object detection. However, these characteristics are usually ignored in previous work. In this paper, we propose a novel multi-task learning (MTL) method to jointly model object detection and distance prediction with a Cartesian product-based multi-task combination strategy. Furthermore, we mathematically prove that the proposed Cartesian product-based combination strategy is more optimal than the linear multi-task combination strategy that is usually used in MTL models, when the multi-task itself is not independent. Systematic experiments show that the proposed approach consistently achieves better object detection and distance prediction performances compared to both the single-task and multi-task dangerous object detection methods.

Policy gradient methods with Gaussian process modelling acceleration

Policy gradient algorithm is often used to deal with the continuous control problems. But as a model-free algorithm, it suffers from the low data efficiency and long learning phase. In this paper, a policy gradient with Gaussian process modelling (PGGPM) algorithm is proposed to accelerate learning process. The system model is approximated by Gaussian process in an incremental way, which is used to explore state action space virtually by generating imaginary samples. Both the real and imaginary samples are used to train the actor and critic networks. Finally, we apply our algorithm to two experiments to verify that Gaussian process can accurately fit system model and the supplementary imaginary samples can speed up the learning phase.

Model-free reinforcement learning for nonlinear zero-sum games with simultaneous explorations.

In this paper, the continuous-time unknown nonlinear zero-sum game is investigated using a model-free online learning method. First, motivated by model-based policy iteration, an iterative equation without any knowledge of system dynamics is derived by introducing simultaneous explorations. Then, the model-free reinforcement learning based on the derived iterative equation is developed to approach the solution of the Hamilton-Jacobi-Isaacs equation. For the online implementation purpose, three neural networks are constructed to approach the value function, control and disturbance policies, respectively. Finally, a simulation example is provided to demonstrate the effectiveness of the proposed scheme.

Event-Triggered H∞ Control for Continuous-Time Nonlinear System

In this paper, the H∞ optimal control for a class of continuous-time nonlinear systems is investigated using event-triggered method. First, the H∞ optimal control problem is formulated as a two-player zero-sum differential game. Then, an adaptive triggering condition is derived for the closed loop system with an event-triggered control policy and a time-triggered disturbance policy. For implementation purpose, the event-triggered concurrent learning algorithm is proposed, where only one critic neural network is required. Finally, an illustrated example is provided to demonstrate the effectiveness of the proposed scheme.

Off-Policy Reinforcement Learning for Partially Unknown Nonzero-Sum Games

In this paper, the optimal control problem of nonzero-sum (NZS) games with partially unknown dynamics is investigated. The off-policy reinforcement learning (RL) method is proposed to approximate the solution of the coupled Hamilton-Jacobi (HJ) equations. A single critic network structure for each player is constructed using neural network (NN) technique. To improve the applicability of the off-policy RL method, the tuning laws of critic weights are designed based on the offline learning and online learning methods, respectively. The simulation study demonstrates the effectiveness of the proposed algorithms.