Shubhendu Bhasin

A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems

An online adaptive reinforcement learning-based solution is developed for the infinite-horizon optimal control problem for continuous-time uncertain nonlinear systems. A novel actor-critic-identifier (ACI) is proposed to approximate the Hamilton-Jacobi-Bellman equation using three neural network (NN) structures-actor and critic NNs approximate the optimal control and the optimal value function, respectively, and a robust dynamic neural network identifier asymptotically approximates the uncertain system dynamics. An advantage of using the ACI architecture is that learning by the actor, critic, and identifier is continuous and simultaneous, without requiring knowledge of system drift dynamics. Convergence of the algorithm is analyzed using Lyapunov-based adaptive control methods. A persistence of excitation condition is required to guarantee exponential convergence to a bounded region in the neighborhood of the optimal control and uniformly ultimately bounded (UUB) stability of the closed-loop system. Simulation results demonstrate the performance of the actor-critic-identifier method for approximate optimal control.

Approximate optimal trajectory tracking for continuous-time nonlinear systems

Adaptive dynamic programming has been investigated and used as a method to approximately solve optimal regulation problems. However, the extension of this technique to optimal tracking problems for continuous-time nonlinear systems has remained a non-trivial open problem. The control development in this paper guarantees ultimately bounded tracking of a desired trajectory, while also ensuring that the enacted controller approximates the optimal controller.

RISE-Based Adaptive Control of a Control Affine Uncertain Nonlinear System With Unknown State Delays

A continuous robust adaptive control method is designed for a class of uncertain nonlinear systems with unknown constant time-delays in the states. Specifically, a robust adaptive control method and a delay-free gradient-based desired compensation adaptation law (DCAL) are utilized to compensate for unknown time-delays, linearly parameterizable uncertainties, and additive bounded disturbances for a general nonlinear system. Despite these disturbances, a Lyapunov Krasovskii-based analysis is used to conclude that the system output asymptotically tracks a desired time varying bounded trajectory.

Composite Adaptation for Neural Network-Based Controllers

This paper presents a novel approach to design a composite adaptation law for neural networks that uses both the system tracking errors and a prediction error containing parametric information by devising an innovative swapping procedure that uses the recently developed Robust Integral of the Sign of the Error (RISE) feedback method. Semi-global asymptotic tracking is proven for an Euler-Lagrange system.

Nonlinear two-player zero-sum game approximate solution using a Policy Iteration algorithm

An approximate online solution is developed for a two-player zero-sum game subject to continuous-time nonlinear uncertain dynamics and an infinite horizon quadratic cost. A novel actor-critic-identifier (ACI) structure is used to implement the Policy Iteration (PI) algorithm, wherein a robust dynamic neural network (DNN) is used to asymptotically identify the uncertain system, and a critic NN is used to approximate the value function. The weight update laws for the critic NN are generated using a gradient-descent method based on a modified temporal difference error, which is independent of the system dynamics. This method finds approximations of the optimal value function, and the saddle point feedback control policies. These policies are computed using the critic NN and the identifier DNN and guarantee uniformly ultimately bounded (UUB) stability of the closed-loop system. The actor, critic and identifier structures are implemented in real-time, continuously and simultaneously.

Nonlinear control of an autonomous underwater vehicle: A RISE-based approach

Autonomous and remotely operated marine vehicles such as ships and submarines are becoming a key component in several aspects of maritime industry and defense. This paper explores the development of a nonlinear controller for a fully actuated autonomous underwater vehicle (AUV) using a robust integral of the sign of the error (RISE) feedback term with a neural network (NN) based feedforward term to achieve semi-global asymptotic tracking results in the presence of complete model uncertainty and unknown disturbances. A simulation is provided to demonstrate the proposed controller on an experimentally validated AUV model.

Robust Identification-Based State Derivative Estimation for Nonlinear Systems

A robust identification-based state derivative estimation method for uncertain nonlinear systems is developed. The identifier architecture consists of a recurrent multilayer dynamic neural network which approximates the system dynamics online, and a continuous robust feedback Robust Integral of the Sign of the Error (RISE) term which accounts for modeling errors and exogenous disturbances. Numerical simulations provide comparisons with existing robust derivative estimation methods including: a high gain observer, a 2-sliding mode robust exact differentiator, and numerical differentiation methods, such as backward difference and central difference.

A model-free robust policy iteration algorithm for optimal control of nonlinear systems

An online model-free solution is developed for the infinite-horizon optimal control problem for continuous-time nonlinear systems. A novel actor-critic-identifier (ACI) structure is used to implement the Policy Iteration algorithm, wherein two neural network structures are used - a robust dynamic neural network (DNN) to asymptotically identify the uncertain system with additive disturbances, and a critic NN to approximate the value function. The weight update laws for the critic NN are generated using a gradient-descent method based on a modified temporal difference error, which is independent of the system dynamics. The optimal control law (or the actor) is computed using the critic NN and the identifier DNN. Uniformly ultimately bounded (UUB) stability of the closed-loop system is guaranteed. The actor, critic and identifier structures are implemented in real-time, continuously and simultaneously.

Predictor-based control for an uncertain Euler-Lagrange system with input delay

Control of nonlinear systems with actuator delay is a challenging problem because of the need to develop some form of prediction of the nonlinear dynamics. The problem becomes more difficult for systems with uncertain dynamics. In this paper, tracking controllers are developed for an Euler-Lagrange system with time-delayed actuation, parametric uncertainty, and additive bounded disturbances. One controller is developed under the assumption that the inertia is known, and a second controller is developed when the inertia is unknown. For each case a predictor-like method is developed to address the time delay in the control input. Lyapunov-Krasovskii functionals are used within a Lyapunov-based stability analysis to prove semi-global uniformly ultimately bounded tracking.

Adaptive neural network satellite attitude control in the presence of inertia and CMG actuator uncertainties

A neural network-based adaptive attitude tracking controller is developed in this paper, which achieves attitude tracking in the presence of parametric uncertainty, nonlinear actuator disturbances, and unmodeled external disturbance torques, which do not satisfy the linear-in-the-parameters assumption (i.e., non-LP). The satellite control torques are produced by means of a cluster of control moment gyroscopes (CMGs), which have uncertain dynamic and static friction in the gimbals in addition to unknown electromechanical disturbances. Some challenges encountered in the control design are that the control input is premultiplied by a non-square, time-varying, nonlinear, uncertain matrix and is embedded in a discontinuous nonlinearity. Controller performance is proven via Lyapunov stability analysis.