Rajneesh Sharma

Review article: Synergizing reinforcement learning and game theory-A new direction for control

Reinforcement learning (RL) has now evolved as a major technique for adaptive optimal control of nonlinear systems. However, majority of the RL algorithms proposed so far impose a strong constraint on the structure of environment dynamics by assuming that it operates as a Markov decision process (MDP). An MDP framework envisages a single agent operating in a stationary environment thereby limiting the scope of application of RL to control problems. Recently, a new direction of research has focused on proposing Markov games as an alternative system model to enhance the generality and robustness of the RL based approaches. This paper aims to present this new direction that seeks to synergize broad areas of RL and Game theory, as an interesting and challenging avenue for designing intelligent and reliable controllers. First, we briefly review some representative RL algorithms for the sake of completeness and then describe the recent direction that seeks to integrate RL and game theory. Finally, open issues are identified and future research directions outlined.

A Markov Game-Adaptive Fuzzy Controller for Robot Manipulators

This paper develops an adaptive fuzzy controller for robot manipulators using a Markov game formulation. The Markov game framework offers a promising platform for robust control of robot manipulators in the presence of bounded external disturbances and unknown parameter variations. We propose fuzzy Markov games as an adaptation of fuzzy Q-learning (FQL) to a continuous-action variation of Markov games, wherein the reinforcement signal is used to tune online the conclusion part of a fuzzy Markov game controller. The proposed Markov game-adaptive fuzzy controller uses a simple fuzzy inference system (FIS), is computationally efficient, generates a swift control, and requires no exact dynamics of the robot system. To illustrate the superiority of Markov game-adaptive fuzzy control, we compare the performance of the controller against a) the Markov game-based robust neural controller, b) the reinforcement learning (RL)-adaptive fuzzy controller, c) the FQL controller, d) the Hinfin theory-based robust neural game controller, and e) a standard RL-based robust neural controller, on two highly nonlinear robot arm control problems of i) a standard two-link rigid robot arm and ii) a 2-DOF SCARA robot manipulator. The proposed Markov game-adaptive fuzzy controller outperformed other controllers in terms of tracking errors and control torque requirements, over different desired trajectories. The results also demonstrate the viability of FISs for accelerating learning in Markov games and extending Markov game-based control to continuous state-action space problems.

EMD and ANN based intelligent fault diagnosis model for transmission line

In the presented work, an intelligent model for fault classification of a transmission line is proposed. Ten different types of faults (LAG, LBG, LCG, LABG, LBCG, LCAG, LAB, LBC, LCA and LABC) have been considered along with one healthy condition on a simulated transmission line system. Post fault current signatures have been used for feature extraction for further study. Empirical Mode Decomposition (EMD) method is used to decompose post fault current signals into Intrinsic Mode Functions (IMFs). These IMFs are used as input variables to an artificial neural network (ANN) based intelligent fault classification model. Relief Attribute Evaluator with Ranker search method is used to select the most relevant input variables for fault classification of a three-phase transmission line. Proposed approach is able to select most relevant input variables and gives better result than other combinations. Ours is a first attempt at using EMD for feature selection in fault classification of transmission lines.

Hybrid Game Strategy in Fuzzy Markov-Game-Based Control

This paper proposes a novel control approach that incorporates a hybrid game strategy in Markov-game-based fuzzy control. Specifically, we aim at designing a ldquosafe and universally consistentrdquo controller that exhibits an ability to maintain performance against large disturbance and environment variations. The proposed hybrid control is a convex combination (based on experiential information) of ldquoa variation of cautious fictitious playrdquo approach and the ldquominimaxrdquo control approach implemented on a fuzzy Markov game platform. We show analytical convergence of Markov-game-based control in the presence of bounded external disturbances, and extend the analysis to show convergence of the proposed Markov-game-based hybrid control approach. Controller simulation and comparison against baseline Markov game fuzzy control and fuzzy Q -learning control on a highly nonlinear two-link robot brings out the superiority of the approach in handling severe environment and disturbance variations over different desired trajectories. This paper illustrates the possibility of obtaining ldquouniversal consistency,rdquo i.e., reasonable performance against severe environment and disturbance variations, by hybridizing ldquocautious fictitious playrdquo with ldquominimaxrdquo approaches in Markov-game-based control.

Bayesian-Game-Based Fuzzy Reinforcement Learning Control for Decentralized POMDPs

This paper proposes a Bayesian-game-based fuzzy reinforcement learning (RL) controller for decentralized partially observable Markov decision processes (Dec-POMDPs). Dec-POMDPs have recently emerged as a powerful platform for optimizing multiagent sequential decision making in partially observable stochastic environments. However, finding exact optimal solutions to a Dec-POMDP is provably intractable (NEXP-complete), necessitating the use of approximate/suboptimal solution approaches. This approach proposes an approximate solution by employing fuzzy inference systems (FISs) in a game-based RL setting. It uses the powerful universal approximation capability of fuzzy systems to compactly represent a Dec-POMDP as a fuzzy Dec-POMDP, allowing the controller to progressively learn and update an approximate solution to the underlying Dec-POMDP. The proposed controller envisages an FIS-based RL controller for Dec-POMDPs modeled as a sequence of Bayesian games (BGs). We implement the proposed controller for two scenarios: 1) Dec-POMDPs with free communication between agents; and 2) Dec-POMDPs without communication. We empirically evaluate the proposed approach on three standard benchmark problems: 1) multiagent tiger; 2) multiaccess broadcast channel; and 3) recycling robot. Simulation results and comparative evaluation against other Dec-POMDP solution approaches elucidate the effectiveness and feasibility of employing FIS-based game-theoretic RL for designing Dec-POMDP controllers.

Finite element analysis for mechanical characterization of 4D inplane carbon/carbon composite with imperfect microstructure

Finite element mesh of multi-directional 4D carbon/carbon (C/C) composite was reconstructed from 2D images obtained by X-ray tomography. Thus, imperfections in the composite such as voids, misalignment and cross-section distortion of the fibre bundles were directly incorporated in the finite element mesh. 2D images of the composite were also used for the characterization of the porosity in the composite. The effect of these micro structural imperfections was studies by assuming perfect bonding at the bundle/matrix interface. The initial mechanical properties of the composite were obtained from unit cell analysis using asymptotic homogenization and moduli in x, y and z directions were 39, 25 and 44 GPa. However, matrix and bundle/matrix interfacial cracks were also clearly visible in the X-ray tomographic images. Later, the effects of debonding was incorporated by using frictional cohesive interaction at bundle/matrix interfaces and matrix cracking was modeled by degrading the elastic properties of matrix. Final, the response of the composite was studied under six individual load cases.

Fuzzy Lyapunov Reinforcement Learning for Non Linear Systems

We propose a fuzzy reinforcement learning (RL) based controller that generates a stable control action by lyapunov constraining fuzzy linguistic rules. In particular, we attempt at lyapunov constraining the consequent part of fuzzy rules in a fuzzy RL setup. Ours is a first attempt at designing a linguistic RL controller with lyapunov constrained fuzzy consequents to progressively learn a stable optimal policy. The proposed controller does not need system model or desired response and can effectively handle disturbances in continuous state-action space problems. Proposed controller has been employed on the benchmark Inverted Pendulum (IP) and Rotational/Translational Proof-Mass Actuator (RTAC) control problems (with and without disturbances). Simulation results and comparison against a) baseline fuzzy Q learning, b) Lyapunov theory based Actor-Critic, and c) Lyapunov theory based Markov game controller, elucidate stability and viability of the proposed control scheme.

Game theoretic Lyapunov fuzzy control for Inverted Pendulum

In this paper we propose a game theoretic Lyapunov fuzzy controller which is both safe and stable. We attempt to optimize a reinforcement learning based controller using Markov games, simultaneously hybridizing it with a Lyapunov theory based control, to impart stability. Our proposed technique results in an RL based game theoretic, adaptive, self learning, optimal fuzzy controller, which is robust and has guaranteed stability for non linear systems. Proposed controller is an “annealed” hybrid of Fuzzy Markov games and Lyapunov theory based control approaches. We have used fuzzy systems as function approximator for a continuous state action space implementation. We test the controller on the standard Inverted Pendulum Control (IPC) problem. Simulation results bring out supremacy of the designed control strategy over baseline Fuzzy Markov game based controller.

Fractional Order PID Control using Ant Colony Optimization

An Ant Colony Optimization (ACO) based Fractional Fuzzy PID controller is proposed in this paper. The resulting controller Ant Colony Fractional Fuzzy PID (AFrFPID) Controller incorporates the characteristics of the Ant Colony System and Fuzzy Control for controlling integer and fractional order plants. Fractional Order PID (FOPID) controllers show better performance for systems that have non-linear and time varying variables. However; the complexity of designing FOPID parameters is increased due to increase in tuning parameters (from 3 to 5). To obtain an initial estimate of these five parameters; the bio-inspired ACO algorithm is used. Ant Colony Optimization is a population based meta-heuristic technique which from the behavior of real ant colonies to find solutions to discrete optimization problems. Fuzzy Control is used to further fine tune the parameters for better control. MATLAB Simulations are presented and the performance of the AFrFPID controller is validated.

A Lyapunov theory based adaptive fuzzy learning control for robotic manipulator

This work proposes to amalgamate the Fuzzy Q learning (FQL) with Lyapunov theory based control resulting in a controller with guaranteed stability for dynamic trajectory tracking control of robotic manipulators. FQL algorithm combines reinforcement learning (RL) approach with fuzzy modeling; however, it fails to address the stability issue of the designed controller. Proposed approach is specifically aimed at addressing this shortcoming. Proposed controller combines powerful generalization and learning capability of fuzzy systems with Lyapunov theory based control that guarantees stability. To demonstrate the viability and effectiveness of the Lyapunov theory based adaptive fuzzy learning approach over basic FQL methodology, we compare the performance of the controller on two degrees of freedom standard two link robot manipulator which is a highly coupled, time varying nonlinear system. Results validate that the proposed hybrid controller indeed leads to a superior performance in terms of both input torques at each joint and tracking accuracy in presence of external disturbances and payload mass variations.