Jose Manuel Lopez-Guede

Transfer learning with Partially Constrained Models: Application to reinforcement learning of linked multicomponent robot system control

Abstract Transfer learning is a hierarchical approach to reinforcement learning of complex tasks modeled as Markov Decision Processes. The learning results on the source task are used as the starting point for the learning on the target task. In this paper we deal with a hierarchy of constrained systems, where the source task is an under-constrained system, hence called the Partially Constrained Model (PCM). Constraints in the framework of reinforcement learning are dealt with by state-action veto policies. We propose a theoretical background for the hierarchy of training refinements, showing that the effective action repertoires learnt on the PCM are maximal, and that the PCM-optimal policy gives maximal state value functions. We apply the approach to learn the control of Linked Multicomponent Robotic Systems using Reinforcement Learning. The paradigmatic example is the transportation of a hose. The system has strong physical constraints and a large state space. Learning experiments in the target task are realized over an accurate but computationally expensive simulation of the hose dynamics. The PCM is obtained simplifying the hose model. Learning results of the PCM Transfer Learning show an spectacular improvement over conventional Q-learning on the target task.

IMPROVING THE CONTROL OF SINGLE ROBOT HOSE TRANSPORT

Single robot hose transport is a limit case of linked multicomponent robotic systems, where one robot moves the tip of a hose to a desired position. The interaction between the passive, flexible hose and the robot introduces highly nonlinear effects in the systems dynamics, requiring innovative control design approaches, such as reinforcement learning. This article improves previous approaches to this problem by introducing a novel reinforcement learning algorithm (TRQ-learning) and a new system state definition for the autonomous derivation of the hose–robot control algorithm. Computational experiments based on accurate geometrically exact dynamic splines hose dynamics simulations show the improvement obtained.

Evaluation of the performance of new extremum seeking control algorithm to locate accurately the peaks on multimodal functions

A new Extremum Seeking Control (ESC) scheme is proposed in this paper for multimodal function with two dimension (2D), named as ESC 2D — scheme. This ESC 2D-scheme is a Single Input — Double Outputs (DISO) control system that combine two ESC 1D-schemes of asymptotic perturbed type. Each ESC 1D-scheme has two control loop. The scanning loop generates the signal that sweep the searching space in order to locate the desired peak on the multimodal pattern. The tuning parameter (k2) will define the space of searching. The locating loop generates the gradient signal that will find accurately this peak. The tuning parameter (k1) will define the searching speed under the constraint of stability of the ESC loop. The performance obtained for the ESC 2D scheme proposed is evaluated on multimodal benchmark patterns.

Linked multicomponent robotic systems: basic assessment of linking element dynamical effect

The Linked Multicomponent Robotic Systems are characterized by the existence of a non-rigid linking element This linking element can produce many dynamical effects that introduce perturbations of the basic system behavior, different from uncoupled systems We show through a simulation of a distributed control of a hose tranportation system, that even a minimal dynamical feature of the hose (elastic forces oppossing stretching) can produce significant behavior perturbations.

A Remote Laboratory for a Basic Course on Control Engineering

The current work presents a remote laboratory for a basic course in control engineering over which several experiments may be performed. The proposed experiments have been carefully selected in order to illustrate the maximum number of concepts learnt in the classroom over a unique plant, a Ball & Hoop system. In this work, Labview has been used to acquire and handle process data whereas OPC technology is used to connect remote servers with web-integrated front-end applications. This choice has been made on the basis that these tools do not require very advanced skills and may be a reasonable approach for a wide range of simple remote laboratories.

Quasi-stationary state transportation of a hose with quadrotors

A hose is a flexible almost unidimensional object. Transportation of a hose by means of a team of collaborating robots poses a new multi-robot control paradigm, because the hose introduces strong non-linear interaction effects in the dynamics of the overall system. In this paper, we consider that a team ( n ? 2 ) of unmanned aerial robots, specifically quadrotors, carry out the hose transportation task. A hose is a Deformable Linear Object (DLO). In this paper, a hose hanging from hovering quadrotors, after reaching a quasi-stationary state is modeled by a catenary curve. We consider the control problem of driving the entire system to a state in which each robot is subjected to the same vertical force (i.e. weight), thus each robot energy consumption will be the same, aiming to prevent that any robot runs out of energy much earlier than the others. This problem can be posed only when we deal with multicatenary systems ( n ? 3 ). We have taken care of defining visually measurable system parameters, allowing visual servoing in real life experimentation. In this paper we present the system model, its dynamic simulation, and the derivation of a control system reaching the desired equiload state. Propose the transportation of hose-like objects by teams of quadrotors.We provide the modeling of the dynamics of the hose by catenary curves.Definition of control to ensure equal energy consumption by all team members.Control is achieved by PID.

STATE-ACTION VALUE FUNCTION MODELED BY ELM IN REINFORCEMENT LEARNING FOR HOSE CONTROL PROBLEMS

This paper addresses the problem of efficiency in reinforcement learning of Single Robot Hose Transport (SRHT) by training an Extreme Learning Machine (ELM) from the state-action value Q-table, obtaining large reduction in data space requirements because the number of ELM parameters is much less than the Q-tables size. Moreover, ELM implements a continuous map which can produce compact representations of the Q-table, and generalizations to increased space resolution and unknown situations. In this paper we evaluate empirically three strategies to formulate ELM learning to provide approximations to the Q-table, namely as classification, multi-variate regression and several independent regression problems.

Empirical study of Q-learning based elemental hose transport control

Non-rigid physical elements attached to robotic systems introduce non-linear dynamics that requires innovative control approaches. This paper describes some of our results applying Q-Learning to learn the control commands to solve a hose transportation problem. The learning process is developed in a simulated environment. Computationally expensive but dynamically accurate Geometrically Exact Dynamic Splines (GEDS) have been used to model the hose to be transported by a single robot, showing the difficulties of controlling flexible elastic passive linking elements.

Particle Swarm Optimization Quadrotor Control for Cooperative Aerial Transportation of Deformable Linear Objects

ABSTRACT We present a cooperative aerial robot system for the transportation of hoses. The hose–robot attachment makes the whole system physically interconnected but not rigid, so that control design becomes a difficult nonlinear optimization problem. The hose in quasistationary state can be modeled by sections of catenary curves. We use proportional integral derivative (PID) controllers for both quadrotor attitude and trajectory control, tuned by particle swarm optimization (PSO). In this work we test PSO minimizing an energy function to achieve the PID controller tuning for horizontal motion of quadrotor teams transporting hoses under different stress conditions.

A Neural Network Approximation of L-MCRS Dynamics for Reinforcement Learning Experiments

The autonomous learning of the control of Linked Multicomponent Robotic Systems (L-MCRS) is an open research issue. We are pursuing the application of Reinforcement Learning algorithms to achieve such control. However, accurate simulations needed for RL trials are time consuming, so that the process of training and validation becomes excesively long. In order to obtain results in affordable time, we perform the approximation of the detailed dynamic model of the L-MCRS by Artificial Neural Networks (ANN).