Jackson P. Matsuura

Using cases as heuristics in reinforcement learning: a transfer learning application

In this paper we propose to combine three AI techniques to speed up a Reinforcement Learning algorithm in a Transfer Learning problem: Case-based Reasoning, Heuristically Accelerated Reinforcement Learning and Neural Networks. To do so, we propose a new algorithm, called L3, which works in 3 stages: in the first stage, it uses Reinforcement Learning to learn how to perform one task, and stores the optimal policy for this problem as a case-base; in the second stage, it uses a Neural Network to map actions from one domain to actions in the other domain and; in the third stage, it uses the case-base learned in the first stage as heuristics to speed up the learning performance in a related, but different, task. The RL algorithm used in the first phase is the Q-learning and in the third phase is the recently proposed Case-based Heuristically Accelerated Q-learning. A set of empirical evaluations were conducted in transferring the learning between two domains, the Acrobot and the Robocup 3D: the policy learned during the solution of the Acrobot Problem is transferred and used to speed up the learning of stability policies for a humanoid robot in the Robocup 3D simulator. The results show that the use of this algorithm can lead to a significant improvement in the performance of the agent.

Using Transfer Learning to Speed-Up Reinforcement Learning: A Cased-Based Approach

Reinforcement Learning (RL) is a well-known technique for the solution of problems where agents need to act with success in an unknown environment, learning through trial and error. However, this technique is not efficient enough to be used in applications with real world demands due to the time that the agent needs to learn. This paper investigates the use of Transfer Learning (TL) between agents to speed up the well-known Q-learning Reinforcement Learning algorithm. The new approach presented here allows the use of cases in a case base as heuristics to speed up the Q-learning algorithm, combining Case-Based Reasoning (CBR) and Heuristically Accelerated Reinforcement Learning (HARL) techniques. A set of empirical evaluations were conducted in the Mountain Car Problem Domain, where the actions learned during the solution of the 2D version of the problem can be used to speed up the learning of the policies for its 3D version. The experiments were made comparing the Q-learning Reinforcement Learning algorithm, the HAQL Heuristic Accelerated Reinforcement Learning (HARL) algorithm and the TL-HAQL algorithm, proposed here. The results show that the use of a case-base for transfer learning can lead to a significant improvement in the performance of the agent, making it learn faster than using either RL or HARL methods alone.

Transferring knowledge as heuristics in reinforcement learning

The goal of this paper is to propose and analyse a transfer learning meta-algorithm that allows the implementation of distinct methods using heuristics to accelerate a Reinforcement Learning procedure in one domain (the target) that are obtained from another (simpler) domain (the source domain). This meta-algorithm works in three stages: first, it uses a Reinforcement Learning step to learn a task on the source domain, storing the knowledge thus obtained in a case base; second, it does an unsupervised mapping of the source-domain actions to the target-domain actions; and, third, the case base obtained in the first stage is used as heuristics to speed up the learning process in the target domain.A set of empirical evaluations were conducted in two target domains: the 3D mountain car (using a learned case base from a 2D simulation) and stability learning for a humanoid robot in the Robocup 3D Soccer Simulator (that uses knowledge learned from the Acrobot domain). The results attest that our transfer learning algorithm outperforms recent heuristically-accelerated reinforcement learning and transfer learning algorithms.

Reinforcement Learning with Case-Based Heuristics for RoboCup Soccer Keepaway

In this paper we propose to combine Case-based Reasoning and Heuristically Accelerated Reinforcement Learning to speed up a Reinforcement Learning algorithm in a Transfer Learning problem. To do so, we propose a new algorithm called SARSA Accelerated by Transfer Learning - SATL, which uses Reinforcement Learning to learn how to perform one task, stores the policy for this problem as a case-base and then uses the learned case-base as heuristics to speed up the learning performance in a related, but different, task. A set of empirical evaluations were conducted in transferring the learning between two domains with multiple agents: an expanded version of Littmans simulated robot soccer and the RoboCup Soccer Keep away. A policy learned by one agent in the Littmans soccer is used to speed up the agent learning in the Keep away soccer. The results show that the use of this new algorithm can lead to a significant improvement in the performance of the learning agents.

TORP: The Open Robot Project

The development of robots has shown itself as a very complex interdisciplinary research field. The predominant procedure for these developments in the last decades is based on the assumption that each robot is a fully personalized project, with the direct embedding of hardware and software technologies in robot parts with no level of abstraction. Although this methodology has brought countless benefits to the robotics research, on the other hand, it has imposed major drawbacks: (i) the difficulty to reuse hardware and software parts in new robots or new versions; (ii) the difficulty to compare performance of different robots parts; and (iii) the difficulty to adapt development needs—in hardware and software levels—to local groups expertise. Large advances might be reached, for example, if physical parts of a robot could be reused in a different robot constructed with other technologies by other researcher or group. This paper proposes a framework for robots, TORP (The Open Robot Project), that aims to put forward a standardization in all dimensions (electrical, mechanical and computational) of a robot shared development model. This architecture is based on the dissociation between the robot and its parts, and between the robot parts and their technologies. In this paper, the first specification for a TORP family and the first humanoid robot constructed following the TORP specification set are presented, as well as the advances proposed for their improvement.

Heuristic Q-learning soccer players: a new reinforcement learning approach to RoboCup simulation

This paper describes the design and implementation of a 4 player RoboCup Simulation 2D team, which was build by adding Heuristic Accelerated Reinforcement Learning capabilities to basic players of the well-known UvA Trilearn team. The implemented agents learn by using a recently proposed Heuristic Reinforcement Learning algorithm, the Heuristically Accelerated Q-Learning (HAQL), which allows the use of heuristics to speed up the well-known Reinforcement Learning algorithm Q-Learning. A set of empirical evaluations was conducted in the RoboCup 2D Simulator, and experimental results obtained while playing with other teams shows that the approach adopted here is very promising.

Detecting compositional changes in dielectric materials simulated by three-dimensional RC network models

This work discusses the detection of small compositional changes in materials that have microstructures containing conducting and dielectric phases, which can be described by networks of resistive (R) and capacitive (C) components in a three-dimensional lattice. For this purpose, a principal component analysis (PCA) method is employed to discriminate normal samples from samples with altered composition on the basis of statistics extracted from the waveform of the network response to a given excitation. This approach obviates the requirement for multivariate regression and simplifies experimental workload for model-building, since only data from normal samples are required in the development of the PCA model. Waveform variability of the excitation source is also accounted for through the use of a nominal model derived using subspace identification. This enables standardization and software based metrology transfer across different labs. The effect of network size on the capability of detecting minute compositional changes was assessed. For networks of 520 components, it was possible to identify changes in the fraction of capacitors down to 10−2 at ±2σ confidence levels.

Evaluation of an ICP Based Algorithm for Simultaneous Localization and Mapping Using a 3D Simulated P3DX Robot

Autonomous mobile robots can be applied to perform activities that should not, or cannot, be performed by humans due to inhospitable conditions or high level of danger. An autonomous mobile robot must be able to navigate safely in unfamiliar environments by reconstructing information from its sensors so as to plan and execute routes. Simultaneous Localization And Mapping, SLAM, technique allows the gradual creation of a map using data obtained from sensors while estimating the robot localization, and the Iterative Closest Point, ICP, algorithm is one of the approaches adopted for SLAM. This work proposes and evaluates an ICP-based algorithm for simultaneous localization and mapping of a robot. The algorithm was implemented in a simulated environment using Microsoft Robotics Developer Studio, MRDS. Experimental results show that, in the evaluated trajectory, the method presented in this work has a better performance than the one obtained by the original ICP algorithm.

Investigation in transfer learning: better way to apply transfer learning between agents

This paper propose to investigate a better way to apply Transfer Learning (TL) between agents to speed up the Q-learning Reinforcement Learning algorithm and combines Case-Based Reasoning (CBR) and Heuristically Accelerated Reinforcement Learning (HARL) techniques. The experiments were made comparing differents approaches of Transfer Learning were actions learned in the acrobot problem can be used to speed up the learning of the policies of stability for Robocup 3D. The results confirm that the same Transfer Learning information can show differents results, depending how is applied.