Anestis Fachantidis

Reinforcement learning agents providing advice in complex video games

This article introduces a teacher–student framework for reinforcement learning, synthesising and extending material that appeared in conference proceedings [Torrey, L., & Taylor, M. E. (2013)]. Teaching on a budget: Agents advising agents in reinforcement learning. {Proceedings of the international conference on autonomous agents and multiagent systems}] and in a non-archival workshop paper [Carboni, N., &Taylor, M. E. (2013, May)]. Preliminary results for 1 vs. 1 tactics in StarCraft. {Proceedings of the adaptive and learning agents workshop (at AAMAS-13)}]. In this framework, a teacher agent instructs a student agent by suggesting actions the student should take as it learns. However, the teacher may only give such advice a limited number of times. We present several novel algorithms that teachers can use to budget their advice effectively, and we evaluate them in two complex video games: StarCraft and Pac-Man. Our results show that the same amount of advice, given at different moments, can have different effects on student learning, and that teachers can significantly affect student learning even when students use different learning methods and state representations.

Transferring task models in Reinforcement Learning agents

The main objective of transfer learning is to reuse knowledge acquired in a previous learned task, in order to enhance the learning procedure in a new and more complex task. Transfer learning comprises a suitable solution for speeding up the learning procedure in Reinforcement Learning tasks. This work proposes a novel method for transferring models to Reinforcement Learning agents. The models of the transition and reward functions of a source task, will be transferred to a relevant but different target task. The learning algorithm of the target tasks agent takes a hybrid approach, implementing both model-free and model-based learning, in order to fully exploit the presence of a source task model. Moreover, a novel method is proposed for transferring models of potential-based reward shaping functions. The empirical evaluation, of the proposed approaches, demonstrated significant results and performance improvements in the 3D Mountain Car and Server Job Scheduling tasks, by successfully using the models generated from their corresponding source tasks.

Transfer learning via multiple inter-task mappings

In this paper we investigate using multiple mappings for transfer learning in reinforcement learning tasks. We propose two different transfer learning algorithms that are able to manipulate multiple inter-task mappings for both model-learning and model-free reinforcement learning algorithms. Both algorithms incorporate mechanisms to select the appropriate mappings, helping to avoid the phenomenon of negative transfer. The proposed algorithms are evaluated in the Mountain Car and Keepaway domains. Experimental results show that the use of multiple inter-task mappings can significantly boost the performance of transfer learning methodologies, relative to using a single mapping or learning without transfer.

Transfer learning with probabilistic mapping selection

When transferring knowledge between reinforcement learning agents with different state representations or actions, past knowledge must be efficiently mapped to novel tasks so that it aids learning. The majority of the existing approaches use pre-defined mappings provided by a domain expert. To overcome this limitation and enable autonomous transfer learning, this paper introduces a method for weighting and using multiple inter-task mappings based on a probabilistic framework. Experimental results show that the use of multiple inter-task mappings, accompanied with a probabilistic selection mechanism, can significantly boost the performance of transfer learning relative to 1) learning without transfer and 2) using a single hand-picked mapping. We especially introduce novel tasks for transfer learning in a realistic simulation of the iCub robot, demonstrating the ability of the method to select mappings in complex tasks where human intuition could not be applied to select them. The results verified the efficacy of the proposed approach in a real world and complex environment.

Model-based reinforcement learning for humanoids: A study on forming rewards with the iCub platform

Technological advancements in robotics and cognitive science are contributing to the development of the field of cognitive robotics. Modern robotic platforms are able to exhibit the ability to learn and reason about complex tasks and to follow behavioural goals in complex environments. Nevertheless, many challenges still exist. One of these great challenges is to equip these robots with cognitive systems that allow them to deal with less constrained situations, beyond constrained scenarios as in industrial robotics. In this work we explore the application of the Reinforcement Learning (RL) paradigm to study the autonomous development of robot controllers without a priori supervised learning. Such a model-based RL architecture is discussed for the cognitive implications of applying RL in humanoid robots. To this end we show a developmental framework for RL in robotics and its implementation and testing for the iCub robotic platform in two novel experimental scenarios. In particular we focus on iCub simulation experiments with comparisons between internal perception-based reward signals and external ones, in order to compare learning performance of the robot guided by its own perception of actions outcomes with the one when the robot has its actions externally evaluated.

Learning to Teach Reinforcement Learning Agents

In this article, we study the transfer learning model of action advice under a budget. We focus on reinforcement learning teachers providing action advice to heterogeneous students playing the game of Pac-Man under a limited advice budget. First, we examine several critical factors affecting advice quality in this setting, such as the average performance of the teacher, its variance and the importance of reward discounting in advising. The experiments show that the best performers are not always the best teachers and reveal the non-trivial importance of the coefficient of variation (CV) as a statistic for choosing policies that generate advice. The CV statistic relates variance to the corresponding mean. Second, the article studies policy learning for distributing advice under a budget. Whereas most methods in the relevant literature rely on heuristics for advice distribution, we formulate the problem as a learning one and propose a novel reinforcement learning algorithm capable of learning when to advise or not. The proposed algorithm is able to advise even when it does not have knowledge of the student’s intended action and needs significantly less training time compared to previous learning approaches. Finally, in this article, we argue that learning to advise under a budget is an instance of a more generic learning problem: Constrained Exploitation Reinforcement Learning.

Transferring Models in Hybrid Reinforcement Learning Agents

The main objective of transfer learning is to reuse knowledge acquired in a previous learned task, in order to enhance the learning procedure in a new and more complex task. Transfer learning comprises a suitable solution for speeding up the learning procedure in Reinforcement Learning tasks. In this work, we propose a novel method for transferring models to a hybrid reinforcement learning agent. The models of the transition and reward functions of a source task, will be transferred to a relevant but different target task. The learning algorithm of the target task’s agent takes a hybrid approach, implementing both model-free and model-based learning, in order to fully exploit the presence of a model. The empirical evaluation, of the proposed approach, demonstrated significant results and performance improvements in the 3D Mountain Car task, by successfully using the models generated from the standard 2D Mountain Car.

Multi-label Modality Classification for Figures in Biomedical Literature

The figures found in biomedical literature are a vital part of biomedical research, education and clinical decision. The multitude of their modalities and the lack of corresponding meta-data, constitute search and information retrieval a difficult task. We present multi-label modality classification approaches for biomedical figures. In particular, we investigate using both simple and compound figures for training a multi-label model to be used for annotating either all figures, or only those predicted as compound by an initial compound figure detection model. Using data from the medical task of ImageCLEF 2016, we train our approaches with visual features and compare them with the standard approach involving compound figure separation into sub-figures. Furthermore, we present a web application for medical figure retrieval, which is based on one of our classification approaches and allows users to search for figures of PubMed Central.

Segmento: An R-based Visualization-rich System for Customer Segmentation and Targeting

Customer segmentation is one of the most efficient and promising tools in a marketers toolbox. In this paper, we introduce Segmento, an R-based customer segmentation system that uses clustering techniques to discover customer segments and offers tools to design and evaluate marketing campaigns. We present the features and the functionality of the system, as well as some of its unique, state-of-the-art visualizations.

A prediction model of passenger demand using AVL and APC data from a bus fleet

In this paper we present the passenger demand prediction model of BusGrid. BusGrid is a novel information system for the improvement of productivity and customer service in public transport bus services. BusGrid receives and processes real time data from the automated vehicle location (AVL) and the automated passenger counting (APC) sensors installed on a bus fleet and assists their operator on the improvement of bus schedules and the design of new bus routes and stops based on the expected demand. For the prediction of passenger demand in any bus stop, the raw sensor data were pre-processed and several different feature sets were extracted and tested as predictors of passenger demand. The pre-processed data were used for the supervised learning of a regression model that predicts people demand for any given bus stop and route. Experimental results show that the proposed approach achieved significant improvements over the baseline approaches. Knowledge representation, through the proposed feature set, played a key role on the ability of the prediction model to generalize well beyond its training set, to new bus stops and routes.