Gavin Taylor

An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning

We show that linear value-function approximation is equivalent to a form of linear model approximation. We then derive a relationship between the model-approximation error and the Bellman error, and show how this relationship can guide feature selection for model improvement and/or value-function improvement. We also show how these results give insight into the behavior of existing feature-selection algorithms.

Kernelized value function approximation for reinforcement learning

A recent surge in research in kernelized approaches to reinforcement learning has sought to bring the benefits of kernelized machine learning techniques to reinforcement learning. Kernelized reinforcement learning techniques are fairly new and different authors have approached the topic with different assumptions and goals. Neither a unifying view nor an understanding of the pros and cons of different approaches has yet emerged. In this paper, we offer a unifying view of the different approaches to kernelized value function approximation for reinforcement learning. We show that, except for different approaches to regularization, Kernelized LSTD (KLSTD) is equivalent to a modelbased approach that uses kernelized regression to find an approximate reward and transition model, and that Gaussian Process Temporal Difference learning (GPTD) returns a mean value function that is equivalent to these other approaches. We also discuss the relationship between our modelbased approach and the earlier Gaussian Processes in Reinforcement Learning (GPRL). Finally, we decompose the Bellman error into the sum of transition error and reward error terms, and demonstrate through experiments that this decomposition can be helpful in choosing regularization parameters.

Towards Modeling the Behavior of Autonomous Systems and Humans for Trusted Operations

Greater unmanned system autonomy will lead to improvements in mission outcomes, survivability and safety. However, an increase in platform autonomy increases system complexity. For example, flexible autonomous platforms deployed in a range of environments place a burden on humans to understand evolving behaviors. More importantly, when problems arise within complex systems, they need to be managed without increasing operator workload. A supervisory control paradigm can reduce workload and allow a single human to manage multiple autonomous platforms. However, this requires consideration of the human as an integrated part of the overall system, not just as a central controller. This paradigm can benefit from novel and intuitive techniques that isolate and predict anomalous situations or state trajectories within complex autonomous systems in terms of mission context to allow efficient management of aberrant behavior. This information will provide the user with improved feedback about system behavior, which will in turn lead to more relevant and effective prescriptions for interaction, particularly during emergency procedures. This, in turn, will enable proper trust calibration. We also argue that by understanding the context of the user’s decisions or system’s actions (seamless integration of the human), the autonomous platform can provide more appropriate information to the user.

Reports of the 2013 AAAI Spring Symposium Series

The Association for the Advancement of Artificial Intelligence was pleased to present the AAAI 2014 Spring Symposium Series, held Monday through Wednesday, March 24–26, 2014. The titles of the eight symposia were Applied Computational Game Theory, Big Data Becomes Personal: Knowledge into Meaning, Formal Verification and Modeling in Human-Machine Systems, Implementing Selves with Safe Motivational Systems and Self-Improvement, The Intersection of Robust Intelligence and Trust in Autonomous Systems, Knowledge Representation and Reasoning in Robotics, Qualitative Representations for Robots, and Social Hacking and Cognitive Security on the Internet and New Media). This report contains summaries of the symposia, written, in most cases, by the cochairs of the symposium.