Bruno Castro da Silva

Dealing with non-stationary environments using context detection

In this paper we introduce RL-CD, a method for solving reinforcement learning problems in non-stationary environments. The method is based on a mechanism for creating, updating and selecting one among several partial models of the environment. The partial models are incrementally built according to the systems capability of making predictions regarding a given sequence of observations. We propose, formalize and show the efficiency of this method both in a simple non-stationary environment and in a noisy scenario. We show that RL-CD performs better than two standard reinforcement learning algorithms and that it has advantages over methods specifically designed to cope with non-stationarity. Finally, we present known limitations of the method and future works.

ITSUMO: an Intelligent Transportation System for Urban Mobility

This paper presents an overview of ITSUMO, a microscopic traffic simulator based on cellular automata. The implementation uses agent technologies with a bottom-up philosophy in mind. We give an overview of the system and some details of its modules (data, simulation, driver and information/visualization).

Learning parameterized motor skills on a humanoid robot

We demonstrate a sample-efficient method for constructing reusable parameterized skills that can solve families of related motor tasks. Our method uses learned policies to analyze the policy space topology and learn a set of regression models which, given a novel task, appropriately parameterizes an underlying low-level controller. By identifying the disjoint charts that compose the policy manifold, the method can separately model the qualitatively different sub-skills required for solving distinct classes of tasks. Such sub-skills are useful because they can be treated as new discrete, specialized actions by higher-level planning processes. We also propose a method for reusing seemingly unsuccessful policies as additional, valid training samples for synthesizing the skill, thus accelerating learning. We evaluate our method on a humanoid iCub robot tasked with learning to accurately throw plastic balls at parameterized target locations.

ITSUMO: an intelligent transportation system for urban mobility

It is well-known that big cities suffer from traffic congestion and all consequences that come with it. This is an especial problem in cities in developing countries where the public transportation system is not reliable and where the fleet of vehicles tend to be old thus increasing air pollution. There is no turnkey solution for this problem, but several improvements have been suggested in the field of urban and traffic management, provided an information system is built which can provide information to both the traffic experts and the user of the system. Such an information system has to incorporate features of an ITS and an ATIS. An underline assumption is that there is a simulation model to provide certain kinds of information in forecast. This paper discusses the model and implementation of such an information system which is based on a microscopic model of simulation and on cellular automata and is implemented using agent technologies and with a bottom-up philosophy in mind. We give here an overview of the project, the details of the modules (data, simulation, driver and information/visualization), as well as discuss an application of the simulation tool.

Improving reinforcement learning with context detection

In this paper we propose a method for solving reinforcement learning problems in non-stationary environments. The basic idea is to create and simultaneously update multiple partial models of the environment dynamics. The learning mechanism is based on the detection of context changes, that is, on the detection of significant changes in the dynamics of the environment. Based on this motivation, we propose, formalize and show the efficiency of a method for detecting the current context and the associated model of prediction, as well as a method for updating each of the incrementally built models.

Distributed constraint propagation for diagnosis of faults in physical processes

Most of the current research on distributed diagnosis in and for multiagent systems focuses on diagnosis of coordination failures. Proposed approaches for this problem are not efficient for diagnosing failures in physical devices. This paper proposes algorithms for distributed troubleshooting of physical devices and processes. The consequences of using distributed representation of the knowledge, ATMS, and distributed reasoning are discussed and algorithms are proposed to deal with the occurrence of conflicts and the computation of the set of candidate diagnosis.

A Task-and-Technique Centered Survey on Visual Analytics for Deep Learning Model Engineering

Abstract Although deep neural networks have achieved state-of-the-art performance in several artificial intelligence applications in the past decade, they are still hard to understand. In particular, the features learned by deep networks when determining whether a given input belongs to a specific class are only implicitly described concerning a considerable number of internal model parameters. This makes it harder to construct interpretable hypotheses of what the network is learning and how it is learning—both of which are essential when designing and improving a deep model to tackle a particular learning task. This challenge can be addressed by the use of visualization tools that allow machine learning experts to explore which components of a network are learning useful features for a pattern recognition task, and also to identify characteristics of the network that can be changed to improve its performance. We present a review of modern approaches aiming to use visual analytics and information visualization techniques to understand, interpret, and fine-tune deep learning models. For this, we propose a taxonomy of such approaches based on whether they provide tools for visualizing a network’s architecture, to facilitate the interpretation and analysis of the training process, or to allow for feature understanding. Next, we detail how these approaches tackle the tasks above for three common deep architectures: deep feedforward networks, convolutional neural networks, and recurrent neural networks. Additionally, we discuss the challenges faced by each network architecture and outline promising topics for future research in visualization techniques for deep learning models.