André Pinz Borges

Interaction Models for Multiagent Reinforcement Learning

This article proposes and compares different interaction models for reinforcement learning based on multi-agent system. The cooperation during the learning process is crucial to guarantee the convergence to a good policy. The exchange of rewards among the agents during the interaction is a complex task and if it is inadequate it may cause delays in learning or generate unexpected transitions, making the cooperation inefficient and con-verging to a non-satisfactory policy. In order to allow the interactive discovery of high quality policies we have developed several cooperation models based on the ex-change of action policies between the agents. Experimental results have shown that the proposed cooperation models are able to speed up the convergence of the agents while achieving optimal action policies even in high-dimensional environments (e.g. traffic), outperforming the standard Q-learning algorithm.

A learning agent to help drive vehicles

This paper presents the development of an intelligent agent used to assist vehicle drivers. The agent has a set of resources to generate its action policy: road and vehicle features and a knowledge base containing conduct rules. The perception of the agent is ensured by a set of sensors, which provide the agent with data such as speed, position and conditions of the brakes. The main agent behaviour is to carry out action plans involving: increase, maintain or reduce speed. The main effort of this research was the induction of conduct rules from data of previous trips. These rules form a classifier used for the selection of actions forming the conduction plan. Results observed with the experiments have showed that the proposed classifier increases the efficiency throughout the conduction of vehicles.

Lessons learned from a simulated environment for trains conduction

This paper consolidates and discuss the results of a software agent development, named SDriver, which is able to drive an intercity freight train in a secure, economic and fast way. The SDriver executes a small set of instructions, named: reducing, increasing or maintaining the acceleration point, and start breaking. Three approaches have been studied to implement the core of SDriver: (i) machine learning (classification methods), (ii) distributed constraint optimization, and (iii) specialized rules (if-then). The SDriver performance was evaluated comparing fuel consumption and actions similarity with a real conduction, using a simulated environment. The validation of the knowledge discovered from the machine learning approach was done quantitatively, calculating a degree of similarity between the simulation and the history of travel. The main results are expressed by their mean values: 32% of fuel consumption reduction and 85% action similarity between the SDriver and the real conductor.

An architecture of BDI agent for autonomous locomotives controller

In this paper we propose an architecture of intelligent agent for automatic locomotives operating. The system agent generates its action policy using a set of resources, such as type of railway, composition, belief perception and reasoning about the actions. The focus of the operator agent is directed to the choice of acceleration points (gear) and preparation of travel plans in a journey guided by goals and objectives. The system is equipped with a module capable to plan the actions to move the vehicle from an initial point P to an end point Q and an executor module that implements the generated plan and modifies the state of the environment. For this purpose, we use the mental model that is based on the triple Belief, Desire and Intention (BDI) to which the perception of the agent is guaranteed by a set of sensors that provide speed information, position and breaks condition. The main focus on this research is the usage of mental model BDI for the resolution of a problem that combines travel naturally conflicting factors, such as safety, time and fuel consumption. Experimental results show that the developed architecture using the mental model BDI increases the efficiency of autonomous vehicles operating.

Improving the distributed constraint optimization using social network analysis

Distributed Constraint Optimization Problem (DCOP) has emerged as one of most important formalisms for distributed reasoning in multiagent systems. Nevertheless, there are few real world applications based on methods for solving DCOP, due to their inefficiency in some scenarios. This paper introduces the use of Social Network Analysis (SNA) techniques to improve the performance in pseudo-tree-based DCOP algorithms. We investigate when the SNA is useful and which techniques can be applied in some DCOP instances. To evaluate our proposal, we use the two most popular complete and optimal DCOP algorithms, named ADOPT and DPOP, and compare the obtained results with others well-known pre-processing techniques. The experimental results show that SNA techniques can speed up ADOPT and DPOP algorithms.

An intelligent system for driving trains using Case-Based Reasoning

This paper presents a planning approach using Case-Based Reasoning (CBR) to generate plans for driving trains. The main idea of a planning strategy is to generate a sequence of actions for an agent, which can use these actions to change its environment. CBR allows using prior experiences in the situation assessment task. In the proposed approach, each previous experience (if not applicable) is adjusted resulting in cases specializations. Our interest is reducing the number of corrections triggered when a case retrieved is not applicable, based on these specializations. Experiments showed that the plans generated using this proposed method had a significant increase in the number of cases recovered satisfactorily, also reducing the need of adaptations for the cases recovered.

A strategy for converging dynamic action policies

In this paper we propose a novel strategy for converging dynamic policies generated by adaptive agents, which receive and accumulate rewards for their actions. The goal of the proposed strategy is to speed up the convergence of such agents to a good policy in dynamic environments. Since it is difficult to have the good value for a state due to the continuous changing in the environment, previous policies are kept in memory for reuse in future policies, avoiding delays or unexpected speedups in the agents learning. Experimental results on dynamic environments with different policies have shown that the proposed strategy is able to speed up the convergence of the agent while achieving good action policies.

A multi-layer architecture proposal for conducting trains employing CBR

This paper presents a planning approach using Case-Based Reasoning (CBR) modeled as a Subsumption Architecture to generate plans for driving trains. The main idea of a planning strategy is to generate a sequence of actions for an agent, which can use these actions to change its environment. CBR allows using prior experiences for new task assignments. In the proposed ap-proach, each previous experience (if not applicable) is adjusted us-ing one or more adaptation methods like substitutive and genetic algorithm. Our interest is to create a flexible architecture for an agent and apply it to simulate train conductions. We expect that the plans generated by this approach generate better results com-pared to another studies already developed for the area mainly considering fuel consumption and travel time.

Towards an optimal driving trains in single line using crossing loops

This paper describes an intelligent approach based on agents that are able to drive and coordinate trains on stretches of railway line containing a crossing loop. Halts close to or even in crossing loops lead to increased consumption of fossil fuels, longer journey times and exhaustion of track capacity. In this paper the agents make use of a set of resources — railway line characteristics, train characteristics, driving rules and information about other trains — to generate their action policy. The agents perception is guaranteed by a set of sensors that provide data such as speed, position and information about the line. The tasks that the agent performs include carrying out actions such as increasing or reducing the speed of the train. The main objective of this study was to avoid unnecessary halts, which are the main cause of increased fuel consumption and journey time. Our results show that strong reductions can be made in terms of fuel consumption (average reduction of 25.5%), journey time (average reduction of 22.5%) and exhaustion of track capacity. Simulations were performed in which traditional driving techniques, with halts at several points along the stretch of track, were compared with driving performed by the multi-agent system, without any halts.

Reinforcement Learning with Multiple Shared Rewards

A major concern in multi-agent coordination is how to select algorithms that can lead agents to learn together to achieve certain goals. Much of the research on multi-agent learning relates to reinforcement learning (RL) techniques. One element of RL is the interaction model, which describes how agents should interact with each other and with the environment. Discrete, continuous and objective-oriented interaction models can improve convergence among agents. This paper proposes an approach based on the integration of multi-agent coordination models designed for reward-sharing policies. By taking the best features from each model, better agent coordination is achieved. Our experimental results show that this approach improves convergence among agents even in large state-spaces and yields better results than classical RL approaches.