Pieter Jan't Hoen

An Evolutionary Dynamical Analysis of Multi-Agent Learning in Iterated Games

In this paper, we investigate Reinforcement learning (RL) in multi-agent systems (MAS) from an evolutionary dynamical perspective. Typical for a MAS is that the environment is not stationary and the Markov property is not valid. This requires agents to be adaptive. RL is a natural approach to model the learning of individual agents. These Learning algorithms are however known to be sensitive to the correct choice of parameter settings for single agent systems. This issue is more prevalent in the MAS case due to the changing interactions amongst the agents. It is largely an open question for a developer of MAS of how to design the individual agents such that, through learning, the agents as a collective arrive at good solutions. We will show that modeling RL in MAS, by taking an evolutionary game theoretic point of view, is a new and potentially successful way to guide learning agents to the most suitable solution for their task at hand. We show how evolutionary dynamics (ED) from Evolutionary Game Theory can help the developer of a MAS in good choices of parameter settings of the used RL algorithms. The ED essentially predict the equilibriums outcomes of the MAS where the agents use individual RL algorithms. More specifically, we show how the ED predict the learning trajectories of Q-Learners for iterated games. Moreover, we apply our results to (an extension of) the COllective INtelligence framework (COIN). COIN is a proved engineering approach for learning of cooperative tasks in MASs. The utilities of the agents are re-engineered to contribute to the global utility. We show how the improved results for MAS RL in COIN, and a developed extension, are predicted by the ED.

A Decommitment Strategy in a Competitive Multi-agent Transportation Setting

Decommitment is the action of foregoing of a contract for another (superior) offer. It has been shown that, using decommitment, agents can reach higher utility levels in case of negotiations with uncertainty about future prospects. In this paper, we study the decommitment concept for the novel setting of a large-scale logistics setting with multiple, competing companies. Orders for transportation of loads are acquired by agents of the (competing) companies by bidding in online auctions. We find significant increases in profit when the agents can decommit and postpone the transportation of a load to a more suitable time. Furthermore, we analyze the circumstances for which decommitment has a positive impact if agents are capable of handling multiple contracts simultaneously.

Learning and Adaption in Multi-Agent Systems

The goal of a self-interested agent within a multiagent system is to maximize its utility over time. In a situation of strategic interdependence, where the actions of one agent may affect the utilities of other agents, the optimal behavior of an agent must be conditioned on the expected behaviors of the other agents in the system. Standard game theory assumes that the rationality and preferences of all the agents is common knowledge: each agent is then able to compute the set of possible equilibria, and if there is a unique equilibrium, choose a best-response to the actions that the other agents will all play. Real agents acting within a multiagent system face multiple problems: the agents may have incomplete information about the preferences and rationality of the other agents in the game, computing the equilibria can be computationally complex, and there might be many equilibria from which to choose. An alternative explanation of the emergence of a stable equilibrium is that it arises as the long-run outcome of a repeated game, in which bounded-rational agents adapt their strategies as they learn about the other agents in the system. We review some possible models of learning for games, and then show the pros and cons of using learning in a particular game, the Compensation Mechanism, a mechanism for the efficient coordination of actions within a multiagent system.

Evolutionary Multi-agent Systems

In Multi-Agent learning, agents must learn to select actions that maximize their utility given the action choices of the other agents. Cooperative Coevolution offers a way to evolve multiple elements that together form a whole, by using a separate population for each element. We apply this setup to the problem of multi-agent learning, arriving at an evolutionary multi-agent system (EA-MAS). We study a problem that requires agents to select their actions in parallel, and investigate the problem solving capacity of the EA-MAS for a wide range of settings.

COllective INtelligence with sequences of actions: coordinating actions in multi-agent systems

The design of a Multi-Agent System (MAS) to perform well on a collective task is non-trivial. Straightforward application of learning in a MAS can lead to sub optimal solutions as agents compete or interfere. The COllective INtelligence (COIN) framework of Wolpert et al. proposes an engineering solution for MASs where agents learn to focus on actions which support a common task. As a case study, we investigate the performance of COIN for representative token retrieval problems found to be difficult for agents using classic Reinforcement Learning (RL). We further investigate several techniques from RL (model-based learning, Q(λ)) to scale application of the COIN framework. Lastly, the COIN framework is extended to improve performance for sequences of actions.

A decommitment strategy in a competitive multi-agent transportation setting

Decommitment is the action of foregoing a contract for another (superior) offer. It has been shown that, using decommitment, agents can reach higher utility levels in case of negotiations with uncertainty about future prospects. In this paper, we study the decommitment concept for a novel large-scale setting in logistics where companies compete for cargo in online auctions.We find significant increases in profit when agents can decommit. Furthermore, we analyze the circumstances for which decommitment has a positive impact if agents are capable of handling multiple contracts simultaneously.

Decommitment in a competitive multi-agent transportation setting

Decommitment is the action of foregoing of a contract for another (superior) offer. It has been analytically shown that, using decommitment, agents can reach higher utility levels in case of negotiations with uncertainty about future opportunities. We study the decommitment concept for the novel setting of a large-scale logistics setting with multiple, competing companies. Orders for transportation of loads are acquired by agents of the (competing) companies by bidding in online auctions. We find significant increases in profit when the agents can decommit and postpone the transportation of a load to a more suitable time. Furthermore, we analyze the circumstances for which decommitment has a positive impact if agents are capable of handling multiple contracts simultaneously. Lastly, we present a demonstrator of the developed model in the form of a Java Applet.

Analyzing multi-agent reinforcement learning using evolutionary dynamics

In this paper, we show how the dynamics of Q-learning can be visualized and analyzed from a perspective of Evolutionary Dynamics (ED). More specifically, we show how ED can be used as a model for Q-learning in stochastic games. Analysis of the evolutionary stable strategies and attractors of the derived ED from the Reinforcement Learning (RL) application then predict the desired parameters for RL in Multi-Agent Systems (MASs) to achieve Nash equilibriums with high utility. Secondly, we show how the derived fine tuning of parameter settings from the ED can support application of the COllective INtelligence (COIN) framework. COIN is a proved engineering approach for learning of cooperative tasks in MASs. We show that the derived link between ED and RL predicts performance of the COIN framework and visualizes the incentives provided in COIN toward cooperative behavior.

Repeated auctions with complementarities

There is an extensive body of literature concerning optimal bidding strategies for agents participating in single shot auctions for single, individually valued goods. However, it remains a largely open question how a bidder should formulate his bidding strategy when there is a sequence of auctions and, furthermore, there are complementarities in the valuation for the bundle of items acquired in the separate auctions. We investigate conditions for which adjusting the bidding horizon beyond the immediate auction is profitable for a bidder. We show how such a strategy, in the limit, reduces agents to zero marginal profits as predicted by the Bertrand economic theory. We support our experimental results by drawing a parallel to the nIPD.

Simulation and Visualization of a Market-Based Model for Logistics Management in Transportation

Distributed logistics and transportation is an important and emerging area of application for multi-agent systems, which has recently attracted a lot of research interest. In previous research ([1], [2]) we have proposed and developed novel techniques to deal with some of the challenges and problems in this application domain. In this paper we describe the software system which was built to visualize and demonstrate our multi-agent model.