Sam Maes

Identifiability of causal effects in a multi-agent causal model

This paper is a first step to extending Judea Pearls work on identification of causal effects to a multi-agent context. We introduce multi-agent causal models consisting of a collection of agents each having access to a non-disjoint subset of the variables constituting the domain. Every agent has a causal model, determined by nonexperimental data and an acyclic causal diagram over its variables. The algorithm under investigation in this paper, tests whether the assumptions made in a causal model are sufficient to calculate the effect of an intervention (i.e. whether the effect of an intervention is identifiable). It is a distributed algorithm with a minimum amount of inter-agent communication concerning solely shared variables and where the details of each local causal model are kept confidential.

On a dynamical analysis of reinforcement learning in games: emergence of Occam's Razor

Modeling learning agents in the context of Multi-agent Systems requires an adequate understanding of their dynamic behaviour. Usually, these agents are modeled similar to the different players in a standard game theoretical model. Unfortunately traditional Game Theory is static and limited in its usefelness. Evolutionary Game Theory improves on this by providing a dynamics which describes how strategies evolve over time. In this paper, we discuss three learning models whose dynamics are related to the Replicator Dynamics(RD). We show how a classical Reinforcement Learning(RL) technique, i.e. Q-learning relates to the RD. This allows to better understand the learning process and it allows to determine how complex a RL model should be. More precisely, Occams Razor applies in the framework of games, i.e. the simplest model (Cross) suffices for learning equilibria. An experimental verification in all three models is presented.

Inference in multi-agent causal models

In this article, we demonstrate the usefulness of causal Bayesian networks as probabilistic reasoning systems. The biggest advantage of causal Bayesian networks over traditional probabilistic Bayesian networks is that they sometimes allow to perform causal inference, i.e. the calculation of the causal effect of one variable on other variables. We treat a state-of-the-art algorithm for performing causal inference that is based on a new factorization of the joint probability distribution and is a systematic approach for the calculation due to Tian and Pearl. We elaborate on the problems that can arise when working with a centralized approach and discuss how a decentralized cooperative multi-agent approach might overcome some of these problems. The main contribution of this article is the introduction of multi-agent causal models as a way to overcome the problems in a centralized setting. They are an extension of causal Bayesian networks to a distributed setting consisting of a number of agents each having access to an overlapping set of the variables. We extend a state-of-the-art causal inference algorithm for this particular domain. We will show that our approach is as powerful in computing causal effects as the centralized algorithm.

Reinforcement Learning in Large State Spaces

Large state spaces and incomplete information are two problems that stand out in learning in multi-agent systems. In this paper we tackle them both by using a combination of decision trees and Bayesian networks (BNs) to model the environment and the Q-function. Simulated robotic soccer is used as a testbed, since there agents are faced with both large state spaces and incomplete information. The long-term goal of this research is to define generic techniques that allow agents to learn in large-scaled multi-agent systems.