Romain Hollanders

The complexity of Policy Iteration is exponential for discounted Markov Decision Processes

The question of knowing whether the Policy Iteration algorithm (PI) for solving stationary Markov Decision Processes (MDPs) has exponential or (strongly) polynomial complexity has attracted much attention in the last 25 years. Recently, a family of examples on which PI requires an exponential number of iterations to converge was proposed for the total-reward and the average-reward criteria. On the other hand, it was shown that PI runs in strongly polynomial time on discounted-reward MDPs, yet only when the discount factor is fixed beforehand. In this work, we show that PI needs an exponential number of steps to converge on discounted-reward MDPs with a general discount factor.

Improved bound on the worst case complexity of Policy Iteration

Solving Markov Decision Processes is a recurrent task in engineering which can be performed efficiently in practice using the Policy Iteration algorithm. Regarding its complexity, both lower and upper bounds are known to be exponential (but far apart) in the size of the problem. In this work, we provide the first improvement over the now standard upper bound from Mansour and Singh (1999). We also show that this bound is tight for a natural relaxation of the problem.

Special issue in the honor of the 14th “Journées Montoises d’Informatique Théorique”: Foreword

It is our pleasure to release this special issue in the honor of the “fourteenth Journees Montoises d’Informatique Theorique” which were held at the Universite catholique de Louvain, Belgium, in September 2012. The conference welcomed more than 80 participants, and about forty speakers from several countries such as Belgium, Canada, Czech Republic, Finland, France, Germany, Italy, Marocco, Russia, UK, US, . . . In particular, we were honored to hear plenary talks by Profs. Marie-Pierre Beal, Jean Cardinal, Maxime Crochemore, Fabien Durand, Julien Hendrickx, Juhani Karhumaki, and Mikhail Volkov. We have selected twelve papers for this special issue. Each paper has been refereed by at least two independent reviewers. The papers reflect well the rich and various topics of the conference which explore the main challenges in formal language and automata theory and its connections with discrete mathematics and theoretical computer science among which numeration systems and Rauzy fractals, combinatorics on words, repetitions in words and pattern avoidance, morphic words, symbolic dynamics, tree languages, enumeration and asymptotic growth of languages, . . . We wish to heartily thank Isabelle Hisette, Prof. Ph. Lefevre, the F.R.S.-FNRS, and the University for their help and support. Also, this Issue would not have been possible without the help of the many referees. Let them all be thanked for their precious and very serious work. We finally thank Prof. Ch. Choffrut and the editorial staff of RAIRO-TIA for allowing us to publish this special issue.

Data-driven traffic and diffusion modeling in peer-to-peer networks: A real case study

Peer-to-peer (p2p) systems have driven a lot of attention in the past decade as they have become a major source of internet traffic. The amount of data flowing through the p2p network is huge and hence difficult both to comprehend and to control. In this work, we take advantage of a new and rich dataset recording p2p activity at a remarkable scale to give some answers to these difficult problems. After extracting the relevant and measurable properties of the network from the data, we develop two models that aim to make the link between the low-level properties of the network, such as the proportion of free-riders or the distribution of the files among the peers, and its highlevel properties, such as the Quality of Service or the diffusion characteristics, which are the interesting ones. We observe a nice agreement between the high-level properties measured on the real data and on the data simulated by our models, which is encouraging for our models to be used in practice as large-scale prediction tools. Using our models, we make a first prediction and show that it is worth spending efforts to reduce the amount of free-riders to improve the availability of files on the network, but only down to about 65% of free-riders.