Segismundo S. Izquierdo

Transient and asymptotic dynamics of reinforcement learning in games

Abstract Reinforcement learners tend to repeat actions that led to satisfactory outcomes in the past, and avoid choices that resulted in unsatisfactory experiences. This behavior is one of the most widespread adaptation mechanisms in nature. In this paper we fully characterize the dynamics of one of the best known stochastic models of reinforcement learning [Bush, R., Mosteller, F., 1955. Stochastic Models of Learning. Wiley & Sons, New York] for 2-player 2-strategy games. We also provide some extensions for more general games and for a wider class of learning algorithms. Specifically, it is shown that the transient dynamics of Bush and Mostellers model can be substantially different from its asymptotic behavior. It is also demonstrated that in general—and in sharp contrast to other reinforcement learning models in the literature—the asymptotic dynamics of Bush and Mostellers model cannot be approximated using the continuous time limit version of its expected motion.

Fuzzy Logic for Social Simulation using NetLogo

Fuzzy Logic is a framework particularly useful to formalise and deal with imprecise concepts and statements expressed in natural language. This paper has three related aims. First, it aims to provide a short introduction to the basics of Fuzzy Logic within the context of social simulation. Secondly, it presents a well-documented NetLogo extension that facilitates the use of Fuzzy Logic within NetLogo. Finally, by providing a concrete example, it shows how researchers can use the Fuzzy Logic extension to build agent-based models in which individual agents hold their own fuzzy concepts and use their own fuzzy rules, which may also change over time. We argue that Fuzzy Logic and the tools provided here can be useful in Social Simulation in different ways. For example, they can assist in the process of analysing the robustness of a certain social theory expressed in natural language to different specifications of the imprecise concepts that the theory may contain (such as e.g. “wealthy†, “poor†or “disadvantaged†). They can also facilitate the exploration of the effect that heterogeneity in concept interpretations may have in a society (i.e. the significance of the fact that different people may have different interpretations of the same concept). Thus, this paper and the tools included in it can make the endeavour of translating social theories into computer programs easier and more rigorous at the same time.

Dynamics of the Bush-Mosteller Learning Algorithm in 2x2 Games

Reinforcement learners interact with their environment and use their experience to choose or avoid certain actions based on the observed consequences. Actions that led to satisfactory outcomes (i.e. outcomes that met or exceeded aspirations) in the past tend to be repeated in the future, whereas choices that led to unsatisfactory experiences are avoided. The empirical study of reinforcement learning dates back to Thorndike’s animal experiments on instrumental learning at the end of the 19th century (Thorndike, 1898). The results of these experiments were formalised in the well known ‘Law of Effect’, which is nowadays one of the most robust properties of learning in the experimental psychology literature: “Of several responses made to the same situation those which are accompanied or closely followed by satisfaction to the animal will, other things being equal, be more firmly connected with the situation, so that, when it recurs, they will be more likely to recur; those which are accompanied or closely followed by discomfort to the animal will, other things being equal, have their connections to the situation weakened, so that, when it recurs, they will be less likely to occur. The greater the satisfaction or discomfort, the greater the strengthening or weakening of the bond.” (Thorndike, 1911, p. 244) Nowadays there is little doubt that reinforcement learning is an important aspect of much learning in most animal species, including many phylogenetically very distant from vertebrates (e.g. earthworms (Maier & Schneirla, 1964) and fruit flies (Wustmann, 1996)). Thus, it is not surprising that reinforcement learning –being one of the most widespread adaptation mechanisms in nature– has attracted the attention of many scientists and engineers for decades. This interest has led to the formulation of various models of reinforcement learning and –when feasible– to the theoretical analysis of their dynamics. In particular, this chapter characterises the dynamics of one of the best known stochastic models of reinforcement learning (Bush & Mosteller, 1955) when applied to decision problems of strategy (i.e. games). The following section is devoted to explaining in detail the context of application of our theoretical analysis, i.e. 2-player 2-strategy games. Section 3 is a brief review of various models of reinforcement learning that have been studied in strategic contexts. Section 4 presents the Bush-Mosteller reinforcement learning algorithm. Section 5 describes two types of critical points that are especially relevant for the dynamics of the process: self-reinforcingequilibria (SREs) and self-correcting-equilibria (SCEs). Sections 6 and 7 detail the relevance

Market Failure Caused by Quality Uncertainty

The classical argument used to explain why markets can fail when there is product quality variability (e.g. the used car market) relies heavily on the presence of asymmetric information —i.e. there must exist some reliable quality indicators that can be observed by sellers, but not by buyers. Using computer simulation, this paper illustrates how such market failures can occur even in the absence of asymmetric information. The mere assumption that buyers estimate the quality of the product they buy using their past experience in previous purchases is enough to observe prices drop, market efficiency losses, and systematic underestimation of actual product quality. This alternative explanation is shown to be valid for a very wide range of learning rules and in various market contexts.

Combining Mathematical and Simulation Approaches to Understand the Dynamics of Computer Models

This chapter shows how computer simulation and mathematical analysis can be used together to understand the dynamics of computer models. For this purpose, we show that it is useful to see the computer model as a particular implementation of a formal model in a certain programming language. This formal model is the abstract entity which is defined by the input–output relation that the computer model executes and can be seen as a function that transforms probability distributions over the set of possible inputs into probability distributions over the set of possible outputs.

Publishing science in the digital age. The case of Neurocirugía

Neurocirugía publishes a printed edition for subscribers, and also an electronic edition which is available online free of charge. The coexistence of these two formats raises some issues regarding their justification and their future evolution, e.g. why does a subscription- based journal offer free online access? Would it be wise to charge for -or somewhat limit- the electronic access to the Journal? How is the Internet changing the benefits to society that the Journal provides? Will the printed and the electronic edition of the Journal continue to coexist? This paper provides some answers and reflections on these questions. Many of our considerations are based on ideas that have been presented and discussed in a series of editorials in Neurocirugía (see Neurocirugía 17 (2), 2006); in this paper we reconsider, complement, and rearrange previous arguments to address the issues mentioned above. Based on an analysis of economic costs and of all the stakeholders involved (authors, readers, the Journal, the Spanish Society of Neurosurgery, and society as a whole), we justify the present coexistence of the two publishing formats, defend free online access, and provide our view on the expected evolution of the Journal. While we focus primarily on Neurocirugía, most of our reflections can be carried over to other scientific journals.

Mixing and Diffusion in a Two-Type Population

The outbreak of epidemics, the rise of religious radicalization or the motivational influence of fellow students in classrooms are some of the issues that can be described as diffusion processes in heterogeneous groups. Understanding the role that interaction patterns between groups (e.g. homophily or segregation) play in the diffusion of certain traits or behaviours is a major challenge for contemporary societies. Here, we study the impact on diffusion processes of mixing (or, alternatively, segregating) two groups that present different sensitivities or propensities to contagion. We find non-monotonic effects of mixing and inefficient segregation levels, i.e. situations where a change in the mixing level can benefit both groups, e.g. where an increase in the mixing level can reduce the expected contagion levels in both groups. These findings can have fundamental consequences for the design of inclusion policies.

Mamdani Fuzzy Systems for Modelling and Simulation: A Critical Assessment

Fuzzy logic presents many potential applications for modelling and simulation. In particular, this paper analyses one of the most popular fuzzy logic techniques: Mamdani systems. Mamdani systems can look particularly appealing because they are designed to incorporate expert knowledge in the form of IF-THEN rules expressed in natural language. While this is an attractive feature for modelling and simulating social and other complex systems, its actual application presents important caveats. This paper studies the potential use of Mamdani systems to explore the logical consequences of a model based on IF-THEN rules via simulation. We show that in the best-case scenario a Mamdani system provides a function that complies with its generating set of IF-THEN rules, which is a different exercise from that of finding the relation or consequences implied by those rules. In general, the logical consequences of a set of rules cannot be captured by a single function. Furthermore, the consequences of an IF-THEN rule in a Mamdani system can be very different from the consequences of that same rule in a system governed by the most basic principles of logical deductive inference. Thus, care must be taken when applying this tool to study “the consequences�? of a set of hypothesis. Previous analyses have typically focused on particular steps of the Mamdani process, while here we present a holistic assessment of this technique for (deductive) simulation purposes.

Artificial Economics: What, Why and How

In this paper we present our views on the distinguishing features of Artificial Economics and on its relation with Theoretical Economics – the field that in our opinion lies closest to Artificial Economics. In this context, we discuss various reasons why conducting research on Artificial Economics may be worthwhile, and provide general guidelines on how to go about it. Our view is that Artificial Economics and Theoretical Economics share the same goals, do not differ conceptually as much as it is sometimes perceived, and their approaches are certainly complementary.

THE “WIN-CONTINUE, LOSE-REVERSE” RULE IN OLIGOPOLIES: ROBUSTNESS OF COLLUSIVE OUTCOMES

The so-called “Win-Continue, Lose-Reverse” (WCLR) rule is a simple iterative procedure that can be used to choose a value for any numeric variable (e.g., setting a price or a production level). The rule dictates that one should evaluate the effect on profits of the last adjustment made to the value (e.g., a price variation), and keep on changing the value in the same direction if the adjustment led to greater profits, or reverse the direction of change otherwise. Somewhat surprisingly, this simple rule has been shown to lead to collusive outcomes in Cournot oligopolies, even though its application requires no information about the other firms’ profits or choices. In this paper, we show that the convergence of the WCLR rule toward collusive outcomes can be very sensitive to small independent perturbations in the cost functions or in the income functions of the firms. These perturbations typically push the process toward the Nash equilibrium of the one-shot game. We also explore the behavior of WCLR against other strategies and demonstrate that WCLR is easily exploitable. We then conduct a similar analysis of the WCLR rule in a differentiated Bertrand model, where firms compete in prices.