Gary Polhill

Transient and asymptotic dynamics of reinforcement learning in games

Abstract Reinforcement learners tend to repeat actions that led to satisfactory outcomes in the past, and avoid choices that resulted in unsatisfactory experiences. This behavior is one of the most widespread adaptation mechanisms in nature. In this paper we fully characterize the dynamics of one of the best known stochastic models of reinforcement learning [Bush, R., Mosteller, F., 1955. Stochastic Models of Learning. Wiley & Sons, New York] for 2-player 2-strategy games. We also provide some extensions for more general games and for a wider class of learning algorithms. Specifically, it is shown that the transient dynamics of Bush and Mostellers model can be substantially different from its asymptotic behavior. It is also demonstrated that in general—and in sharp contrast to other reinforcement learning models in the literature—the asymptotic dynamics of Bush and Mostellers model cannot be approximated using the continuous time limit version of its expected motion.

Nonlinearities in biodiversity incentive schemes: A study using an integrated agent-based and metacommunity model

We report results from over 20,000 runs of a coupled agent-based model of land use change and species metacommunity model. We explored the effect of increasing government incentive to improve biodiversity, in the context of other influences on land manager decision-making: aspirations, input costs, and price variability. The experiments test the four kinds of policy varying along two dimensions: activity-versus-outcome-based incentive, and individual-versus-collective incentive. The results from the experiments using boundedly rational agents, and comparison with profit-maximisation reveal thresholds in incentive schemes, where a sharp increase in environmental benefit occurs for a small increase in incentive. Further, the context affects the level of incentive at which turning points occur, and the degree of effect. Variability in outcome can also change with incentive and context, and some evidence suggests that environmental benefits are not always monotone increasing functions of incentives. Intuitively, if the incentive signal is large enough, land managers will farm the subsidy; and if the subsidy does not exactly match desired landscape outcomes, deterioration in environmental benefits may occur for higher incentives. Our results, whilst they suggest that outcome-based incentives may be more robust than activity-based, also highlight the importance of context in determining the success of agri-environmental incentive schemes. As such, they lend theoretical support to schemes, such as the Scottish Rural Development Programme, that include a localised component. Highlights? We examine the effect of increasing agri-environmental incentives on biodiversity. ? Experiments are conducted using a coupled agent-based and metacommunity model. ? Results show a nonlinear relationship between incentive and biodiversity. ? Scenarios often show declining biodiversity at high incentives. ? Context-sensitivity supports schemes designed to include a local component.

Documenting Social Simulation Models: The ODD Protocol as a Standard

The clear documentation of simulations is important for their communication, replication, and comprehension. It is thus helpful for such documentation to follow minimum standards. The ‘overview, design concepts, and details’ document protocol (ODD) is specifically designed to guide the description of individual- and agent-based simulation models (ABMs) in journal articles. Popular among ecologists, it is also increasingly used in the social simulation community. Here, we describe the protocol and give an annotated example of its use, with a view in facilitating its wider adoption and encouraging higher standards in simulation description.

A semantic workflow mechanism to realise experimental goals and constraints

Workflow technologies provide scientific researchers with a flexible problem-solving environment, by facilitating the creation and execution of experiments from a pool of available services. In this paper we argue that in order to better characterise such experiments we need to go beyond low-level service composition and execution details by capturing higher-level descriptions of the scientific process. Current workflow technologies do not incorporate any representation of such experimental constraints and goals, which we refer to as the scientistpsilas intent. We have developed a framework based upon use of a number of semantic Web technologies, including the OWL ontology language and the semantic Web rule language (SWRL), to capture scientistpsilas intent. Through the use of a social simulation case study we illustrate the benefits of using this framework in terms of workflow monitoring, workflow provenance and enrichment of experimental results.

Using provenance to analyse agent-based simulations

In this paper we investigate the role of provenance in agent-based simulation and discuss typical queries we need to perform over a provenance record. Using examples based on a simulation environment using OWL ontologies to represent the state and structure of the simulation, we discuss some of the challenges posed by the large volume of fine-grained provenance associated with the execution of simulation models.

A decision-making model for environmental behavior in agent-based modeling

Agent-based modeling (ABM) is an increasingly popular technique for modeling organizations or societies. In this paper, a new approach for modeling decision-making for the environmental decisions of agents in an organization modeled using ABM is devised. The decision-making model has been constructed using data obtained by responses of individuals of the organizations to a questionnaire. As the number of responses is small, while the number of variables measured is relatively high, and obtained decision rules should be explicit, decision trees were selected to generate the model after applying different techniques to properly preprocess the data set. The results obtained for an academic organization are presented.

How Precise Are the Specifications of a Psychological Theory? Comparing Implementations of Lindenberg and Steg’s Goal-Framing Theory of Everyday Pro-environmental Behaviour

This chapter compares four implementations of (Lindenberg and Steg, J Soc Issues 63(1):117–137, 2007) Goal-Framing Theory of everyday pro-environmental behaviour. Two are from different versions of CEDSS (Community Energy Demand Social Simulator, versions 3.3 and 3.4); the other two are different versions of a completely different model that also draws on Goal-Framing Theory (Rangoni and Jager, Modeling social phenomena in spatial context. Lit Verlag, Zurich, Switzerland, 2013). We find that despite some similarities in the models, the implementations are different in a number of important ways, driven in part by the case studies to which they are applied, but also by areas where Goal-Framing Theory doesn’t specify any mechanism. We anticipate that as more and more agent-based models draw on social theories, comparisons such as that herein will enable advances in both modelling and the social sciences.

Lessons learnt from the deployment of a semantic virtual research environment

The ourSpaces Virtual Research Environment makes use of Semantic Web technologies to create a platform to support multi-disciplinary research groups. This paper introduces the main semantic components of the system: a framework to capture the provenance of the research process, a collection of services to create and visualise metadata and a policy reasoning service. We also describe different approaches to authoring and accessing metadata within the VRE. Using evidence gathered from data provided by the users of the system we discuss the lessons learnt from deployment with three case study groups.

The Importance of Ontological Structure: Why Validation by ‘Fit-to-Data’ Is Insufficient

This chapter will briefly describe some common methods by which people make quantitative estimates of how well they expect empirical models to make predictions. However, the chapter’s main argument is that fit-to-data, the traditional yardstick for establishing confidence in models, is not quite the solid ground on which to build such belief some people think it is, especially for the kind of system agent-based modelling is usually applied to. Further, the chapter will show that the amount of data required to establish confidence in an arbitrary model by fit-to-data is often infeasible, unless there is some appropriate ‘big data’ available. This arbitrariness can be reduced by constraining the choice of model. In agent-based models, these constraints are introduced by their descriptiveness rather than by removing variables from consideration or making assumptions for the sake of simplicity. By comparing with neural networks, we show that agent-based models have a richer ontological structure. For agent-based models, in particular, this richness means that the ontological structure has a greater significance and yet is all too commonly taken for granted or assumed to be ‘common sense’. The chapter therefore also discusses some approaches to validating ontologies.

Lessons Learned Replicating the Analysis of Outputs from a Social Simulation of Biodiversity Incentivisation

This chapter reports on an exercise in replicating the analysis of outputs from 20,000 runs of a social simulation of biodiversity incentivisation (FEARLUS-SPOMM) as part of the MIRACLE project. Typically, replication refers to reconstructing the model used to generate the output from the description thereof, but for larger-scale studies, the output analysis itself may be difficult to replicate even when given the original output files. Tools for analysing simulation output data do not facilitate keeping records of what can be a lengthy and complicated process. We provide an outline design for a tool to address this issue, and make some recommendations based on the experience with this exercise.