Models we Can Trust: Toward a Systematic Discipline of (Agent-Based) Model Interpretation and Validation
aa r X i v : . [ c s . M A ] F e b Models we Can Trust: Toward a SystematicDiscipline of (Agent-Based) Model Interpretationand Validation [Preliminary Version of AAMAS’21 Blue Sky Track Paper]
Gabriel IstrateWest University of Timi¸soara and the e-Austria Research InstituteTimi¸soara, Romania, email: [email protected] 24, 2021
Abstract
We advocate the development of a discipline of interacting with and ex-tracting information from models, both mathematical (e.g. game-theoreticones) and computational (e.g. agent-based models). We outline some di-rections for the development of a such a discipline: - the development of logical frameworks for the systematic formal spec-ification of stylized facts and social mechanisms in (mathematical andcomputational) social science. Such frameworks would bring to attentionnew issues, such as phase transitions , i.e. dramatical changes in the va-lidity of the stylized facts beyond some critical values in parameter space.We argue that such statements are useful for those logical frameworksdescribing properties of ABM. - the adaptation of tools from the theory of reactive systems (such asbisimulation) to obtain practically relevant notions of two systems ”havingthe same behavior”. - the systematic development of an adversarial theory of model perturba-tions, that investigates the robustness of conclusions derived from modelsof social behavior to variations in several features of the social dynam-ics. These may include: activation order, the underlying social network,individual agent behavior. Keywords agent-based simulation; game-theoretic models; robustness; log-ical frameworks; adversarial perturbations; bisimulation.
Despite the recent surge in experimental studies of human behavior induced bythe availability of (mostly online) social data, a large percentage of work in themathematical and computational social sciences is still devoted to theorizing ,1hat is building models of social phenomena, rather than analyzing social data.Whether mathematical or computational agent-based ones, a dizzying varietyof models is proposed and analyzed in the scientific literature.And yet, controversy (if not outright dissatisfaction) about the status andtrue meaning of such models, and of the modeling process itself, is prevalentthroughout the social sciences. An example is the vivid debate on the role of(mathematical) models in economics [1]. Economic models can be interpretedas ”credible worlds” [2], analogies [3], (thought) experiments [4], parables/fables[5], intermediate byproducts of robustness analysis [6], or ludic devices similar tochildren’s toys [7]. In any case, the discussion about the robustness of scientificmodels , currently taking place in the Philosophy of Science literature [8], ishighly relevant.A similar debate takes place in the social simulation literature. The criticalresearch problem is that of verifying and validating agent-based models (ABM)[9, 10] (in short, the
V & V problem ). Theoretical frameworks have been pro-posed that attempt to deal with this issue, such as the generative approach tosocial simulations [11, 12], model calibration, docking/alignment [13, 14], repli-cation [15], model-to-model analysis [16], etc. But there is no consensus on whatverification and validation mean (see also [17, 18, 19, 20]).There are multiple reasons that make the V&V problem important and diffi-cult. A first reason is scale : whereas Schelling [21] could conceive his celebratedsegregation model using pen and paper only, recent simulations models andprojects aim to reach global dimensions [22, 23, 24, 25, 26]. A second reasonhas to deal with the potential social consequences : social simulations increas-ingly serve as consultants to (and implicitly affect) public policy [27, 28, 29].A dramatic illustration of this fact in the context of the global pandemic of2020 has been the controversy around the recommendations of the ImperialCollege epidemiological model [30]. This has led to significant discussion in thesocial simulation community, illustrated e.g. by the programmatic article [31]and the subsequent comments (e.g. [32, 33, 34, 35, 36, 37]). A final reasonthat makes the V&V question difficult is the very nature of simulation models,incomplete abstractions of reality, subject to complex behavior [38] that ofteninvolves multiple types of emergence [39].It has been noted [40] that the proposals put forward in the ABM literatureoften have an ad-hoc nature, and that a more systematic theory is needed.
Thegoal of this paper is to advocate the use of logic and formal methods as useful toolsfor the systematic development of such theories . We discuss a number of ways inwhich this may happen, and outline several research challenges associated withour proposals. The distinctive feature of the kind of frameworks we advocateis that they require a highly unusual combination of two areas withvery different languages: logic and formal methods [41], on one hand,sociological theory [42], on the other.
Importantly, the logical frameworkswe envision should actively seek to avoid becoming what Edmonds [43] calledthe “philosophical approach” to logic. Instead, they should attempt to formalizegenuine aspects of social theory (e.g. organizational logic, see e.g. [44, 45]),help with addressing issues related to V&V, and serve as ”middleware” to the2gent-based simulations, helping in advancing conclusions that are robust andbelievable.
Is there any role for logical formalizations in describing and analyzing social dy-namics, in ABM in particular ? This is a question that seems to have been askedso many times, with so many different interpretations in mind that a completesurvey of this literature would not be particularly enlightening. Early on, Elster[46] argued that “logical theory can be applied not only in the formalization ofknowledge already obtained by other means, but that logic can enter in thecreative and constructive phase of scientific work” (op.cit. pp. 1). He exploredthe role of quantified modal logic in describing social reality, with a particularfocus towards developing his method as an alternative to Hegelian dialectics.Closer to present Hannan [47] (see also [48, 49]) proposed a rational reconstruc-tion of social theory (organization science in particular) using techniques basedon first-order predicate logic. Logical methods are, of course, well-establishedin economics. To give just one example, the so-called interactive epistemol-ogy program [50] is by now a classical part of theoretical economics, and a keyingredient of a recent proposal for a common foundation of all social sciences[51].The use of logic-based methods would certainly not be controversial to alarge part of the AAMAS audience: In fact, one could justifiably ask what isnovel in such a proposal. After all, formal methods based on temporal logic area particularly significant success story - techniques such as model checking [41]and runtime verification [52] lie behind eliminating errors in designing computercircuits, in writing software for technological artifacts (from remote controls andmobile devices to airplanes) or the Mars Rover [53]. Logical methods are widelyused in in the area of multiagent systems [54, 55, 56]. Model checking techniquesare useful in the verification of software agents [57, 58, 59] and auctions [60].Yet, the above optimism seems not to be shared by the practicingsocial simulation community.
The mentioned advances in software agents do not necessarily translate into corresponding advances on simulating socialagents [61]. The techniques developed in the former literature rely too littleon existing sociological knowledge, and address to an insufficient degree theconcerns of social scientists. Unsurprisingly, they have been criticized (Edmonds[43], see also [62, 63, 64]) as ”not useful given the state of MAS” and ”not [...]useful in either understanding or building MAS”.We believe that logical methods can indeed help in increasing the reliabilityof conclusions derived from social simulations. However, to be useful, such log-ics have to be tailored to the needs of the social scientist, not definedas an object of intrinsic mathematical interest , and have properties thatmake them useful: - the logics to be developed should be expressive enough to help formalize notonly game-theoretic aspects of social theories (see e.g. [65, 66, 67, 68]) but also3 variety of aspects of sociological theory [42, 69]. We give in the sequel twoexamples of concepts that we would like to see formalized: stylized facts and social mechanisms . - the study of logical frameworks we propose should be driven by consider-ations related to their implementation in (and applications to) ABM .Their primary goal should not be that of enabling deductive reasoning aboutsocial phenomena. Instead, they should be used to formally specify the ob-served social facts, in a way that enables the construction of automated ”mon-itors” serving as ”middleware” between the social simulation and the decisionsupport level by recognizing (and signaling) the emergence of the given fact ina given simulation run. Our proposal is naturally related to the recent call forthe development of live simulations [70], i.e. continuously feeding a simulationmodel with real-world data. In contrast, however, our proposal is related to(automatically) extracting data from the simulation , and using it to understandin a more systematic fashion the unraveling of the social dynamics. - it is not that important whether deciding implication in the newlogics is tractable (we can just run the simulation to see if a certain factbecomes true). However the model checking problem (given a description of thestate of the world, is a stylized fact true in it ?) should have efficient algorithms(see also [71]). - one problem of significant importance is the monitoring question for a logicalformula φ : given a sequence of ”states of the world” W i (corresponding to asimulation run) and a statement φ , how do we efficiently detect that φ becomestrue at some point in ( W i ) ? This is a question pertaining to runtime verification [72], so we should use the inspiration from this literature but, given the rootingof the logical frameworks in social theory, it is likely that a simple adaptationof existing logics will not be enough. - several new research topics, motivated by our vision of studying ”robust”stylized facts observed from simulation runs, may gain preeminence. We givean example: the study of ”continuity” properties of parameterized families oflogical statements, as we vary the parameters of a given model. The oppositescenario, that of emergence of critical points (phase transitions) in the propertiesof social systems (and in their logical description) is also interesting. A first application domain for the logics we envision is the formal specificationof stylized facts . There is no agreement what a stylized fact is (however, see [73],as well as [74] for some relevant philosophical work). To advance a working defi-nition, according to the former paper, at least in microeconomics, ”stylized factsare currently understood as broad, but robust enough statistical propertiespertaining to a certain economic phenomenon”.4he requirement that stylized facts are robust is crucial in deciding what isand what is not a useful stylized fact: consider e.g. the following trivial baselinescenario (only important as a pedagogical example): Each of n agents may bein one of two states, A and B . Each agent prefers state A to B . Agents arescheduled at random; when scheduled, each agent changes its state according tothe best-response dynamics , moving to the state that gives it the highest utility.Hence, when scheduled they will turn to state A (and subsequently stay thatway, even if scheduled again).An obvious conclusion about the dynamics, and a candidate stylized fact,could be the following: eventually every agent will play strategy A . This is not,however, a robust stylized fact. This can be seen by parameterizing the baselinemodel and modifying agent behavior: we will assume a single parameter ǫ ≥ A with probability 1 − ǫ and B withprobability ǫ . The baseline model corresponds to the case ǫ = 0.It is easy to see that the proposed stylized fact ceases to be true for ǫ > theconclusion that every agent eventually holds state A is not robust to even theslightest variation in agents’ choice probability , hence it cannot be consid-ered as a (robust) stylized fact . A more robust formulation is one thatclaims that agents’ state converges to a stationary distribution with each agentindependently being in A w.p. 1 − ǫ and B w.p. ǫ . Note that: - to formalize the robust version of this stylized fact we don’t deal anymore withindividual statements, but with parameterized families of logical statements.They encode a (single) social fact, expressed slightly differently across variationsof the model. - in a very well-defined intuitive sense the baseline fact (all agents even-tually adopt state A ) is ”the limit”, as ǫ → of the correspondingparameterized statements for ǫ > . Existing logical frameworks cannot,however, deal with such examples: while probabilistic/continuous logical frame-works (and their model checking) exist and might be useful in ABM [75, 76],and parametricity is important in such settings [77], at the metalevel logic is stilllargely a discrete framework, with no concept of ”distance between statements”,or ”continuous limits of statements” - in other scenarios the continuous behavior of stylized facts is no longer true.Instead, social systems display phase transitions : abrupt changes in the va-lidity of certain stylized facts beyond some critical value of a given parameter.While the study of phase transitions is a well established topic in ComplexSystems and A.I. [78, 79], with phase transitions apearing even in settings rel-evant to model checking [80], the logical study of such ”phase transitions” isstill a relatively underdeveloped area. An exception is the topic of ”zero-onelaws” in the theory of random graphs [81]. There are many social phenomenawhere such concepts seem relevant. An example is the discussion about tippingpoints . Whether one talks about natural or social phenomena [82] there is aconsiderable interest in anticipating such tipping points [83]. In the theory of5andom graphs the characterization of monotone properties that have ”phasetransitions” is fairly well understood: such properties have a ”global” nature,depending crucially on the presence of most of the edges of the network [84].In contrast ”local properties”, e.g. the existence of a fixed subgraph, lack aphase transition [85]. The nature of logical theories in which one formulatesthe stylized facts also impacts the detection of tipping points: for instance, theemergence of the giant component in a random graph cannot be ”sensed” byfirst-order logic [86]. Finally, similar results exist in scenarios with a dynamicalflavor: start with an empty graph, add random edges, measuring the time whena certain graph property appears. It may be possible to extend such results tosettings relevant to ABM: Challenge 1.
Develop a theory of logical frameworks that admit ”parameter-ized statements”, and study ”phase transitions” in such statements. Ideally thisstudy would yield algorithmic methods to anticipate ”tipping points” in agent-based social simulations. Having such methods would operationalize the discus-sion about the robustness of stylized facts: to argue whether a given stylized factholds in reality one could ask whether the parameters of the real world lie in theregion of the parameter space where the stylized fact varies continuously.
It is not only (stylized) facts that are in need of a logical formalization. Afterall, in a social simulation we are not interested in facts only, but in illuminatingthe causal reasons that lead to their emergence. Often (e.g. in the area ofAnalytical Sociology [69]) such causal explanations involve social mechanisms [87, 88, 89].There is little consensus what a social mechanism is: Hedstr¨om ([87] pp. 25)compiles a list of seven definitions (due to Bunge, Craver, Elster, Hedstr¨om andSwedberg, Little and Stinchcombe). Of these seven, the most useful is due toMachamer ([90], also [91, 92]). As paraphrased in [87] “mechanisms can be saidto consist of entities (with their properties) and the activities that these entitiesengage in, either by themselves or in concert with other entities. These activi-ties bring about change [...]. A social mechanism, as here defined, describes aconstellation of entities and activities that are organized such that they regularlybring about a certain type of outcome. We explain a social phenomenon by re-ferring to the social mechanism by which such phenomena are regularly broughtabout” .Social mechanisms are complemented by other approaches: Hedstr¨om lists covering-law explanations [93] and statistical explanations. These alternativesare not mutually exclusive: social mechanisms can, e.g., be sometimes inferredfrom statistical considerations; they can have themselves stochastic/statisticalingredients.In any case, whatever social mechanisms are, they seem to have a complexstructure: they can appear in families [94], can concatenate [95] and be hierar-chically nested [91]. It seems, therefore, that:6
Verifying and validating social models (including simulation models) needsto address issues pertaining to explanation and causality. Statistical testingguidelines pertaining to replication, such as those discussed in [9], or generativeexplanations such as those proposed in [11, 12] are necessary but not sufficient.On the other hand social mechanisms, being in one acception “interpretationsin term of individual behavior of a model that abstractly reproduces the phe-nomenon that needs explaining” [94] naturally complete and complement thesemethods (see also [96]). - The role of social mechanisms in validating social models could be informallydescribed as follows: simulations should reproduce known social mechanismsthat are part of the expert knowledge in the area of concern and, of course,perhaps suggest new ones. - In accord with [97], “formalizing models is a prerequisite to illuminate socialmechanisms” and may help in making this notion precise. As a consequence wepropose the following
Challenge 2.
Give logical formalizations of the various notions of social mech-anism in Analytical Sociology, and use these formalizations for the automaticrecognition and inference of concrete social mechanisms in ABM runs.
It is clear by now that some form of robustness analysis [8] is crucial to the veri-fication and validation of social models. The concept has been heavily discussedin the Philosophy of Science literature, and can be applied to both mathematicalmodels (e.g. the robust Volterra principle [98]) and to ABM (see [99]).In contrast there is relatively little work on approaches to robustness with apractical potential: it is known, for instance, that scheduling order can severelyimpact the conclusions derived from game-theoretic and related models [100,101]: indeed, a rich literature on this topic has developed in the cellular au-tomata community (e.g. [102, 103, 104]). A more general direction, the adver-sarial scheduling approach put forward in [105] (see also [106, 107, 108]) advo-cates the study of mathematical and computational models under generalizedmodels of agent activation, as a way to increase the robustness of conclusions de-rived from these models. Paraphrasing [105], adversarial scheduling is specifiedby the following principles:-
Start with a “base case” stylized fact P , valid under a particular (scheduling)model, often random. Then attempt to ”break P ” by creating adversarialschedulers under which P no longer holds true. - Analyzing perhaps these examples, identify structural properties of the schedul-ing order that causally impact the validity of P . Use these insights to general-ize P ”from below” by identifying classes of schedulers (including the randomone) under which P is valid. In the process we may need to reformulate the original statement ina way that makes it hold under larger classes of schedulers , thusmaking it more robust.
As described above, adversarial scheduling is obviously important in increas-ing the robustness of conclusions drawn from mathematical models: But couldsomething like this be systematically implemented, and be useful for (logic-based specifications of) social simulations as well ? We believe that the answeris positive, and are going to give a pedagogical example, using the baseline sce-nario from Section 2.1. Indeed, one can logically describe the candidate stylizedfact in temporal logic as ( ∀ i ) ♦ [ State ( i ) = A ] (”every agent will eventually holdstate A ”). Is such a statement true under adversarial scheduling ? The an-swer is clearly no: informally, an adversarial scheduler which never schedules aparticular agent x whose state is B will preclude the system from reaching thestate ”all A ”. In other words, to ensure that the baseline stylized fact remainstrue under adversarial scheduling we need to require the scheduler to be fair .The random scheduler is fair (at least with probability 1 − o (1), as the numberof steps tends to ∞ ).Could have we reached the above conclusion about the necessity of fairnessin scheduling in a logical framework ? The answer is yes. To do so we need toconsider a simple logical description of the the effect axioms corresponding tothe baseline dynamics: Scheduled ( i ) → (cid:3) [ State ( i ) = A ](”if an agent i is scheduled then globally (from now on) the agent will havestate A ”; we formulated our axiom this way in order to avoid having to dealin this pedagogical example with the frame problem). Can we derive the state-ment ( ∀ i ) ♦ [ State ( i ) = A ], expressing the baseline stylized fact from the actionaxiom described above ? The answer is negative: to do so we would also need( ∀ i ) ♦ Scheduled ( i ) (”eventually every agent is scheduled”). However, back-ward chaining [109] applied to this example would identify the state-ment expressing scheduler fairness as a necessary condition for thevalidity of the stylized fact. This is evidence that adversarial scheduling mightbe feasible even for ABM, thus we propose the following: Challenge 3.
Extend the theory of adversarial scheduling to more central mod-els of social dynamics, including ABM.
Scheduling is not the only aspect of a mathematical or computational modelwhich could be studied from an adversarial perspective. Many other aspects aresusceptible of a similar treatment. For instance, in many game-theoretic andagent-based models the underlying dynamics takes place on a social network.One may vary this social network and attempt to understand the robustness ofthe baseline result to changes in the social network. The same can be done with(adversarial perturbations of) initial conditions. Some results in this directionhave recently appeared [110]. 8 hallenge 4.
Develop a theory of adversarial perturbation of social networksand initial conditions for models of social dynamics. Extend it and apply it toABM.
The verification and validation problem is related to the the question in thetitle of this section: when can we really consider two such models, perhaps withdifferent ontologies (e.g. system dynamics and ABM) as ”equivalent” ? Again,there is little agreement what a right answer may be. [14] argue that it isnot enough to ”eyeball” the outputs of the two models. One of more interestingattempts at an answer is [99] (Chapter 8), where model equivalence is formalizedas a ”weighted feature-matching” problem.The theory of reactive systems [111] provides an elegant mathematical notionof system equivalence: in this setting, the equivalence of two reactive systemsis formalized by the notion of bisimulation . A seminal theorem due to Hennesyand Milner [112] states that two bisimilar systems satisfy the same statementsin a certain modal logic M , and conversely. That is, bisimilar systems satisfythe same set of ”stylized facts” formalizable in M . As impressive as this resultis, there is a wide gulf between such theory and the realities of ABM. There aremultiple reasons that bisimulation is inadequate for social simulation. The mostimportant one is that bisimulation is too ”microscopic”: it requires the factthat every single move of one of the system is enabled in the corresponding stateof the second system. In contrast, cross-validation of ABM is coarser and oftenqualitative [113]. In an ABM we don’t mean to reproduce the actions of everyagents: it’s only macro patterns that we care about.
There is some hope, though, that such methods are relevant to the studyof ABM after all: recent results [114, 115, 116], some even from AAMAS [117]have related bisimulation to game-theoretic scenarios. It is thus reasonable topropose
Challenge 5.
Develop a theory of (bi)simulation of (social) systems alignedwith (an relevant to) the practice of V& V in ABM.
We believe that logical formalization plays an important rule in assessing (andincreasing) the reliability of results in social simulations. We have highlighteda couple of research directions that (if successful) would orient and ground thecurrent discussion on model validity in (computational) social sciences. Wedon’t believe that the directions we outlined are going to completely solve thisproblem. But, besides the obvious intellectual interest of developing such con-cepts, they may contribute to turning simulation and modeling in social settingsfrom largely being an art (which still is now) to an engineering discipline.9 eferences [1] Robert Sugden. Credible worlds, capacities and mechanisms.
Erkenntnis ,70(1):3–27, 2009.[2] Robert Sugden. Credible worlds: the status of theoretical models in eco-nomics.
Journal of economic methodology , 7(1):1–31, 2000.[3] Itzhak Gilboa, Andrew Postlewaite, Larry Samuelson, and DavidSchmeidler. Economic models as analogies.
The Economic Journal ,124(578):F513–F533, 2014.[4] Uskali M¨aki. Models are experiments, experiments are models.
Journalof Economic Methodology , 12(2):303–315, 2005.[5] Nancy Cartwright. Models: Parables v fables. In
Beyond Mimesis andConvention , pages 19–31. Springer, 2010.[6] Jaakko Kuorikoski, Aki Lehtinen, and Caterina Marchionni. Economicmodelling as robustness analysis.
The British Journal for the Philosophyof Science , 61(3):541–567, 2010.[7] Adam Toon.
Models as make-believe: Imagination, fiction and scientificrepresentation . Springer, 2012.[8] Michael Weisberg. Robustness analysis.
Philosophy of Science , 73(5):730–742, 2006.[9] R. Axelrod. Advancing the art of simulation in the social sciences.
Com-plexity , 3(2):16–22, 1997.[10] Claus Beisbart and Nicole J Saam.
Computer simulation validation .Springer, 2019.[11] J. Epstein. Agent-based computational models and generative social sci-ence.
Complexity , 4(5):41–60, 1999.[12] J. Epstein.
Generative Social Science: Studies in Agent-based Computa-tional Modeling . Princeton University Press, 2007.[13] R. Axtell, R. Axelrod, J. Epstein, and M. Cohen. Aligning simulationmodels: a case study and results.
Computational and Mathematical Or-ganization Theory , 1:123–141, 1996.[14] Bruce Edmonds and David Hales. Replication, replication and replication:Some hard lessons from model alignment.
Journal of Artificial Societiesand Social Simulation , 6(4), 2003.[15] Uri Wilensky and William Rand. Making models match: Replicating anagent-based model.
Journal of Artificial Societies and Social Simulation ,10(4):2, 2007. 1016] David Hales, Juliette Rouchier, and Bruce Edmonds. Model-to-modelanalysis.
Journal of Artificial Societies and Social Simulation , 6(4), 2003.[17] G¨unter K¨uppers and Johannes Lenhard. Validation of simulation: Pat-terns in the social and natural sciences.
Journal of Artificial Societies andSocial Simulation , 8(4):3, 2005.[18] Riccardo Boero and Flaminio Squazzoni. Does empirical embeddednessmatter? methodological issues on agent-based models for analytical socialscience.
Journal of Artificial Societies and Social Simulation , 8(4):6, 2005.[19] Paul Windrum, Giorgio Fagiolo, and Alessio Moneta. Empirical validationof agent-based models: Alternatives and prospects.
Journal of ArtificialSocieties and Social Simulation , 10(2):8, 2007.[20] Scott Moss. Alternative approaches to the empirical validation of agent-based models.
Journal of Artificial Societies and Social Simulation ,11(1):5, 2008.[21] Thomas C. Schelling. Dynamic models of segregation.
Journal of Mathe-matical Sociology , 1:143–186, 1971.[22] TRANSIMS web page. http://code.google.com/p/transims/, 2011.[23] S. Eubank, H. Guclu, V.S. Anil Kumar, M.V. Marathe, A. Srinivasan,Z. Toroczkai, and N. Wang. Monitoring and mitigating smallpox epi-demics: Strategies drawn from a census data instantiated virtual city.
Nature , 429(6988):180–184, 2004.[24] S. Bishop, D. Helbing, P. Lukowicz, and R. Conte. FuturICT: FET flag-ship pilot project, 2011.[25] Christopher L Barrett, Keith R Bisset, Stephen G Eubank, Xizhou Feng,and Madhav V Marathe. Episimdemics: an efficient algorithm for simu-lating the spread of infectious disease over large realistic social networks.In
SC’08: Proceedings of the 2008 ACM/IEEE Conference on Supercom-puting , pages 1–12. IEEE, 2008.[26] Christopher Barrett, Keith Bisset, Shridhar Chandan, Jiangzhuo Chen,Youngyun Chungbaek, Stephen Eubank, Yaman Evrenoso˘glu, BryanLewis, Kristian Lum, Achla Marathe, et al. Planning and response inthe aftermath of a large crisis: An agent-based informatics framework.In , pages 1515–1526. IEEE,2013.[27] W.L. Martinez. Modeling and simulation for defense and national se-curity.
Statistical methods in counterterrorism: game theory, modeling,syndromic surveillance, and biometric authentication
Communications of the ACM , 56(7):88–96, 2013.[30] Neil Ferguson, Daniel Laydon, Gemma Nedjati Gilani, Natsuko Imai,Kylie Ainslie, Marc Baguelin, Sangeeta Bhatia, Adhiratha Boonyasiri,ZULMA Cucunuba Perez, Gina Cuomo-Dannenburg, et al. Report 9: Im-pact of non-pharmaceutical interventions (npis) to reduce covid19 mortal-ity and healthcare demand. 2020.[31] Flaminio Squazzoni, J Gareth Polhill, Bruce Edmonds, Petra Ahrweiler,Patrycja Antosz, Geeske Scholz, ´Emile Chappin, Melania Borit, HarkoVerhagen, Francesca Giardini, et al. Computational models that matterduring a global pandemic outbreak: A call to action.
Journal of ArtificialSocieties and Social Simulation , 23(2), 2020.[32] Carlos A de Matos Fernandes and Marijn A Keijzer. No one can predictthe future: More than a semantic dispute.
Review of Artificial Societiesand Social Simulation , 2020.[33] Patrick Steinmann, Jason R Wang, George AK van Voorn, and Jan HKwakkel. Don’t try to predict COVID-19. if you must, use deep uncer-tainty methods.
Review of Artificial Societies and Social Simulation , 2020.[34] Bruce Edmonds. Basic modelling hygiene–keep descriptions about modelsand what they model clearly distinct.
Review of Artificial Societies andSocial Simulation , 2020.[35] Edmund Chattoe-Brown. The policy context of covid19 agent-based mod-elling.
Review of Artificial Societies and Social Simulation , 2020.[36] Umberto Gostoli and Eric Silverman. Sound behavioural theories, notdata, is what makes computational models useful.
Review of ArtificialSocieties and Social Simulation , page 22, 2020.[37] Bruce Edmonds. Good modelling takes a lot of time and many eyes.
Review of Artificial , 2020.[38] K. Sawyer.
Social Emergence . Cambridge University Press, 2005.[39] N. Gilbert. Varieties of emergence. In
Proceedings of the Agent’02 Confer-ence: Social agents: ecology, exchange, and evolution , Chicago, IL, 2002.[40] Iris Lorscheid, Uta Berger, Volker Grimm, and Matthias Meyer. Fromcases to general principles: A call for theory development through agent-based modeling.
Ecological Modelling , 393:153–156, 2019.[41] G. Clarke, O. Grumberg, and U. Peled. Model checking.
MIT Press , 1999.1242] James S Coleman.
Foundations of social theory . Harvard university press,1994.[43] B. Edmonds. How formal logic can fail to be useful for modelling ordesigning MAS.
Regulated Agent-Based Social Systems , pages 1–15, 2004.[44] Davide Grossi, Frank Dignum, Mehdi Dastani, and Lamb`er Royakkers.Foundations of organizational structures in multiagent systems. In
Pro-ceedings of the fourth international joint conference on Autonomous agentsand multiagent systems , pages 690–697, 2005.[45] Virginia Dignum and Frank Dignum. A logic of agent organizations.
LogicJournal of the IGPL , 20(1):283–316, 2012.[46] J. Elster.
Logic and society: Contradictions and possible worlds . WileyNew York, 1978.[47] M.T. Hannan, L. P´olos, and G. Carroll.
Logics of organization theory:Audiences, codes, and ecologies . Princeton University Press, 2007.[48] G. P´eli, J. Bruggeman, M. Masuch, and B.O. Nuall´ain. A logical approachto formalizing organizational ecology.
American Sociological Review , pages571–593, 1994.[49] G.L. Peli, L. P´olos, and M.T. Hannan. Back to inertia: Theoretical impli-cations of alternative styles of logical formalization.
Sociological Theory ,18(2):195–215, 2000.[50] R.J. Aumann. Collected papers-vol. 1.
M.I.T. Press , 2000.[51] H. Gintis.
The bounds of reason: game theory and the unification of thebehavioral sciences . Princeton University Press, 2009.[52] H. Barringer, A. Goldberg, K. Havelund, and K. Sen. Rule-based runtimeverification. In
Verification, Model Checking, and Abstract Interpretation ,pages 277–306. Springer, 2004.[53] G. Brat, D. Drusinsky, D. Giannakopoulou, A. Goldberg, K. Havelund,M. Lowry, C. Pasareanu, A. Venet, W. Visser, and R. Washington. Ex-perimental evaluation of verification and validation tools on martian roversoftware.
Formal Methods in System Design , 25(2):167–198, 2004.[54] M. Wooldridge.
Reasoning about rational agents . MIT Press, 2000.[55] M. Wooldridge.
An introduction to multiagent syst.
Wiley, 2002.[56] Y. Shoham and K. Leyton-Brown.
Multiagent systems: Algorithmic,game-theoretic, and logical foundations . Cambridge University Press,2009. 1357] M. Wooldridge, M. Fisher, M.P. Huget, and S. Parsons. Model checkingmulti-agent systems with MABLE. In
Proceedings of the first internationaljoint conference on Autonomous agents and multiagent systems: part 2 ,pages 952–959. ACM, 2002.[58] A. Lomuscio and F. Raimondi. Model checking knowledge, strategies, andgames in multi-agent systems. In
Proceedings of the fifth internationaljoint conference on Autonomous agents and multiagent systems , pages161–168. ACM, 2006.[59] A. Lomuscio, H. Qu, and F. Raimondi. Mcmas: A model checker for theverification of multi-agent systems. In
Computer Aided Verification , pages682–688. Springer, 2009.[60] Emmanuel M Tadjouddine, Frank Guerin, and Wamberto Vasconcelos.Abstractions for model-checking game-theoretic properties of auctions.In
Proceedings of the 7th international joint conference on Autonomousagents and multiagent systems-Volume 3 , pages 1613–1616, 2008.[61] Virginia Dignum and Frank Dignum. Agents are dead. long live agents! In
Proceedings of the 19th International Conference on Autonomous Agentsand MultiAgent Systems , pages 1701–1705, 2020.[62] M. Fasli. Formal systems and agent-based social simulation equals null?
Journal of Artificial Societies and Social Simulation , 7(4), 2004.[63] F. Dignum, B. Edmonds, and L. Sonenberg. The use of logic in agent-based social simulation.
Journal of Artificial Societies and Social Simula-tion , 7(4), 2004.[64] B. Gaudou, A. Herzig, E. Lorini, and C. Sibertin-Blanc. How to do socialsimulation in logic: modelling the segregation game in a dynamic logic ofassignments.
Proceedings of MABS , pages 24–35, 2011.[65] Rohit Parikh. Social software.
Synthese , 132(3):187–211, 2002.[66] Marc Pauly and Mike Wooldridge. Logic for mechanism design—a mani-festo. In
Proceedings of the 2003 Workshop on Game Theory and DecisionTheory in Agent Systems (GTDT-2003), Melbourne, Australia . Citeseer,2003.[67] Johan Van Benthem.
Logic in games . MIT press, 2014.[68] Andr´es Perea.
Epistemic game theory: reasoning and choice . CambridgeUniversity Press, 2012.[69] P. Hedstr¨om and P. Bearman.
The Oxford handbook of analytical sociology .Oxford University Press, USA, 2009.1470] Samarth Swarup and Henning S Mortveit. Live simulations. In
Pro-ceedings of the 19th International Conference on Autonomous Agents andMultiAgent Systems , pages 1721–1725, 2020.[71] Joseph Y Halpern and Moshe Y Vardi. Model checking vs. theorem prov-ing: a manifesto.
Artificial intelligence and mathematical theory of com-putation , 212:151–176, 1991.[72] Ezio Bartocci, Yli`es Falcone, Adrian Francalanza, and Giles Reger. In-troduction to runtime verification. In
Lectures on Runtime Verification ,pages 1–33. Springer, 2018.[73] Matthias Meyer. How to use and derive stylized facts for validating simula-tion models. In
Computer Simulation Validation , pages 383–403. Springer,2019.[74] Luciano Floridi. The method of levels of abstraction.
Minds and machines ,18(3):303–329, 2008.[75] Marta Kwiatkowska, Gethin Norman, and David Parker. Prism 4.0: Ver-ification of probabilistic real-time systems. In
International conference oncomputer aided verification , pages 585–591. Springer, 2011.[76] Andr´e Platzer. Logics of dynamical systems. In , pages 13–24. IEEE, 2012.[77] Davide Ancona, Angelo Ferrando, and Viviana Mascardi. Parametricruntime verification of multiagent systems. In
AAMAS , volume 17, pages1457–1459, 2017.[78] P. Cheeseman, B. Kanefsky, and W. Taylor. Where the really hard prob-lems are. In
Proceedings of the 11th IJCAI , pages 331–337, 1991.[79] Bart Selman. Stochastic search and phase transitions: Ai meets physics.In
IJCAI (1) , pages 998–1002, 1995.[80] C. Moore, G. Istrate, D. Demopoulos, and M. Vardi. A continuous-discontinuous second-order transition in the satisfiability of a class of Hornformulas.
Random Structures and Algorithms , 31(2):173–185, 2007.[81] Joel Spencer.
The strange logic of random graphs , volume 22. SpringerScience & Business Media, 2001.[82] Damon Centola, Joshua Becker, Devon Brackbill, and Andrea Baronchelli.Experimental evidence for tipping points in social convention.
Science ,360(6393):1116–1119, 2018.[83] Marten Scheffer. Foreseeing tipping points.
Nature , 467(7314):411–412,2010. 1584] E. Friedgut. Necessary and sufficient conditions for sharp thresholds ofgraph properties, and the k-SAT problem. With an appendix by J. Bour-gain.
Journal of the A.M.S. , 12:1017–1054, 1999.[85] N. Alon, P. Erd˝os, and J. Spencer.
The probabilistic method . John Wileyand Sons, second edition, 1992.[86] Saharon Shelah and Joel Spencer. Can you feel the double jump?
RandomStructures & Algorithms , 5(1):191–204, 1994.[87] P. Hedstr¨om.
Dissecting the social: on the principles of analytical sociol-ogy . Cambridge University Press, 2005.[88] P. Hedstr¨om and R. Swedberg, editors.
Social Mechanisms: An AnalyticalApproach to Social Theory . Cambridge University Press, 2006.[89] P. Demeulenaere.
Analytical sociology and social mechanisms . CambridgeUniversity Press, 2011.[90] P. Machamer, L. Darden, and C.F. Craver. Thinking about mechanisms.
Philosophy of Science , 67(1):1–25, 2000.[91] C.F. Craver. Role functions, mechanisms, and hierarchy.
Philosophy ofScience , pages 53–74, 2001.[92] C.F. Craver. When mechanistic models explain.
Synthese , 153(3):355–376,2006.[93] C. Hempel. Aspects of scientific explanation.
Free Press , 1965.[94] T. Schelling. Social mechanisms and social dynamics. in Social mecha-nisms: An analytical approach to social theory, pages 32–44, 1998.[95] D. Gambetta. Concatenations of mechanisms. in Social mechanisms: Ananalytical approach to social theory, page 102, 1998.[96] Lynne Hamill. Agent-based modelling: The next 15 years.
Journal ofArtificial Societies and Social Simulation , 13(4):7, 2010.[97] F. Squazzoni. The micro-macro link in social simulation.
Sociologica , 1(2),2008.[98] Michael Weisberg and Kenneth Reisman. The robust volterra principle.
Philosophy of science , 75(1):106–131, 2008.[99] Michael Weisberg.
Simulation and similarity: Using models to understandthe world . Oxford University Press, 2012.[100] B. Huberman and N. Glance. Evolutionary games and computer sim-ulations.
Proceedings of the National Academy of Science of the USA ,90:7716–7718, 1993. 16101] Christopher Weimer, John O Miller, Raymond Hill, and Douglas Hodson.Agent scheduling in opinion dynamics: A taxonomy and comparison usinggeneralized models.
Journal of Artificial Societies and Social Simulation ,22(4), 2019.[102] Nazim A Fates and Michel Morvan. An experimental study of robust-ness to asynchronism for elementary cellular automata.
Complex Systems ,11:1–1, 1997.[103] Olivier Bour´e, Nazim Fates, and Vincent Chevrier. Probing robustness ofcellular automata through variations of asynchronous updating.
NaturalComputing , 11(4):553–564, 2012.[104] Nazim Fates and Michel Morvan. Perturbing the topology of the gameof life increases its robustness to asynchrony. In
International Conferenceon Cellular Automata , pages 111–120. Springer, 2004.[105] Gabriel Istrate, Madhav V. Marathe, and S.S. Ravi. Adversarial schedul-ing analysis of discrete models of social dynamics.
Mathematical Structuresin Computer Science , 22(5):788–815, 2012. (journal version of [118]).[106] G. Istrate, M. Marathe, and S. Ravi. Adversarial scheduling analysis ofgame-theoretic models of norm diffusion.
Logic and Theory of Algorithms ,pages 273–282, 2008.[107] Shahrzad Shirazipourazad, Brian Bogard, Harsh Vachhani, ArunabhaSen, and Paul Horn. Influence propagation in adversarial setting: howto defeat competition with least amount of investment. In
Proceedingsof the 21st ACM international conference on Information and knowledgemanagement , pages 585–594, 2012.[108] Gabriel Istrate. Stochastic stability in schelling’s segregation model withmarkovian asynchronous update. In
International Conference on CellularAutomata , pages 416–427. Springer, 2018.[109] Russell Stuart and Norvig Peter. Artificial intelligence-a modern approach4th ed., 2020.[110] Ahad N. Zehmakan. Majority opinion diffusion in social networks: Anadversarial approach. In
Proceedings of the 35th AAAI conference (toappear) , 2021. available as arxiv.org preprint arXiv:2012.03143.[111] R Milner.
Communication and Concurrency . Prentice-Hall, Inc., 1989.[112] Matthew Hennessy and Robin Milner. Algebraic laws for nondeterminismand concurrency.
Journal of the ACM (JACM) , 32(1):137–161, 1985.[113] Scott Moss and Bruce Edmonds. Sociology and simulation: Statistical andqualitative cross-validation.
American journal of sociology , 110(4):1095–1131, 2005. 17114] Marc Pauly et al. Game constructions that are safe for bisimulation.
JFAK-Essays Dedicated to Johan van Benthem on the Occasion of His50th Birthday , 1999.[115] Julian Gutierrez, Paul Harrenstein, Giuseppe Perelli, and MichaelWooldridge. Nash equilibrium and bisimulation invariance. In , page 17, 2017.[116] Michael Wooldridge, Giuseppe Perelli, Paul Harrenstein, and JulianGutierrez. Nash equilibrium and bisimulation invariance.
Logical Methodsin Computer Science , 15, 2019.[117] Francesco Belardinelli, Rodica Condurache, Catalin Dima, Wojciech Jam-roga, and Andrew V Jones. Bisimulations for verifying strategic abilitieswith an application to threeballot. In
Proceedings of the 16th Conferenceon Autonomous Agents and MultiAgent Systems , pages 1286–1295, 2017.[118] Gabriel Istrate, Madhav V. Marathe, and S.S. Ravi. Adversarial modelsin evolutionary game dynamics. In