A Survey of Requirements for COVID-19 Mitigation Strategies. Part II: Elicitation of Requirements
aa r X i v : . [ c s . C Y ] J a n A Survey of Requirementsfor COVID-19 Mitigation StrategiesPart II: Elicitation of Requirements
Wojciech Jamroga , Interdisciplinary Centre on Security, Reliability and Trust, SnT,University of Luxembourg Institute of Computer Science, Polish Academy of Sciences,Warsaw, Poland
Abstract
The COVID-19 pandemic has influenced virtually all aspects of ourlives. Across the world, countries have applied various mitigation strate-gies, based on social, political, and technological instruments. We postu-late that multi-agent systems can provide a common platform to study(and balance) their essential properties. We also show how to obtain acomprehensive list of the properties by “distilling” them from media snip-pets. Finally, we present a preliminary take on their formal specification,using ideas from multi-agent logics.
Keywords:
COVID-19, mitigation strategies, specification, agent logics
COVID-19 has influenced virtually all aspects of our lives. Across the world,countries applied wildly varying mitigation strategies for the epidemic, rangingfrom minimal intrusion in the hope of obtaining “herd immunity”, to imposingsevere lockdowns on the other extreme. It seems clear at the first glance what allthose measures are trying to achieve, and what the criteria of success are. Butis it really that clear? Quoting an oft-repeated phrase, with COVID-19 we fight an unprecedented threat to health and economic stability [Soltani et al., 2020].While fighting it, we must protect privacy, equality and fairness [Morley et al.,2020] and do a coordinated assessment of usefulness, effectiveness, technologicalreadiness, cyber security risks and threats to fundamental freedoms and humanrights [Stollmeyer et al., 2020]. Taken together, this is hardly a straightforwardset of goals and requirements. Thus, paraphrasing [Stollmeyer et al., 2020], onemay ask:
What problem does a COVID mitigation strategy solve exactly? actual goals and require-ments for a COVID-19 mitigation strategy. One way to achieve it is to lookat what is considered relevant by the general public, and referred to in themedia. To this end, we collected a number of news quotes on the topic, or-dered them thematically and with respect to the type of concern, and presentedin [Jamroga et al., 2020]. Here, we take the news clips from [Jamroga et al.,2020], and distill a comprehensive list of goals, requirements, and most relevantrisk. The list is presented in Section 2. In Section 3, we make the first steptowards a formalization of the properties by formulas of multi-agent logics. Weconclude in Section 4.Besides potential input to the design of anti-COVID-19 strategies, the maincontribution of this paper is methodological: we demonstrate how to obtain acomprehensive and relatively unbiased specification of properties for complexMAS by searching for hints in the public space.
Specification of properties is probably the most neglected part of formal veri-fication for MAS. The research on formal verification usually concentrates ondefining the decision problem, establishing its theoretical properties, and design-ing algorithms that solve the problem at an abstract level Dastani et al. [2010].Fortunately, the algorithms are more and more often implemented in the formof a publicly available model-checker [Alur et al., 2000, Lomuscio et al., 2017,Behrmann et al., 2004, Kant et al., 2015, Kurpiewski et al., 2019]. The toolscome with examples of how to model the behavior of a system, but writingthe input formulas is generally considered easy. The big question, however, is:
Where do the formulas come from?
In a realistic multi-agent scenario, it is notclear at all. 2itigating COVID-19 illustrates the point well. Research on mitigationmeasures is typically characterized by: (a) strong focus on the native domainof the authors, and (b) focus on the details, rather than the general picture.In order to avoid “overlooking the forest for the trees,” we came up with adifferent methodology. We looked for relevant phrases that appeared in themedia, with no particular method of source selection [Jamroga et al., 2020].Then, we extracted the properties, and whenever possible generalized statementson specific measures to the mitigation strategy in general. Finally, we sortedthem thematically, and divided into 3 categories: goals , additional requirements ,and potential risks and threats .While most of the collected snippets focus on digital contact tracing, therelevance of the requirements goes clearly beyond that, and applies to all theaspects of this epidemic, as well as the ones that may happen in the future.
COVID-19 is first and foremost a threat to people’s health and lives. Accord-ingly, we begin with requirements related to this aspect of mitigation strategies.
The goal of the mitigation strategy in general, and digital measures in particular,is to:(i) provide an epidemic response [Soltani et al., 2020](ii) bring the pandemic under control [Morley et al., 2020](iii) slow the spread of the virus [Woodhams, 2020, NCS, 2020, Soltani et al.,2020, Bicheno, 2020, hel, 2020, Ilves, 2020](iv) prevent deaths [AFP, 2020](v) reduce the reproduction rate of the virus, i.e., how many people are infectedby someone with the virus [AFP, 2020].The specific goals of digital measures are to:(i) trace the spread of the virus and identify dangerous Covid-19 clusters [Ilves,2020](ii) find potential new infections [Timberg, 2020](iii) register contacts between potential carriers and those who might be in-fected [Ilves, 2020](iv) deter people from breaking quarantine [Clarance, 2020]Requirements:(1) The efforts must meet public health needs best [Soltani et al., 2020, Ilves,2020]. 32) Digital measures should be a component of the epidemic response [Soltani et al.,2020], and enhance traditional forms of contact tracing [Timberg, 2020](3) They should be designed to help the health authorities [hel, 2020].
Requirements:(1) The strategy should be effective [Soltani et al., 2020, Stollmeyer et al.,2020](2) It should make a difference [Burgess, 2020].Risks and threats:(a)
Inaccurate detection of carriers and infected people due to the limitationsof the technology and the underlying model of human interaction [Soltani et al.,2020](b) Specifically, this may adversely impact relaxation of lockdowns [Woodhams,2020](c) Misguided assurance that going out is safe [Soltani et al., 2020].
The strategy should support rapid identification and notification of the mostconcerned. That is, it should allow:(1) to identify people who might have been exposed to the virus [Zastrow, 2020](2) to alert those people [Morley et al., 2020, hel, 2020, Timberg, 2020, POL,2020].(3) The identification and notification must be rapid [Zastrow, 2020, Morley et al.,2020].
The containment strategy should enable:(1) monitoring the state of the pandemic , e.g., the outbreaks and the spreadof the virus [POL, 2020, Frasier, 2020](2) monitoring the behavior of people , in particular if they are following therules [Scott and Wanat, 2020](3) to monitor the effectiveness of the strategy [Davies, 2020].4 .1.5 Tradeoffs
There are tradeoffs between effective containment of the epidemic and otherconcerns, such as privacy and protection of fundamental freedoms [McCarthy,2020, Clarance, 2020, POL, 2020, Ilves, 2020]. E.g., effective monitoring is oftenat odds with privacy [Davies, 2020]. The strategy should(1) strike the right balance between different concerns [Ilves, 2020].We will see more tradeoff-related requirements in the subsequent sections.
Most measures to contain the epidemic are predominantly social (cf. lockdown),and have strong social and economic impact.
The containment strategy should:(1) minimize the cost to local economies and the negative impact on economicgrowth [Soltani et al., 2020, AFP, 2020](2) allow for return to normal economy and society and make resumption ofeconomic and social activities safer [Timberg, 2020, Taylor, 2020].
The containment strategy (and digital measures in particular) should:(1) ease lockdowns and home confinement [Soltani et al., 2020, Stollmeyer et al.,2020, Zastrow, 2020, Taylor, 2020](2) minimize adverse impact on social relationships and personal well-being [Soltani et al.,2020](3) prohibit economic and social discrimination on the basis of informationand technology being part of the strategy [Soltani et al., 2020](4) protect the communities that can be harmed by the collection and ex-ploitation of personal data [Soltani et al., 2020].Detailed requirements:(1) Surveillance technologies should not become compulsory for public and so-cial engagements , with unaffected individuals restricted from participatingin social and economic activities [Soltani et al., 2020].Risks and threats:(a)
Little knowledge about social impact of the measures [O’Neill et al., 2020]5b)
Discrimination and creation of social divides [Soltani et al., 2020, Mat,2020](c)
Disinformation and information abuse [Soltani et al., 2020, Woodhams,2020](d) Providing a false sense of security [Soltani et al., 2020](e)
Political manipulation , creating social unrest , and dishonest competition by false reports of coronavirus [Soltani et al., 2020](f) Too much political influence of IT companies on the decisions of sovereigndemocratic countries [Ilves, 2020].
Requirements:(1) The financial cost of the measures should be minimized [Hern, 2020](2) Minimization of the involved human resources [Scott and Wanat, 2020,Soltani et al., 2020](3)
Timeliness [Hern, 2020](4)
Coordination between different institutions and authorities[Tahir and Lima,2020, Eur, 2020], including the establishment of common standards [Tahir and Lima,2020].
In this section, we look at requirements that aim at the long-term robustnessand resilience of the social structure. (1) The mitigation strategy must be ethically justifiable [Morley et al., 2020](2) The measures should be necessary , proportionate , legitimate , just , scientif-ically valid , and time-bound [Morley et al., 2020, Woodhams, 2020, Oslo,2020, Scott and Wanat, 2020, Mat, 2020](3) They should not be invasive [Clarance, 2020] and must not be done at theexpense of individual civil rights [Bicheno, 2020, O’Neill et al., 2020, Mat,2020](4) Means of protection should be available to anyone [Morley et al., 2020](5) They should be voluntary [O’Neill et al., 2020, NCS, 2020](6) The measures must comply with legal regulations [Mat, 2020, McCarthy,2020, Wodinsky, 2020](7) Implementation and impact must also be considered [Morley et al., 2020,Woodhams, 2020] 68)
Impact assessment should be conducted and made public [Mat, 2020]. (a) Serious and long-lasting harms to fundamental rights and freedoms [Morley et al.,2020](b) Costs of not devoting resources to something else [Morley et al., 2020](c) Measures designed and implemented without adequate scrutiny [Woodhams,2020](d) Measures that support extensive physical surveillance [Woodhams, 2020](e)
Mandatory use of digital measures, collecting sensitive information , shar-ing the data with the government [Clarance, 2020, Zastrow, 2020](f) Censorship practices to silence critics and control the flow of informa-tion [Woodhams, 2020].
Privacy-related issues for COVID-19 mitigation strategies have triggered heateddiscussion, and at some point gained much media coverage. This is under-standable, since privacy and data protection is an important aspect of medicalinformation flow, even in ordinary times. Moreover, the IT measures againstCOVID-19 are usually designed by computer scientists and specialists, for whomsecurity requirements are relatively easy to identify and understand. (1) The strategy should be designed with privacy and information security inmind [Soltani et al., 2020, Timberg, 2020, O’Neill et al., 2020](2) It should mitigate privacy concerns inherent in a technological approach [Soltani et al.,2020](3) It should be anonymous under data protection laws, i.e., it cannot lead tothe identification of an individual [Burgess, 2020](4) The information about users should be protected at all times [NCS, 2020](5) The design should include recommendations for how back-end systemsshould be secured , and identify vulnerabilities as well as unintended conse-quences [Soltani et al., 2020].Risks and Threats:(a) Lack of clear privacy policies [Woodhams, 2020, Eisenberg, 2020, O’Neill et al.,2020] 7b)
Exploitation of personal information by authorities or third parties [Eisenberg,2020, Woodhams, 2020, Garthwaite and Anderson, 2020], in particular liveor near-live tracking of users’ locations and linking sensitive personal in-formation to an individual [Garthwaite and Anderson, 2020](c)
Linking different datasets at some point in the future [Wodinsky, 2020](d) Alerts can be too revealing [BBC, 2020](e) It may be possible to work out who is associating with whom [McCarthy,2020].
Here, the key question is:
What data is collected and who is it shared with ? [O’Neill et al.,2020, Soltani et al., 2020] This leads to the following requirements:(1) Clear and reasonable limits on the data collection types [Tahir and Lima,2020, Clarance, 2020, NCS, 2020, Soltani et al., 2020, Timberg, 2020](2) Limitations on how the data is used [O’Neill et al., 2020](3) In particular, the data is to be used strictly for disease control and not shared with law enforcement agencies [Clarance, 2020, Taylor, 2020](4) Less state access and control over user data [Bicheno, 2020](5) Data collection should be minimized [O’Neill et al., 2020] and based on informed consent of the participants [Ilves, 2020](6) Giving access to one’s data should be voluntary [O’Neill et al., 2020](7) One should be able to delete their personal information at any time [hel,2020, McCarthy, 2020](8) One should have the right to access their own data [hel, 2020, McCarthy,2020](9) For digital measures, the user should be able to remove the software and disable more invasive features [hel, 2020].Risks and threats:(a) Data storage that can be hacked and exploited [Davies, 2020, Zastrow,2020, Woodhams, 2020](b)
Data breaches due to insider threats [Eisenberg, 2020](c)
Function creep and state surveillance [Zastrow, 2020](d)
Sharing data across agencies or selling to a third party [Eisenberg, 2020,Woodhams, 2020](e) Integration with commercial services [Woodhams, 2020].8 .4.3 Sunsetting, Safeguards, and Monitoring
Requirements:(1) Sunsetting: the measures should be terminated as soon as possible [Scott and Wanat,2020, Soltani et al., 2020, hel, 2020](2) Data should be eventually or even periodically destroyed [Scott and Wanat,2020, O’Neill et al., 2020, hel, 2020, McCarthy, 2020, Soltani et al., 2020,Timberg, 2020], in particular when it is no longer needed to help managethe spread of coronavirus [NCS, 2020](3)
Transparency of data collection [O’Neill et al., 2020](4) There should be clear policies to prevent abuse [O’Neill et al., 2020](5) Privacy must be backed up with clear lines of accountability and processesfor evaluation and monitoring [Wodinsky, 2020](6)
Judicial oversight must be provided [Soltani et al., 2020](7) Safeguards should be backed by an independent figure [Scott and Wanat,2020].Risks and threats:(a) Surveillance might continue to be used after the threat of the coronavirusrecedes [Garthwaite and Anderson, 2020](b) Data can stay with the government longer than necessary [Scott and Wanat,2020].
Requirements:(1) People must get the information they need to protect themselves and oth-ers [BBC, 2020](2) There must be protections against economic and social discrimination based on information and technology designed to fight the pandemic, inparticular with respect to communities vulnerable to collection and ex-ploitation of personal data [Soltani et al., 2020](3) Information should be used in such a way that people who fear beingjudged will not put other people in danger [BBC, 2020].Risks and threats:(a)
Fear of social stigma [BBC, 2020](b) Online judgement and ridicule [BBC, 2020].9 .4.5 Privacy vs. Epidemiological Efficiency
There is a tradeoff between protecting privacy vs. collecting and processing allthe information that can be useful in fighting the epidemic:• Privacy hinders making the best possible use of the data , including anal-ysis of the population , contact matching , modeling the network of con-tacts , enabling epidemiological insights such as revealing clusters and su-perspreaders , and providing advice to people [McCarthy, 2020, Zastrow,2020, Taylor, 2020]• Privacy-preserving solutions put users in more control of their information and require no intervention from a third party [McCarthy, 2020].The relationship is not simply antagonistic, though:• Privacy is instrumental in building trust . Conversely, lack of privacy un-dermines trust, and may hinder the epidemiological, economic, and socialeffects of the mitigation activities [Eisenberg, 2020]. While it might be necessary to waive users’ privacy in the short term in orderto contain the epidemic, one must look for mechanisms such that(1) exploiting the risks would require significant effort by the attackers forminimal reward [Zastrow, 2020].
The measures must be adopted and followed by the people, in order to makethem effective.
Goals:(i)
High acceptance rate for the mitigation measures [Timberg, 2020].(ii)
Creating incentives and overcoming incentive problems for individual peo-ple to adopt the strategy [Soltani et al., 2020]Risks and threats:(a) Lack of immediate benefits for the participants [Soltani et al., 2020](b) Perceived privacy and security risks [Timberg, 2020](c) Some measures can divert attention from more important measures , and make people less alert [Szymielewicz et al., 2020](d) Creating false sense of security from the pandemic [Frasier, 2020].10ountermeasures:(a) Pointing out indirect benefits (e.g., opening of the schools and businesses,reviving the national economy) [Soltani et al., 2020](b) Reliance on personal responsibility [Stollmeyer et al., 2020].
Requirements:(1)
Enough people should download and use the app to make it effective [Timberg,2020, O’Neill et al., 2020, Zastrow, 2020, Bezat, 2020]. Note: this require-ment is graded rather than binary O’Neill [2020], Hinch et al. [2020].Risks and threats:(a) Lack of users’ trust [Burgess, 2020, Eisenberg, 2020], see also the connec-tion between privacy and trust in Section 2.4.5(b) Lack of social knowledge and empathy by the authorities [POL, 2020].
General requirements:(1) The concrete measures and tools must be operational [McCarthy, 2020,Scott and Wanat, 2020](2) In particular, they should be compatible with their environment of imple-mentation [Wodinsky, 2020](3) Design and implementation should be transparent [O’Neill et al., 2020,SDZ, 2020].Specific requirements for digital measures:(1) They should be compatible with most available devices [Wodinsky, 2020](2) Reasonable use of battery [Wodinsky, 2020](3)
Usable interface [Wodinsky, 2020](4)
Accurate measurements of how close two devices are [Zastrow, 2020](5)
Cross-border interoperability [Cyb, 2020](6) Possibility to verify the code by the public and experts [SDZ, 2020].11 .7 Evaluation and Learning for the Future
COVID-19 mitigation activities should be rigorously assessed. Moreover, theiroutcomes should be used to extend our knowledge about the pandemic, andbetter defend ourselves in the future. The main goal here is:(i) to use the collected data in order to develop efficient infection controlmeasures and gain insight into the effect of changes to the measures forfighting the virus [hel, 2020, McCarthy, 2020].Requirements:(1) A review and exit strategy should be defined [Morley et al., 2020](2) Before implementing the measures, an institutional assessment is neededof their usefulness, effectiveness, technological readiness, cyber-securityrisks and threats to fundamental freedoms and human rights [Stollmeyer et al.,2020](3) After the pandemic, there must be the society’s assessment whether thestrategy has been effective and appropriate [BBC, 2020](4) The assessments should be conducted by an independent body at regularintervals [Morley et al., 2020]. Here, we briefly show how the requirements presented in Section 2 can be rewrit-ten in a more formal way. To this end, we use modal logics for distributedand multi-agent systems that have been in constant development for over 40years [Emerson, 1990, Fagin et al., 1995, Wooldridge, 2000, Broersen et al., 2001,Alur et al., 2002, Bulling et al., 2015]. Note that the following specifications areonly semi -formal, as we do not fix the models nor give the precise semantics ofthe logical operators and atomic predicates. We leave that step for the futurework.
The simplest kind of requirements are those that refer to achievement or main-tenance of a particular state of affairs. Typically, they can be expressed byformulas of the branching-time logic
CTL ⋆ [Emerson, 1990], with path quan-tifiers E ( there is a path ), A ( for all paths ), and temporal operators X ( in thenext moment ), F ( sometime from now on ), G ( always from now on ), and U ( until ). For example, goal (ii) in Section 2.1 can be tentatively rewritten as the CTL ⋆ formula A F control-pandemic , control-pandemic must eventu-ally hold. Similarly, goal (iii) can be expressed by formula ∀ n . ( R0=n ) → A F (
R0 CTL ⋆ [Baier and Kwiatkowska, 1998] or ATL ∗ [Chen et al., 2013]. For instance, formula hh a ii P ≥ . F t ≤ hh i ii compl ≤ F K i (F − exposed i ) , refines the previous specification by demanding that the authority can success-fully notify i with probability at least . The requirements presented in Section 2, and formalized above, refer to the“correctness” of a given mitigation strategy. Two meta-properties, well knownin computer science, can be also useful in case of the present scenario, namely diagnosability and resilience [Ezekiel and Lomuscio, 2017]. Given a correctnessrequirement Φ and a responsible agent a , those can be expressed by the followingtemplates: A G ( ¬ Φ → hh a ii F K a ¬ Φ) (diagnosability) A G ( ¬ Φ → hh a ii F Φ) (resilience) The templates can be used e.g. for monitoring-type requirements. Ideally, one would like to automatically evaluate COVID-19 strategies with re-spect to the requirements, and choose the best one. In the future, we planto use model checking tools, such as MCMAS [Lomuscio et al., 2017], Up-paal [Behrmann et al., 2004], or PRISM [Kwiatkowska et al., 2002], to formally14erify our formulas over micro-level models created to simulate and predict theprogress of the pandemic [Neil M. Ferguson, Daniel Laydon, Gemma Nedjati-Gilani et al.,2020, Adamik et al., 2020, Bock et al., 2020, McCabe et al., 2020]. As we al-ready pointed out, different requirements may be in partial conflict. Thus,selecting an optimal mitigation strategy may require solving a multicriterialoptimization problem [Zionts, 1981, Collette and Siarry, 2004, Radulescu et al.,2020], e.g., by identifying the Pareto frontier and choosing a criterion to selecta point on the frontier. In this paper, we make the first step towards a systematic analysis of strategiesfor effective and trustworthy mitigation of the current pandemic. The strategiesmay incorporate medical, social, economic, as well as technological measures.Consequently, there is a large number of medical, social, economic, and tech-nological requirements that must be taken into account when deciding whichstrategy to adopt. For computer scientists, the latter kind of requirements ismost natural, which is exactly the pitfall that computer scientists must avoid.The goals (and acceptability criteria) for a mitigation strategy are much morediverse, and we must consciously choose a solution that satisfies the multiplecriteria to a reasonable degree. We suggest that formal methods for MAS pro-vide an excellent framework for that. We also propose a methodology to collectpreliminary requirements while avoiding the usual bias of research papers. Acknowledgments. The author acknowledges the support of the Luxem-bourg National Research Fund (FNR) under the COVID-19 project SmartExit,and the support of the National Centre for Research and Development Poland(NCBR) and the Luxembourg National Research Fund (FNR), under the Pol-Lux/CORE project STV (POLLUX-VII/1/2019). References Coronavirus privacy: Are South Korea’s alerts too re-vealing? BBC News , 5 March 2020. URL . Retrieved on27.05.2020.Cybernetica proposes privacy-preserving decentralised architecture for COVID-19 mobile application for Estonia. Cybernetica , 6 May 2020. URL https://cyber.ee/news/2020/05-06/ . Retrieved on 15.05.2020.Coronavirus: Member States agree on an interoperabil-ity solution for mobile tracing and warning apps. Eu-ropean Commission - Press release , 16 June 2020. URL https://ec.europa.eu/digital-single-market/en/news/coronavirus-member-states-agree-interoperability-solution-mobile-tracing-and-warning-apps .Retrieved on 26.06.2020. 15egal advice on smartphone contact tracing pub-lished. matrix chambers , 3 May 2020. URL .Retrieved on 26.05.2020.NHS COVID-19: the new contact-tracing appfrom the nhs. NCSC , 14 May 2020. URL .Retrieved on 15.05.2020.Getting it right: States struggle with contact trac-ing push. POLITICO , 17 May 2020. URL .Retrieved on 17.05.2020.German coronavirus tracing app now available in Lux-embourg. RTL Today , 25 June 2020. URL https://today.rtl.lu/news/luxembourg/a/1539625.html . Retrievedon 25.06.2020.Corona-app soll open source werden. Süddeutsche Zeitung , 6 May 2020. URL .Retrieved on 26.05.2020.Together we can fight coronavirus — Smittestopp. helsenorge , 28 April 2020.URL https://helsenorge.no/coronavirus/smittestopp?redirect=false .Retrieved on 28.05.2020.Barbara Adamik, Marek Bawiec, Viktor Bezborodov, Przemyslaw Biecek, Wolf-gang Bock, Marcin Bodych, Jan Pablo Burgard, Tyll Krueger, Agata Mi-galska, Tomasz Ożański, Barbara Pabjan, Magdalena Rosińska, MalgorzataSadkowska-Todys, Piotr Sobczyk, and Ewa Szczurek. Estimation of thesevereness rate, death rate, household attack rate and the total numberof COVID-19 cases based on 16 115 Polish surveillance records. , 2020. doi: http://dx.doi.org/10.2139/ssrn.3696786. URL https://ssrn.com/abstract=3696786 .AFP. Major finding: Lockdowns averted 3 million deaths in 11European nations: study. RTL Today , 9 June 2020. URL https://today.rtl.lu/news/science-and-environment/a/1530963.html .Retrieved on 25.06.2020.N. Alechina, B. Logan, H.N. Nguyen, and A. Rakib. Resource-boundedalternating-time temporal logic. In Proceedings of International Joint Confer-ence on Autonomous Agents and Multiagent Systems (AAMAS) , pages 481–488, 2010. 16. Alechina, B. Logan, H.N. Nguyen, and F. Raimondi. Model-checking forresource-bounded ATL with production and consumption of resources. Jour-nal of Computer and System Sciences , 88:126–144, 2017. doi: 10.1016/j.jcss.2017.03.008.R. Alur, L. de Alfaro, T. A. Henzinger, S.C. Krishnan, F.Y.C. Mang, S. Qadeer,S.K. Rajamani, and S. Tasiran. MOCHA: Modularity in model checking.Technical report, University of Berkeley, 2000.R. Alur, T. A. Henzinger, and O. Kupferman. Alternating-time Temporal Logic. Journal of the ACM , 49:672–713, 2002. doi: 10.1145/585265.585270.Christel Baier and Marta Z. Kwiatkowska. Model checking for a probabilisticbranching time logic with fairness. Distributed Comput. , 11(3):125–155, 1998.doi: 10.1007/s004460050046.Françoise Baylis. Ten reasons why immunity passports area bad idea. Nature Comment , 21 May 2020. URL . Retrievedon 26.06.2020.G. Behrmann, A. David, and K.G. Larsen. A tutorial on uppaal . In FormalMethods for the Design of Real-Time Systems: SFM-RT , number 3185 inLNCS, pages 200–236. Springer, 2004.Jean-Michel Bezat. L’application StopCovid, activée seulement par 2% de lapopulation, connaît des débuts décevants. Le Monde , 10 June 2020. URL .Retrieved on 26.06.2020.Scott Bicheno. Unlike France, Germany decides to do smartphone con-tact tracing the Apple/Google way. telecoms.com , 27 April 2020. URL https://telecoms.com/503931/unlike-france-germany-decides-to-do-smartphone-contact-tracing-the-apple-google-way/ .Retrieved on 24.06.2020.Wolfgang Bock, Barbara Adamik, Marek Bawiec, Viktor Bezborodov, MarcinBodych, Jan Pablo Burgard, Thomas Goetz, Tyll Krueger, Agata Migalska,Barbara Pabjan, Tomasz Ozanski, Ewaryst Rafajlowicz, Wojciech Rafa-jlowicz, Ewa Skubalska-Rafajlowicz, Sara Ryfczynska, Ewa Szczurek, andPiotr Szymanski. Mitigation and herd immunity strategy for COVID-19is likely to fail. medRxiv , 2020. doi: 10.1101/2020.03.25.20043109. URL .M.E. Bratman. Intentions, Plans, and Practical Reason . Harvard UniversityPress, 1987.T. Brihaye, A. Da Costa Lopes, F. Laroussinie, and N. Markey. ATL withstrategy contexts and bounded memory. In Proceedings of LFCS , volume5407 of Lecture Notes in Computer Science , pages 92–106. Springer, 2009.17. Broersen, M. Dastani, Z. Huang, and L. van der Torre. The BOID archi-tecture: conflicts between beliefs, obligations, intentions and desires. In J.P.Müller, E. Andre, S. Sen, and C. Frasson, editors, Proceedings of the FifthInternational Conference on Autonomous Agents , pages 9–16. ACM Press,2001.N. Bulling and B. Farwer. Expressing properties of resource-bounded systems:The logics RTL* and RTL. In Proceedings of Computational Logic in Multi-Agent Systems (CLIMA) , volume 6214 of Lecture Notes in Computer Science ,pages 22–45, 2010.N. Bulling, W. Jamroga, and J. Dix. Reasoning about temporal propertiesof rational play. Annals of Mathematics and Artificial Intelligence , 53(1-4):51–114, 2008.N. Bulling, V. Goranko, and W. Jamroga. Logics for reasoning about strategicabilities in multi-player games. In J. van Benthem, S. Ghosh, and R. Ver-brugge, editors, Models of Strategic Reasoning. Logics, Games, and Com-munities , volume 8972 of Lecture Notes in Computer Science , pages 93–136.Springer, 2015. doi: 10.1007/978-3-662-48540-8.Matt Burgess. Just how anonymous is the NHS Covid-19 contact tracing app? Wired , 12 May 2020. URL .Retrieved on 25.06.2020.T. Chen, V. Forejt, M. Kwiatkowska, D. Parker, and A. Simaitis. PRISM-games:A model checker for stochastic multi-player games. In Proceedings of Toolsand Algorithms for Construction and Analysis of Systems (TACAS) , volume7795 of Lecture Notes in Computer Science , pages 185–191. Springer, 2013.Andrew Clarance. Aarogya Setu: Why India’s Covid-19 contact trac-ing app is controversial. BBC News , 15 May 2020. URL . Retrieved on15.05.2020.P.R. Cohen and H.J. Levesque. Intention is choice with commitment. ArtificialIntelligence , 42:213–261, 1990.Y. Collette and P. Siarry. Multiobjective Optimization: Principles and CaseStudies . Springer, 2004.M. Dastani, K. Hindriks, and J.-J. Meyer, editors. Specification and Verificationof Multi-Agent Systems . Springer, 2010.Jamie Davies. UK snubs Google and Apple privacy warningfor contact tracing app. telecoms.com , 28 April 2020. URL https://telecoms.com/503967/uk-snubs-google-and-apple-privacy-warning-for-contact-tracing-app/ .Retrieved on 27.05.2020. 18manda Eisenberg. Privacy fears threaten New York City’scoronavirus tracing efforts. Politico , 4 June 2020. URL .Retrieved on 25.06.2020.E.A. Emerson. Temporal and modal logic. In J. van Leeuwen, editor, Handbookof Theoretical Computer Science , volume B, pages 995–1072. Elsevier, 1990.J. Ezekiel and A. Lomuscio. Combining fault injection and model check-ing to verify fault tolerance, recoverability, and diagnosability in multi-agent systems. Information and Computation , 254:167–194, 2017. doi:https://doi.org/10.1016/j.ic.2016.10.007.R. Fagin, J. Y. Halpern, Y. Moses, and M. Y. Vardi. Reasoning about Knowledge .MIT Press, 1995.Sarah Lewin Frasier. Coronavirus antibody tests have a math-ematical pitfall. Scientific American , 1 July 2020. URL .Retrieved on 30.10.2020.Rosie Garthwaite and Ian Anderson. Coronavirus: Alarm over ’invasive’Kuwait and Bahrain contact-tracing apps. BBC News , 16 June 2020. URL . Retrieved on30.10.2020.D.P. Guelev, C. Dima, and C. Enea. An alternating-time temporal logic withknowledge, perfect recall and past: axiomatisation and model-checking. Jour-nal of Applied Non-Classical Logics , 21(1):93–131, 2011.Alex Hern. UK abandons contact-tracing app for Appleand Google model. The Guardian , 18 June 2020. URL .Retrieved on 26.06.2020.Robert Hinch, Will Probert, Anel Nurtay, Michelle Kendall, Chris Wymant,Matthew Hall, Katrina Lythgoe, Ana Bulas Cruz, Lele Zhao, AndreaStewart, Luca Ferretti, Michael Parker, Ares Meroueh, Bryn Mathias,Scott Stevenson, Daniel Montero, James Warren, Nicole K. Mather, An-thony Finkelstein, Lucie Abeler-Dörner, David Bonsall, and ChristopheFraser. Effective configurations of a digital contact tracing app: Areport to NHSX. Technical report, Oxford University, 2020. URL https://github.com/BDI-pathogens/covid-19_instant_tracing/blob/master/Report-EffectiveConfigurationsofaDigitalContactTracingApp.pdf .Ieva Ilves. Why are Google and Apple dictating how european democ-racies fight coronavirus? The Guardian , 16 June 2020. URL .Retrieved on 24.06.2020. 19lla Jakubowska. COVID-Tech: the sinister consequencesof immunity passports. EDRi , 10 June 2020. URL https://edri.org/covid-tech-the-sinister-consequences-of-immunity-passports/ .Retrieved on 26.06.2020.W. Jamroga. Logical Methods for Specification and Verification of Multi-AgentSystems . ICS PAS Publishing House, 2015. ISBN 978-83-63159-25-2.Wojciech Jamroga, Vadim Malvone, and Aniello Murano. Natural strategicability. Artificial Intelligence , 277, 2019. doi: 10.1016/j.artint.2019.103170.Wojciech Jamroga, David Mestel, Peter B. Rønne, Peter Y. A. Ryan, andMarjan Skrobot. A survey of requirements for COVID-19 mitigationstrategies. part I: newspaper clips. CoRR , abs/2011.07887, 2020. URL https://arxiv.org/abs/2011.07887 .G. Kant, A. Laarman, J. Meijer, J. van de Pol, S. Blom, and T. van Dijk.LTSmin: High-performance language-independent model checking. In Toolsand Algorithms for the Construction and Analysis of Systems. Proceedings ofTACAS , volume 9035 of Lecture Notes in Computer Science , pages 692–707.Springer, 2015. doi: 10.1007/978-3-662-46681-0\_61.Michal Knapik, Étienne André, Laure Petrucci, Wojciech Jamroga, and Woj-ciech Penczek. Timed ATL: forget memory, just count. Journal of ArtificialIntelligence Research , 66:197–223, 2019. doi: 10.1613/jair.1.11612.Karol Kunat. Rzadowa aplikacja ma blad pozwalajacy na sprawdzenie,czy nasi znajomi podlegaja kwarantannie. Tabletowo , 29 April 2020. URL .Retrieved on 26.05.2020.Damian Kurpiewski, Wojciech Jamroga, and Michał Knapik. STV: Model check-ing for strategies under imperfect information. In Proceedings of the 18th In-ternational Conference on Autonomous Agents and Multiagent Systems AA-MAS 2019 , pages 2372–2374. IFAAMAS, 2019.M. Kwiatkowska, G. Norman, and D. Parker. PRISM: probabilistic sym-bolic model checker. In Proceedings of TOOLS , volume 2324 of LectureNotes in Computer Science , pages 200–204. Springer, 2002. doi: 10.1007/3-540-46029-2_13.F. Laroussinie and Ph. Schnoebelen. A hierarchy of temporal logics with past. Theoretical Computer Science , 148(2):303–324, 1995.A. Lomuscio, H. Qu, and F. Raimondi. MCMAS: An open-source modelchecker for the verification of multi-agent systems. International Jour-nal on Software Tools for Technology Transfer , 19(1):9–30, 2017. doi:10.1007/s10009-015-0378-x. 20uth McCabe, Mara D. Kont, Nora Schmit, Charles Whittaker, Alessan-dra Loechen, Marc Baguelin, Edward Knock, Lilith Whittles, John Lees,Patrick G.T. Walker, Azra C. Ghani, Neil M. Ferguson, Peter J. White,Christl A. Donnelly, Katharina Hauck, and Oliver Watson. Modelling ICUcapacity under different epidemiological scenarios of the COVID-19 pandemicin three western European countries. Technical Report 36 (16-11-2020), Im-perial College London, 2020.Kieren McCarthy. UK finds itself almost alone with centralizedvirus contact-tracing app that probably won’t work well, asks foryour location, may be illegal. The Register , 5 May 2020. URL .Retrieved on 26.05.2020.F. Mogavero, A. Murano, and M.Y. Vardi. Reasoning about strategies. In Proceedings of FSTTCS , pages 133–144, 2010.Jessica Morley, Josh Cowls, Mariarosaria Taddeo, and Luciano Floridi. Ethicalguidelines for COVID-19 tracing apps. Nature Comment , pages 29–31, 4 June2020. URL . Re-trieved on 9.06.2020.Neil M. Ferguson, Daniel Laydon, Gemma Nedjati-Gilani et al. Impact ofnon-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality andhealthcare demand. Technical Report 9 (16-03-2020), Imperial College Lon-don, 2020.Patrick Howell O’Neill. No, coronavirus apps don’t need 60% adop-tion to be effective. MIT Technology Review , 5 June 2020. URL .Retrieved on 25.06.2020.Patrick Howell O’Neill, Tate Ryan-Mosley, and Bobbie Johnson. Aflood of coronavirus apps are tracking us. now it’s time to keeptrack of them. MIT Technology Review , 7 May 2020. URL .Retrieved on 25.06.2020.AFP Oslo. Norway suspends virus-tracing app due to pri-vacy concerns. The Guardian , 15 June 2020. URL .Retrieved on 26.06.2020.Imogen Parker and Elliot Jones. Something to declare? surfacing issueswith immunity certificates. Ada Lovelace Institute , 02 June 2020. URL .Retrieved on 26.06.2020. 21. Penczek and A. Polrola. Advances in Verification of Time Petri Nets andTimed Automata: A Temporal Logic Approach , volume 20 of Studies in Com-putational Intelligence . Springer, 2006.Roxana Radulescu, Patrick Mannion, Diederik M. Roijers, and Ann Nowé.Multi-objective multi-agent decision making: a utility-based analysis andsurvey. Autonomous Agents and Multi Agent Systems , 34(1):10, 2020. doi:10.1007/s10458-019-09433-x.A.S. Rao and M.P. Georgeff. Modeling rational agents within a BDI-architecture. In Proceedings of the 2nd International Conference on Principlesof Knowledge Representation and Reasoning , pages 473–484, 1991.Mark Scott and Zosia Wanat. Poland’s coronavirus app offers play-book for other governments. POLITICO , 2 April 2020. URL .Retrieved on 27.05.2020.Y. Shoham and K. Leyton-Brown. Multiagent Systems - Algorithmic, Game-Theoretic, and Logical Foundations . Cambridge University Press, 2009. ISBN978-0-521-89943-7.Ashkan Soltani, Ryan Calo, and Carl Bergstrom. Contact-tracing apps are nota solution to the COVID-19 crisis. Brookings Tech Stream , 27 April 2020. URL .Retrieved on 23.06.2020.Alice Stollmeyer, Marietje Schaake, and Frank Dignum. The dutch trac-ing app ’soap opera’ - lessons for europe. euobserver , 7 May 2020. URL https://euobserver.com/opinion/148265 . Retrieved on 15.05.2020.Katarzyna Szymielewicz, Anna Obem, and Tomasz Zieliński. Jak polska walczyz koronawirusem i dlaczego aplikacja nas przed nim nie ochroni? Panop-tykon , 5 May 2020. URL https://panoptykon.org/protego-safe-ryzyka .Retrieved on.Darius Tahir and Cristiano Lima. Google and Apple’s rules for virustracking apps sow division among states. Politico , 10 June 2020. URL .Retrieved on 26.06.2020.Josh Taylor. How did the Covidsafe app go from being vitalto almost irrelevant? The Guardian , 23 May 2020. URL .Retrieved on 26.05.2020.Craig Timberg. Most Americans are not willing or able to use anapp tracking coronavirus infections. that’s a problem for Big Tech’splan to slow the pandemic. Washington Post , 29 April 2020. URL22 .Retrieved on 17.05.2020.G. Weiss, editor. Multiagent Systems. A Modern Approach to Distributed Arti-ficial Intelligence . MIT Press: Cambridge, Mass, 1999.Shoshana Wodinsky. The UK’s contact-tracing app breaks the UK’sown privacy laws (and is just plain broken). Gizmodo , 13 May 2020. URL https://gizmodo.com/the-uk-s-contact-tracing-app-breaks-the-uk-s-own-privac-1843439962 .Retrieved on 25.06.2020.Samuel Woodhams. COVID-19 digital rightstracker. Top10VPN , 10 June 2020. URL .Retrieved on 26.06.2020.M. Wooldridge. Reasoning about Rational Agents . MIT Press : Cambridge,Mass, 2000.M. Wooldridge. An Introduction to Multi Agent Systems . John Wiley & Sons,2002.Mark Zastrow. Coronavirus contact-tracing apps: can they slow the spreadof COVID-19? Nature (Technology Feature) , 19 May 2020. URL . Retrieved on25.06.2020.Stanley Zionts. A multiple criteria method for choosing among discrete al-ternatives.