A Deontic Logic Analysis of Autonomous Systems' Safety
AA Deontic Logic Analysis of Autonomous Systems’ Safety
Colin Shea-Blymyer [email protected] State University
Houssam Abbas [email protected] State University
ABSTRACT
We consider the pressing question of how to model, verify, andensure that autonomous systems meet certain obligations (like theobligation to respect traffic laws), and refrain from impermissiblebehavior (like recklessly changing lanes). Temporal logics are heav-ily used in autonomous system design; however, as we illustratehere, temporal (alethic) logics alone are inappropriate for reasoningabout obligations of autonomous systems. This paper proposesthe use of Dominance Act Utilitarianism (DAU), a deontic logic ofagency, to encode and reason about obligations of autonomous sys-tems. We use DAU to analyze Intel’s Responsibility-Sensitive Safety(RSS) proposal as a real-world case study. We demonstrate thatDAU can express well-posed RSS rules, formally derive undesirableconsequences of these rules, illustrate how DAU could help designsystems that have specific obligations, and how to model-checkDAU obligations.
CCS CONCEPTS • Computer systems organization → Robotic autonomy ; •
Computing methodologies → Modeling methodologies ; Modelverification and validation ; Knowledge representation and reason-ing.
ACM Reference Format:
Colin Shea-Blymyer and Houssam Abbas. 2020. A Deontic Logic Analysisof Autonomous Systems’ Safety. In
ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3365365.3382203
There is now a realistic prospect that Autonomous ground Vehi-cles (AVs) will be deployed on public roads in the next few years,with Waymo already charging customers for self-driving taxi inArizona [10]. While companies produce ‘event reports’ to regula-tors, there is a worrying sparsity of rigorous verification methods,and of external independent assessment, of the vehicles’ perfor-mance. The most pressing issue is that of verifying safety. So far,the vast majority of the work in formal verification of AVs used thetools of alethic temporal logic (like Linear [17] or Metric Temporal
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected].
HSCC ’20, April 22–24, 2020, Sydney, NSW, Australia © 2020 Association for Computing Machinery.ACM ISBN 978-1-4503-7018-9/20/04...$15.00https://doi.org/10.1145/3365365.3382203
Logic [13]) to express behavioral specifications of system models.Alethic logic is the logic of necessity and possibility : for example, if p is a predicate, p says that p is true in every accessible world - thatis, p is necessary. Possibility is then formalized as p ∶= ¬ ¬ p :saying that p is possible is the same as saying that it is not thecase that ¬ p is necessary. And so on. The best known instantiationof this in Verification is LTL [15], in which an accessible world isa moment in the (linear) future. Thus p formalizes ‘ p is true inevery future moment’, and p formalizes ‘ p is true in some futuremoment’.It is, however, equally important to think in terms of obligationsand permissions of the autonomous system: for instance, we maywish to say that ‘It is obligatory for the AV to not rear-end a car’,or ‘It is permissible to drive on the shoulder if the car ahead brakessuddenly’. Obligations, permissions and prohibitions are also per-vasive when discussing ethical questions: what should the AV dowhen faced with two equally unsavory but inevitable alternatives?Obligations and permissions are collectively called norms and state-ments about them are called normative statements. A prominentexample of a proposed normative system for Autonomous Vehicles(AVs) is Intel’s Responsibility-Sensitive Safety (RSS) [21], whichstates what the AV should and should not do to avoid accidents. It is essential to logically formalize proposed norms for autonomoussystems to enable automatic reasoning about their logical consistency,consequences, and automate system design . While all current workin AV verification and testing uses temporal logics [24], which aretypes of alethic logic, it has been understood for over 70 years that the logic of norms is different from that of necessity [16]: applyingalethic logic rules to normative statements leads to conclusions thatare intuitively paradoxical or undesirable. Consider the followingstatements:A. The car will eventually change lanes: this is a statement aboutpossibility. It says nothing about whether the car plays an activerole in the lane change (e.g., perhaps it will hit a slippery roadpatch).B. The car sees to it that it changes lanes: this is a statement aboutagency. It tells us that the car is an active agent in the lanechange, or is choosing to change lanes.C. The car can change lanes: this is a statement about ability. Thecar might be able to do something, but have no ‘choice’ or agencyin the matter.D. The car ought to change lanes: this is a statement about obliga-tion, a concept not captured in the first three statements.These are qualitatively different statements and there is no a prioriequivalence between any two of them. The logic we adopt shouldreflect this: its operators and inference rules should model theseaspects. Alethic logics like LTL cannot do so.We now give a simple but fundamental example, drawn from [16],illustrating this point. (In Section 2 we give an AV-specific example.) a r X i v : . [ c s . L O ] S e p SCC ’20, April 22–24, 2020, Sydney, NSW, Australia Colin Shea-Blymyer and Houssam Abbas
One might be tempted to formalize obligation using the necessityoperator : that is, formalize ‘The AV should stay in its lane’ by stay-in-lane . However, in alethic logic, p (cid:212)⇒ p : if p is nec-essarily true then it is true. If we interpret as obligation this readsas Obligatory p (cid:212)⇒ p : this is clearly non-sensical because agentssometimes violate their obligations so some obligatory things arenot true. This leads us to a major question in studying obligations:the automatic derivation of what an agent should do when someprimary obligations are violated. I.e. we wish to study statementsof the form Obligatory p ∧ ¬ p (cid:212)⇒ ... . This is simply impossiblein pure alethic logic, since p ∧ ¬ p (cid:212)⇒ q is trivially true for any p and q . Thus alethic logics (including common temporal logicslike LTL, MTL or CTL [6]) are not appropriate, on their own, forautomatic reasoning about norms. Deontic logic [7] has been developed specifically to reason aboutnormative statements, starting with von Wright [23]. It is widelyused in contract law, including software contracts. There are manyflavors of deontic logic [11]. In this paper, we adopt Dominance ActUtilitarianism (DAU) developed by Horty [12] because it explicitlymodels all four aspects above: necessity, agency, ability and obliga-tion. It includes a temporal logic as a component so we can describetemporal behaviors essential to system design, and it uses branchingtime, essential for modeling uncontrollable environments.To assess whether DAU is appropriate for reasoning about thenorms of autonomous systems, we formalize a subset of Intel’sResponsibility-Sensitive Safety, or RSS, in DAU. RSS proposes a setof norms or rules that, if followed by all cars in traffic, would lead tozero accidents [21]. The RSS proposal is expressed in the languageof continuous-time dynamical systems and ordinary differentialequations, but the rules to be followed are not formalized logically,so it is not possible to reason about them. This work integratesformal methods in AV design by complementing the dynamicalequations-based presentation of RSS in [21] with a deontic logicformalism. We formalize RSS in DAU, which achieves three pur-poses: first, it demonstrates the usefulness of DAU in a real usecase, namely, the analysis of a safety proposal by a major playerin autonomous driving technology. Second, it realizes a necessaryfirst step towards automated system design. Finally, it allows asystematic discovery of implicit assumptions, and undesirable con-sequences of any such proposals. A framework to do this is stillmissing from the literature. Our contributions in this paper are to:(1) formalize the normative system of RSS in DAU, to highlightthe subtle decisions that need to be made when developinga rigorous safety specification (Section 3.2);(2) partially infer the system structure using the DAU formal-ization (Section 3.3);(3) derive undesirable consequences of the RSS norms, point-ing the way to further necessary refinements of the norms(Section 3.4); and(4) develop a model-checking algorithm of DAU specificationsthat allows to establish whether a system has a given obliga-tion or not (Section 4).
There is a wide variety of deontic logics, tailored to different ends [7].Standard Deontic Logic has many well-known paradoxes [11], which have spurred the proposal of alternatives to remedy them.Some variations are commonly used to specify legal and softwarecontracts as in [18]. Various attempts were made to integrate de-ontic logic with temporal modalities (e.g., [9] and [19]). Decisionprocedures exist for some logics, like the checker in [14]. Gerdes etal. [8] have compared a deontological approach to AV design witha consequentialist approach by formalizing them as an optimal con-trol problem. Rizaldi et al. [20] formalize six traffic rules in HigherOrder Logic to be passed to an interactive theorem prover. As it isour goal to logically analyze normative safety rules and use themin system design, [8] and [20] present directions of investigationthat are orthogonal to ours. Alternating-time Temporal Logic (ATL)was proposed in [1] and extended in [22] to reason about groupsof agents. ATL seems to use sure-thing reasoning, like DAU (seeSection 2), but does not natively support a notion of obligation. TheRSS proposal itself [21] uses a point mass dynamical model to de-rive definitions of minimum safe distances between two cars. It alsoproposes motion planning policies to avoid accidents; e.g., if thecar ahead hits maximum brakes, then the following car should hitmaximum brakes within a delay τ , and so on. The RSS rules are notformalized in any logic in [21], nor are its logical consequences ex-amined. This paper leverages DAU’s formulation of agency [12, 3.3]to formalize well-posed RSS rules and analyze their implications.DAU further distinguishes itself through its distinction betweenwhat ought to be the case and what an agent ought to do [12, 3.3].A related formulation to DAU is found in [3]. This section summarizes the main aspects of DAU developed in [12],starting with classical branching time models. Let
Tree be a set of moments with an irreflexive, transitive ordering relation < suchthat for any three moments m , m , m in Tree , if m < m and m < m then either m < m or m < m . There is a unique rootmoment of the tree satisfying root < m ′ for all m ′ ≠ root . A history is a maximal linearly ordered set of moments from Tree : intuitively,it is a branch of the tree that extends infinitely. Given a moment m ∈ Tree , the set of histories that go through m is H m ∶= { h ⋃︀ m ∈ h } .See Fig. 1 We will frequently refer to moment/history pairs m ⇑ h ,where m ∈ Tree and h ∈ H m .Definition 1. [12, Def. 2.2] With AP a set of atomic propositions,a branching time model is a tuple ℳ = (
Tree , < , v ) where Tree is atree of moments with ordering < and v is a function that maps m ⇑ h pairs in ℳ to sets of atomic propositions from AP . A branching time model can be seen as the result of executinga non-deterministic automaton that models all agents in the sys-tem. While we will frequently speak of one agent’s obligations forsimplicity, the reader should keep in mind that a model ℳ canrepresent the possible evolutions of several agents.We will use CTL ∗ as the tense logic on branching time models -see [6] for details. CTL ∗ includes computational tree logic (CTL)and linear temporal logic (LTL), and has become widely used inmodel checking. CTL ∗ can produce sentences like ϕ ∶= ∃ X ( p ) ∧ The development of DAU in [12] uses a restricted temporal logic, but that is immaterialhere.
Deontic Logic Analysis of Autonomous Systems’ Safety HSCC ’20, April 22–24, 2020, Sydney, NSW, Australia m m m h h h K K K K K h h h h h h A A A A A B
10 109783 m m Figure 1: A utilitarian stit model for an agent α , showing moments m < m ′ with sets of histories H m = { h , . . . , h } and H m ′ = { h , . . . , h } . Each moment is marked with the actions available to α at that moment: Choice mα = { K , K } and Choice m ′ α ={ K , K , K } . Action K = { h , h } and K = { h } . Each history is marked with the formula(s) that it satisfies and with its value V alue ( h ) , e.g., h satisfies A and has value 3. m ⇑ h ⊧ (︀ α cstit ∶ A ⌋︀ since Choice mα ( h ) = K , and both h and h satisfy A . Onthe other hand, m ⇑ h ⊭ (︀ α cstit ∶ A ⌋︀ since Choice mα ( h ) = K = { h , h , h , h } and h does not satisfy A . Optimal mα = { K } so m ⇑ h ⊧ ⊙(︀ α cstit ∶ A ⌋︀ . Optimal m ′ α = { K , K } and so α has no obligations at m ′ since there is no formula ϕ s.t. ⋃︀ ϕ ⋃︀ m ′ ⊇ K ∪ K (SeeDef. 6). Finally, m ⇑ h ⊧ (︀ α dstit ∶ A ⌋︀ because K ⊂ ⋃︀ A ⋃︀ m and H m ≠ ⋃︀ A ⋃︀ m = { h , h , h , h , h } . ∀ ( p ) which can be interpreted as ‘there exists a path where p holds at the next state, and all paths will eventually always satisfy p ’.CTL ∗ allows us to formalize the temporal evolution of events alonga given history h (e.g., ϕ ), and quantify over histories passingthrough a moment m (e.g., ∀ ϕ meaning ‘for all histories, ϕ holds’).In this paper, to retain a uniform satisfaction relation like [12], wewill speak of formulas holding or not at an m ⇑ h pair: for a pair m ⇑ h in a model ℳ , we write ℳ , m ⇑ h ⊧ ϕ , where it is always the casethat h ∈ H m . There should be no confusion as a CTL ∗ path formulais evaluated along h and a state formula is evaluated at m .A formula ϕ is identified at moment m with the set of historieswhere it holds ⋃︀ ϕ ⋃︀ ℳ m ∶= { h ∈ H m ⋃︀ ℳ , m ⇑ h ⊧ ϕ } (1)Where there’s no risk of ambiguity, we drop ℳ from the notation,writing ⋃︀ ϕ ⋃︀ m , etc.The rest of this section is dedicated to the exposition of theproperly deontic aspects of DAU. Choice.
Let
Aдent be a set of agents, which represent, for exam-ple, the cars in traffic. Consider an agent α ∈ Aдent and a givenmodel ℳ . Then at every moment m , α is faced with a choice ofactions which we denote by Choice mα . Intuitively, an action causessome histories from H m to no longer be realizable, while othersstill are. Thus we can identify each action K ∈ Choice mα with theset of histories that are still realizable after taking the action, andwe may write K ⊆ H m . See moments and actions in Fig. 1. Choice mα must obey certain constraints which we relegate to Appendix A. Agency.
Agency is defined via the Chellas ‘sees to it’ operator cstit , named in honor of Brian Chellas who introduced an analogousoperator in [5]. (Saying ‘John sees to it that the window is open’means that John ensures the window is open). Intuitively, an agent sees to it that A by taking action K at m ⇑ h iff, whatever otherhistory h ′ could’ve resulted from the action, A is true at m ⇑ h ′ aswell. Thus, the non-determinism does not prevent α from achieving A . Let Choice mα ( h ) be the unique action that contains h . In Fig. 1 Choice mα ( h ) = K = { h , h , h , h } .Definition 2 (Chellas stit). [12, Def. 2.7] With agent α andformula ϕ ℳ , m ⇑ h ⊧ (︀ α cstit ∶ ϕ ⌋︀ iff Choice mα ( h ) ⊆ ⋃︀ ϕ ⋃︀ ℳ m See Fig. 1. We also define a deliberative stit operator, whichcaptures the notion that an agent can only truly be said to dosomething if it also has the choice of not doing it.Definition 3 (Deliberative stit). [12, Def. 2.8] With α and ϕ as before, ℳ , m ⇑ h ⊧ (︀ α dstit ∶ ϕ ⌋︀ iff Choice mα ( h ) ⊆ ⋃︀ ϕ ⋃︀ ℳ m and ⋃︀ ϕ ⋃︀ ℳ m ≠ H m Thus (︀ α dstit ∶ A ⌋︀ iff some histories don’t satisfy A but α ’s choiceensures A . See Fig. 1. The operators cstit and dstit are not inter-changeable and they fulfill complementary roles. This paper focuseson obligation statements of the following form.Definition 4 (Obligations). Let α be an agent. An obligation A is either a CTL ∗ formula, or a statement of the form (︀ α dstit ∶ ϕ ⌋︀ or ¬(︀ α dstit ∶ ϕ ⌋︀ where ϕ is a CTL ∗ formula. Like Eq. (1) for CTL ∗ formula, we identify an obligation A atmoment m with the set of histories where it holds ⋃︀ A ⋃︀ ℳ m ∶= { h ∈ H m ⋃︀ ℳ , m ⇑ h ⊧ A } (2)Obligations can be used in stit formulations, similarly to formulasin definitions 2 and 3: ℳ , m ⇑ h ⊧ (︀ α cstit ∶ A ⌋︀ iff Choice mα ( h ) ⊆ ⋃︀ A ⋃︀ ℳ m SCC ’20, April 22–24, 2020, Sydney, NSW, Australia Colin Shea-Blymyer and Houssam Abbas and ℳ , m ⇑ h ⊧ (︀ α dstit ∶ A ⌋︀ iff Choice mα ( h ) ⊆ ⋃︀ A ⋃︀ ℳ m and ⋃︀ A ⋃︀ ℳ m ≠ H m Optimal actions.
To speak of an agent’s obligations, we will needto speak of ‘optimal actions’, those actions that bring about an idealstate of affairs. We make the simplifying assumption that all agentsin the system collaborate to achieve a common goal. This is consis-tent with the RSS assumption that all agents are following the samerules to avoid collisions anywhere in traffic. Let
V alue ∶ H root → R be a value function that maps histories of ℳ to utility values fromthe real line R . This value represents the utility associated by allthe agents to this common history.Definition 5. A utilitarian stit frame is a tuple ( Tree , < , Aдent , Choice , V alue ) where Tree and < are as in branching time frames, Aдent is a set of agents,
Choice is a choice mapping (which is special-ized as
Choice mα for each agent and moment), and V alue is a valuefunction. A utilitarian stit model is a model based on a utilitarianstit frame. If
Choice mα is finite for every α ∈ Aдent and m , the modelis said to be finite-choice . All models in what follows are finite-choice utilitarian stit models.Given two sets of histories X and Y , we order them as X ≤ Y iff V alue ( h ) ≤ V alue ( h ′ ) ∀ h ∈ X , h ′ ∈ Y (3)Let State mα ∶= Choice mAдent ∖{ α } be the set of background states against which α ’s decisions are to be evaluated. These are otheragents’ independent actions. Given two actions K , K ′ in Choice mα , K ⪯ K ′ iff K ∩ S ≤ K ′ ∩ S for all S ∈ State mα . That is, K ′ dominates K iff it is preferable to it regardless of what the other agents do(known as sure-thing reasoning ). Strict inequalities are naturallydefined. Optimal actions are given by [12] Optimal mα ∶= { K ∈ Choice mα ⋃︀ ⇑∃ K ′ ∈ Choice mα . K ≺ K ′ } (4) Optimal mα is non-empty in finite-choice utilitarian stit models [12,Thm. 4.10]. Dominance Ought.
Intuitively we will want to say that at moment m , agent α ought to see to it that A iff A is a necessary condition ofall the histories considered ideal at moment m . This is formalizedin the following dominance Ought operator , which is pronounced“ α ought to see to it that A holds”.Definition 6 (Dominance ought). With α an agent and A anobligation in a model ℳ , ℳ , m ⇑ h ⊧ ⊙(︀ α cstit ∶ A ⌋︀ iff K ⊆ ⋃︀ A ⋃︀ ℳ m for all K ∈ Optimal mα See Fig. 1 for examples. If K ⊆ ⋃︀ A ⋃︀ m we say that K guarantees A . Note that the dominance Ought is only defined with the cstit operator and not dstit ; this is because it leads to a simpler logic. Thedominance ought satisfies a number of pleasing logical properties;we refer the reader to [12, Ch. 4]. Conditional obligation.
It is often necessary to say that an obli-gation is imposed only under certain conditions. Where A and B are obligations, the statement ℳ , m ⇑ h ⊧ ⊙((︀ α cstit ∶ A ⌋︀⇑ B ) (5)expresses that α ought to see to it that A , under the condition that B holds. Definition 7 (Conditional ought). With α an agent and A , B as obligations in a model ℳ , ℳ , m ⇑ h ⊧ ⊙((︀ α cstit ∶ A ⌋︀⇑ B ) iff K ⊆ ⋃︀ A ⋃︀ ℳ m for all K ∈ Optimal mα ⇑⋃︀ B ⋃︀ ℳ m where Optimal mα ⇑ B ( α ’s optimal actions under the condition B )is the set of actions available to α that are optimal if we ignore B -violating histories [12].We note that conditional obligation is not the same as B (cid:212)⇒⊙(︀ α cstit ∶ A ⌋︀ . Conditional obligation only considers B -guaranteeingdominating histories, while this latter formula still considers alloptimal actions, not only those that guarantee the truth of B . Syntax.
We now summarize the syntax of DAU statements. Obli-gations are generated as follows. A ∶∶= ϕ ⋃︀ (︀ αdstit ∶ A ⌋︀ ⋃︀ ¬ A where ϕ ∈ CTL ∗ , and the semantics of (︀ αdstit ∶ A ⌋︀ were given inDef. 3. Ought statements are in one of two forms: ⊙(︀ α cstit ∶ A ⌋︀ or ⊙ ((︀ α cstit ∶ A ⌋︀⇑ B ) where α is an agent and A and B are obligations. The semanticswere given in Def. 6. We now offer an AV-specific example of the advantage that a DAUformalization offers over pure temporal logic. Specifically, DAUallows deriving obligations over time by construction and in auniform manner; attempts to do so using pure temporal logic areunsatisfactory. Consider the stit model in Fig. 2, which models thesituation on the left: agent α could either stay in its lane behind theslower β ( K ), or pass β by going into the opposite lane ( K ) andrisk a head-on collision. Every history in K is deemed preferableto every history in K because K eliminates the risk of collision, sowe assign history values accordingly, as shown. If the agent does K , then it needs to get back into its lane. Thus at m ′ , every historyin K is preferable to every history in K , and this is reflected inthe values. Naturally, the histories in K at m satisfy ψ ∶= ∀(¬ p ) ( α does not pass, i.e., does not change lanes), those in K at m ′ satisfy π = ∀ Collision (since α remains in the opposite lanein this case), and those in K at m ′ satisfy φ t ∶= (︀ ∶ t ⌋︀ p , t = , , α changes lanes in at most t time steps ( (︀ n ∶ m ⌋︀ ϕ ∶= X n ϕ ∨ X n + ϕ . . . ∨ X m ϕ and X t is X repeated t times). Moreover,suppose K histories at m satisfy some arbitrary formula χ . Thefollowing obligations are then automatically derived from the stitmodel :At m , ⊙(︀ α cstit ∶ ψ ∧ χ ⌋︀ (6)At m ′ , ⊙(︀ α cstit ∶ φ ⌋︀ ( since φ is true if φ or φ are ) (7)Thus it emerges that at m , α ought to not change lanes. Also at m , α ought to see to it that χ - which may have nothing to do with howthe values were assigned to the histories. E.g., χ might constrain This is not a well-formed DAU expression, but we can extend the logic to give thisexpression its natural definition as ¬ B ∨ ⊙(︀ α cstit ∶ A ⌋︀ . In DAU, ⊙(︀ α cstit ∶ ϕ ⌋︀ ∧ ⊙(︀ α cstit ∶ ψ ⌋︀ is equivalent to ⊙ (︀ α cstit ∶ ϕ ∧ ψ ⌋︀ Deontic Logic Analysis of Autonomous Systems’ Safety HSCC ’20, April 22–24, 2020, Sydney, NSW, Australia K K h h h h h ⇡ m K K ' ' h ↵ m ' h Figure 2:
Deriving obligations for α from the stit model. K : stay in lane, K : change lanes. the motor’s energy consumption; it is nonetheless an obligationbecause it’s a necessary condition for achieving an optimal history.If the agent violates (6) at m by doing K , then automatically themodel yields that its obligation at m ′ is (7). As explained in theIntroduction, such generation of new obligations is not possiblein pure temporal logic, and would have to be added somewhatawkwardly to the atomic propositions or imposed from outside thelogic. For example, the agent might try to satisfy something impliedby ψ ∧ φ , like ∃( (︀ ∶ ⌋︀ p ) (i.e. there exists a path that satisfies p within the next two states). However, at m this is too permissive,since we really do prefer not changing lanes at all. And at m ′ it istoo restrictive, since φ is a perfectly legitimate way of meeting α ’s obligations then. Another method may be to specify behaviorthrough reactive implications, e.g. "oncoming-traffic (cid:212)⇒ changelanes", but this sort of explicit rule must be built in by a humandesigner. The conclusion is that there is a need to use a logic thatcaptures preferences and derives obligations from them, as wellas what agents are able and unable to do; a logic of agency and obligation. Responsibility-Sensitive Safety, or RSS, is a proposal put forth byIntel’s Mobileye division [21]. It proposes rules or requirementsthat, if followed by all cars in traffic, would lead to zero accidents.Our objective here is to formalize some of the RSS rules in thelanguage of Dominance Act Utilitarianism (DAU), and study theirlogical consequences. Three important points must be made:(A) The formalization does not depend on the dynamical equationsthat govern the cars because we wish our conclusions to beindependent of these lower-level concerns. This is consistentwith the standard AV control architecture where a logical plan-ner decides what to do next (‘change lanes’ or ‘turn right’) anda lower-level motion planner executes these decisions. Ourlogical analysis concerns the logical planner.(B) We are not trying to formalize general traffic laws or drivingscenarios, which is outside the scope of this paper. We are onlyformalizing the RSS rules. (C) Every formalization, in any logic, can always be refined. Weare not aiming for the most detailed formalization; we aim fora useful formalization.We have three objectives in doing so: demonstrating the usefulnessof DAU in a real use case; highlighting the ambiguities implicit insuch proposals, which would go unnoticed without formalization;and automating the checking of logical consistency and derivingof conclusions. We first present the RSS rules in natural language(Section 3.1), then their formalization (Section 3.2), and finally weanalyze the rules’ logical consequences.
The rules for Responsibility-Sensitive Safety are [21]:RSS1. Do not hit someone from behind.RSS2. Do not cut-in (to a neighboring lane) recklessly.RSS3. Right-of-way is given, not taken.RSS4. Be careful of areas with limited visibility.RSS5. If you can avoid an accident without causing another one,you must do it.RSS6. To change lanes, you should not wait forever for a perfectgap: i.e., you should not wait for a gap large enough to getinto even when the other car, already in the lane, maintainsits current motion.RSS6 is derived directly from the following in [21, Section 3]:“the interpretation [of the duty-of-care law] should lead to [...] anagile driving policy rather than an overly-defensive driving whichinevitably would confuse other human drivers and will block traffic[...]. As an example of a valid, but not useful, interpretation is toassume that in order to be “careful” our actions should not affectother road users. Meaning, if we want to change lane we shouldfind a gap large enough such that if other road users continuetheir own motion uninterrupted we could still squeeze-in withouta collision. Clearly, for most societies this interpretation is over-cautious and will lead the AV to block traffic and be non-useful.”
Note that, consistently with points (A)-(C) above, this is stated withoutany reference to dynamics or specific scenarios.
The RSS authors areconcerned that overlay cautious driving might lead to unnatural
SCC ’20, April 22–24, 2020, Sydney, NSW, Australia Colin Shea-Blymyer and Houssam Abbas traffic, so RSS aims to allow cars to move a bit assertively, anddefines correct reactions to that.We will not study RSS4 and 5 as they are currently too vague forformalization.
Formalizing RSS1 . Let ϕ be a formula denoting ‘Hit someone frombehind’. A plausible formalization of RSS1 is then RSS . ⊙ (︀ α cstit ∶ ¬ ϕ ⌋︀ That is, α ought to see to it that it does not hit anyone from behind.However, suppose that α finds itself, through no fault of its own,in a situation where a collision is unavoidable at time m , that is, H m = ⋃︀ ϕ ⋃︀ ℳ m . Then we can show that RSS automatically, as a matter oflogic , remove the obligation when a collision becomes unavoidable.This can be done using dstit of Def. 3 as follows:
RSS r . ⊙ (︀ α cstit ∶ ¬(︀ α dstit ∶ ϕ ⌋︀⌋︀ This says that α should see to it that it does not deliberately ensurean accident ϕ . This form of obligation is called refraining : in thiscase, α refrains from hitting anyone from behind. RSS
RSS r are not logically equivalent . If ⋃︀ ϕ ⋃︀ ℳ m = H m , then (︀ α dstit ∶ ϕ ⌋︀ is nec-essarily false, and RSS r is trivially satisfied since ⊙(︀ α cstit ∶ ⊺⌋︀ isa theorem of DAU. Thus RSS r does not impose unrealistic obliga-tions on the agent. Of course, a test engineer should then examinewhy the inevitable situation arose in the first place - but that is aseparate debugging effort. The control engineer can now focus ondesigning a controller that meets the more realistic RSS r . Formalizing RSS2 . Define two CTL ∗ formulas, ψ ∶ a non-recklesscut-in, and ψ r : a reckless cut-in. Then RSS2 is formalizable as RSS . ⊙ (︀ α cstit ∶ ∀ ( ψ ∨ ψ r (cid:212)⇒ ¬ ψ r )⌋︀ . That is, α should see to it that always, if a cut-in happens, thenit is a non-reckless cut-in. Formalizing RSS3 . Formalizing this rule requires some care. First,note that RSS3 should probably be amended to say that ‘Right-of-way is given, not taken, and some car is given the right-of-way ’- otherwise, traffic comes to a standstill. We will first focus onformalizing the prohibition (nobody should take the r-o-w), thenwe will formalize the positive obligation (somebody must be givenit).Let
Aдent = { α , β , γ , . . . } be a finite set of agents. Define theatomic propositions GROW αβ : β gives right-of-way to α and p α : α proceeds/drives through the conflict region. Then T ROW α ∶= p α ∧ ¬( GROW αβ ∧ GROW αγ ∧ . . . ) formalizes taking the r-o-w: α proceeds without being given the right of way by everybody. Wecould now express the prohibition in RSS3: every α ought to see toit that it does not take the r-o-w: RSS prohib . ⋀ α ∈ Aдent ⊙(︀ α cstit ∶ ¬
T ROW α ⌋︀ (8) The difficulty with this formulation is that it could lead to α beingobliged to force everybody else to give it the r-o-w - something overwhich, a priori, it has no control. To see this, we need the following,whose proof is omitted due to lack of space.Theorem 1. Given obligations A and B , ⊙(︀ α cstit ∶ A ∨ B ⌋︀ ∧(∀¬ A ) (cid:212)⇒ ⊙(︀ α cstit ∶ B ⌋︀ In other words, if α has an obligation to fulfill A or B at m ⇑ h ,but every available history violates A ( ∀¬ A ), then its obligationis effectively to fulfill B . Applied to Eq. (8) with A = ¬ p α and B = ∧ β ≠ α GROW αβ , Thm. 1 says that if α is in a situation where ithas no choice but to proceed (e.g. as a result of slippage on a wetroad, say), then its obligation is to see to it that everybody else givesit the right-of-way, which is unreasonable.To remedy this, we first formalize the positive obligation: some-body must be given the right-of-way. This seems to be a groupobligation : the group must give r-o-w to one of its members. Groupobligations are formally defined in [12, Ch. 6]. Therefore, we definean atomic proposition д α : r-o-w is Granted to α . Then we formalize RSS pos . ⊙ (︀ Aдent cstit ∶ ∃ ∨ α ∈ Aдent д α ⌋︀ (9)This says the group Aдent has an obligation to give r-o-w to some-one, and the only choice is in who gets it. We now come back toformalizing the prohibition:
RSS prohib . ⋀ α ∈ Aдent ⊙(︀ α cstit ∶ (¬ д α (cid:212)⇒ ¬ p α )⌋︀ (10)Finally, we formalize RSS
RSS prohib ∧ RSS pos . Formalizing RSS6 . This rule says that if the car wants to changelanes, it shouldn’t wait for the perfect gap (otherwise, traffic isstalled). First, let’s formalize ‘waiting for the perfect gap’, that is,waiting until the other car, already in the lane, gives the AV theright-of-way (e.g., by slowing down). Let the atomic proposition w α mean ‘ α wants to change lanes’ and recall that p α means ‘ α proceedsthrough the conflict region’ while д α means ‘ α is Granted theright-of-way’. For conciseness, let’s introduce the bounded Releaseoperator ℛ N , which informally says that over the next N steps,either ψ does not hold at all, or it does and ϕ holds continuouslyuntil ψ holds. ψ ℛ ϕ = ψ ∨ ( ϕ ∧ Xψ ) ∨ ( ϕ ∧ Xϕ ∧ X ψ ) ∨ . . .. . . ∨ ( ϕ ∧ Xϕ ∧ . . . ∧ X N − ϕ ∧ X N ψ ) ∨ ( ϕ ∧ Xϕ ∧ . . . X N ϕ ) Then ¬ p α ℛ д α says that α waits for the perfect gap up to N timesteps (but we don’t know what happens after this). (︀ α dstit ∶ ¬ p α ℛ д α ⌋︀ formalizes the agent deliberately seeing to it that it waits to be giventhe right-of-way, when it doesn’t have to. Finally, RSS . ⊙ ((︀ α cstit ∶ ¬(︀ α dstit ∶ ¬ p α 𝒰 N д α ⌋︀⌋︀⇑ w α ) (11)formalizes that α ought to refrain from seeing to it that it waits forthe right-of-way given that it wants to change lanes. This obligationdoes not delay the lane change - in particular, it does not require thecar to wait for the perfect gap. It also does not rush α : it can wait ifit wishes to. We emphasize that RSS assertive driving requires thatan AV sometimes force its way, as expressed in (11). Deontic Logic Analysis of Autonomous Systems’ Safety HSCC ’20, April 22–24, 2020, Sydney, NSW, Australia K K K K m i : p ^ ¬ gm : ¬ g ˜ h h h h Figure 3: The optimal action at m , K , necessarily containshistory ˜ h in which comes a moment i ′′ s.t. i ′′ ⇑ ˜ h ⊧ p ∧ ¬ д . Thecontroller must choose an action, prior to i ′′ , that does notcontain ˜ h . Since the controller always chooses optimal ac-tions, the V alue function must favor K , as shown. In DAU, obligations are automatically derived from the stit modelvia Def. 2. Given an obligation that we want the system to have, howshould we structure the stit model so that it has that obligation?This is similar to synthesis-from-specifications, an active researcharea in programming and in Cyber-Physical Systems. This sectiongives an example where it is possible to manually partially inferthe stit model structure from the RSS obligations.Consider again the
RSS prohib and RSS
A stit model has both obligations
RSS prohib and RSS at m if for every optimal action K ∈ Optimal mα , it holds that ⋃︀ K ⋃︀ ≥ , and there exist a history ˜ h ∈ K and a moment m ′ > m in ˜ h s.t. ⋃︀ Choice m ′ α ⋃︀ ≥ , m ⇑ ˜ h ⇑⊧ CTL ∗ (¬ д α (cid:212)⇒ ¬ p α ) and ˜ h is not in anyoptimal action at m ′ . The proof is omitted due to lack of space. The conclusion ofthe Proposition, illustrated in Fig. 3, is counter-intuitive: it necessi-tates the existence of a history ˜ h along which one of the formulas, (¬ д α (cid:212)⇒ ¬ p α ) , is violated . But since the inferred structureplaces ˜ h in a non-optimal action (via V alue ), this doesn’t lead to an obligation violation . One of the main tenets of RSS is that an AV is only responsible foravoiding potential accidents between itself and other cars (so-called‘star calculations’); interactions between 2 other cars are not itsconcern [21, Remarks 1 and 8]. Yet everyday driving experiencemakes clear that our actions can be faulted for at least facilitating anaccident: e.g., by repeated braking, I may cause the car behind meto do the same, leading the car behind it to rear-end it. Or I mightmake a sudden lane change over two lanes, causing the car in the lane next to me to over-react when I speed past it, and collide withsomeone else. We now show how this intuition is automaticallycaptured by the DAU logic, and that RSS star-calculations lead toundesirable behavior of the AV.Let ϕ ∈ CTL ∗ denote a formula expressing “Accident between twoother cars”, and assume the accident is such that α can facilitate it asin the above 2 examples. Then (︀ α dstit ∶ ϕ ⌋︀ says that α (deliberately)sees to it that the accident happens even though it could avoiddoing so; given what we assumed about this accident, this means α facilitates the accident. Then (︀ α dstit ∶ ¬(︀ α dstit ∶ ϕ ⌋︀⌋︀ expresses that α sees to it that it does not facilitate the accident: this is a form ofrefraining. Finally, (︀ α dstit ∶ ¬(︀ α dstit ∶ ¬(︀ α dstit ∶ ϕ ⌋︀⌋︀⌋︀ says that α refrains from refraining, that is, α does not refrain from facilitatingthe accident (even though it could). The RSS position is that it is OKfor α to refrain from refraining [21, Remarks 1 and 8], as formalizedhere.However, refraining from refraining is the same as doing. For-mally [12, 2.3.3.] (︀ α dstit ∶ ¬(︀ α dstit ∶ ¬(︀ α dstit ∶ ϕ ⌋︀⌋︀⌋︀ ≡ (︀ α dstit ∶ ϕ ⌋︀ And we argue that this matches our intuition: to not refrain fromfacilitating an accident even though one could is the same as facili-tating it. In other words, under this formalization, the RSS positionis tantamount to allowing AVs to facilitate accidents between others- clearly, an undesirable conclusion. This aspect of RSS, therefore,needs refinement to take into account longer-range interactionsbetween traffic participants.
The system designer’s job is to design a system that has the right obligations; it is then the control engineer’s job to design a con-troller that makes the system meet these obligations. In DAU, obli-gations are automatically derived from stit models/trees, but design-ers usually model an agent as an automaton or a similar structure.The question then naturally poses itself: given an agent model,how do we verify whether it has a given obligation? Answeringthis question is a crucial design step: there is no point designingcontrollers that meet the wrong obligations. This can be cast as amodel-checking question, which this section tackles. All proofs arein the appendices.
Definition 8 (Stit automaton).
Let AP be a finite set of atomicpropositions. A stit automaton T is a tuple T = ( Q , q , 𝒦 , F , ∆ , L , w , λ ) ,where Q is a finite set of states, q is the initial state, 𝒦 is a finite setof actions ( 𝒦 ⊂ H root ) ), F ⊂ Q is a set of final states, ∆ ⊂ Q × 𝒦 × Q is a finite transition relation such that if ( q , K , q ′ ) and ( q , K ′ , q ′ ) arein ∆ then K = K ′ , L ∶ Q → AP is a labeling function, w ∶ ∆ → R is aweight function, and λ ∶ R ω → R is an accumulation function. Denote by ∆ ( q ) ⊂ ∆ the set of outgoing transitions from q ( ∆ ( q ) = {( q , K , q ′ ) ∈ ∆ } ), by Post ( q , K ) = { q ′ ⋃︀ ( q , K , q ′ ) ∈ ∆ ( q )} the successors of q under K , and by Post ( q ) = ∪ K ∶( q , K , q ′ )∈ ∆ ( q ) Post ( q , K ) all the successors of q . Finally, we denote by T . q theinitial state of T when there’s a need to clarify the automaton.Note that T is a type of non-deterministic weighted automaton. Its SCC ’20, April 22–24, 2020, Sydney, NSW, Australia Colin Shea-Blymyer and Houssam Abbas q q q q q K K K K K K K T root = 0 K K K K K K m K ( q , m K ( q , m K ( q , m K ( q , m K ( q , m K ( q , m K ( q , ... ... ... ... ... M T q q q q q K K K K K K T K K K K K K T q q q q q K K K K K K K q ren q ren q ren q ren Figure 4: Left: a stit model generated by executing the stit automaton T (transition weights not shown). Center and right:Automata T n and T ′ n used in Algorithm 1. T only has K as first action, and T ′ is obtained by re-naming states of T and‘addingâĂŹ a copy of T to it. Executions of T ′ are simply the execution of T that start with K . unweighted counterpart T u is a classical transition system; thusfor a CTL ∗ formula ϕ , we could model-check whether T u ⊧ ϕ . A setof agents is modeled by the product of all individual stit automata,which is itself a stit automaton. (When taking the product, wemust define how weights are combined and how to construct theproduct’s accumulation function, which are application-specificconsiderations.) Therefore the rest of this section applies to stitautomata, whether they model one or multiple agents. We willcontinue to refer to one agent α for simplicity. From automata to stit models.
Let S ω denote the set of infi-nite sequences ( a i ) i ∈ N with a i ∈ S . An execution of a stit au-tomaton T is a sequence π ∈ ∆ ω of transitions of the form π =( q , K , q )( q , K , q ) . . . . The corresponding sequence of actions K , K , . . . ∈ 𝒦 ω is called a strategy . Because of non-determinism, astrategy can produce multiple executions. An execution of the au-tomaton generates a stit model in the natural way: starting in state q the automaton takes an infinite sequence of actions from 𝒦 , thusnon-deterministically traversing an infinite number of transitions e from ∆ . These sequences of transitions form the histories in thecorresponding stit model, with every transition e adding a momentto the histories. The value(s) of those histories are obtained by ac-cumulating w ( e ) along the traversed transitions using function λ .See Fig. 4 for an example. The formal construction and proof are inAppendix B.Theorem 2. The structure ℳ T obtained by executing a stit au-tomaton T is a utilitarian stit model with finite Choice mα for everyagent α and moment m . The cstit model-checking problem is: Given a stit automaton T that models an agent α and an obligation A , determine whether ℳ T , root ⇑ h ⊧ ⊙(︀ α cstit ∶ A ⌋︀ for some h ∈ H root . The case of condi-tional oughts ⊙((︀ α cstit ∶ A ⌋︀⇑ B ) is similarly handled and we omitthe details. Given the structure of an obligation given in Def. 4, the model-checking problem can be broken down into two parts: what is the setof optimal actions at root , Optimal rootα ? And out of these optimalactions, which ones guarantee the truth of A ? (Recall Eqs. (3)-(4):action optimality is determined solely by the V alue function, andnot by which obligations its histories satisfy). If all optimal actionsguarantee A , then by Def. 2, ℳ T has obligation A at root ⇑ h . Thealgorithm is presented in Algorithm 1 page 9. In it, ⊧ CTL ∗ denotesthe classical CTL ∗ satisfaction relation.Theorem 3. Algorithm 1 returns True iff ℳ , root ⇑ h ⊧ ⊙(︀ α cstit ∶ A ⌋︀ . It has complexity O ( m (⋃︀ T ⋃︀+ c λ +⋃︀ T ⋃︀⋅ ⋃︀ ϕ ⋃︀ )) , where c λ is the cost ofcomputing the minimum and maximum values of a strategy executedon automaton T and ⋃︀ T ⋃︀ is the number of states and transitions in T . The proof is in Appendix C. This algorithm can be amendedto accept a conditional obligation ⊙((︀ α cstit ∶ A ⌋︀⇑ B ) by acceptingonly those actions K in Optimal rootα that guarantee A and B . Thecomputation of the minimum and maximum values of a strategy’sexecution line 8 clearly depends on the function λ used for accu-mulating weights along the execution: e.g., if λ is addition and allthe weights are positive, then all executions have infinite value,and every future is ideal, which is a comforting thought but oflittle interest in modeling the real world. This question is related tobut distinct from temporal logic accumulation [2] and quantitativelanguages [4]. We give now one example of a λ that can model real-world phenomena, and lead to finite values of u n . Take λ = min.For instance, if w (( q , K , q ′ )) is the time-to-collision resulting fromaction K then V alue ( h ) is the shortest time-to-collision encoun-tered along the history, and an optimal history is one with thehighest minimum time-to-collision. It’s a simple matter to provethat u n is the maximum weight of any reachable transition from q ,which can be computed in a finite number of steps. (Unfortunately,different λ s will, in general, require different customized analyzes.) Deontic Logic Analysis of Autonomous Systems’ Safety HSCC ’20, April 22–24, 2020, Sydney, NSW, Australia
Data:
A stit automaton T = ( Q , q , 𝒦 , F , ∆ , L , w , λ ) , anobligation A Result: ℳ T , root ⇑ h ⊧ ⊙(︀ α cstit ∶ A ⌋︀ Set root = Set
Choice rootα = { K ∈ 𝒦 ⋃︀ ( q , K , q ′ ) ∈ ∆ for some q ′ } ={ K , . . . , K m } // First step: find optimal actions at root for ≤ n ≤ m do /* Construct automaton T ′ n s.t. every executionof T ′ n is an execution of T starting withaction K n . See Fig. 4. */ Create automaton T n by deleting all transitions ( q , K , q ′ ) with K ≠ K n Create a copy T ren n of T n Create the automaton T ′ n as a union of T ren n and T , withevery transition ( q , K , T ren n . q ) in T ren n replaced by atransition ( q , K , T . q ) Compute the max value, u n , and min value, ℓ n , of any T ′ n strategy starting at q end /* An interval (︀ ℓ n , u n ⌋︀ is un-dominated if there isno other interval (︀ ℓ ′ n , u ′ n ⌋︀ , computed in the abovefor-loop, s.t. ℓ ′ n > u n */ Find all un-dominated intervals (︀ ℓ n , u n ⌋︀ Set
Optimal rootα = { K n ∈ Choice rootα ⋃︀ (︀ ℓ n , u n ⌋︀ is un-dominated } /* Second step: decide whether all actions K in Optimal rootα guarantee A , i.e., K ⊆ ⋃︀ A ⋃︀ root . */ for K n ∈ Optimal rootα do if A is a CTL ∗ formula then /* Does every execution of T starting with K n satisfy A ? */ Use CTL ∗ model-checking to check whether T ′ n ⊧ CTL ∗ ∀ A if T ′ n ⇑⊧ CTL ∗ ∀ A // Optimal action K n does notguarantee A then return False end else if A = (︀ α dstit ∶ ϕ ⌋︀ with ϕ ∈ CTL ∗ then // This is true iff H root = ⋃︀ ϕ ⋃︀ root Model-check whether T ⊧ CTL ∗ ∀ ϕ /* This is true iff K n guarantees ϕ , is notequiv. to line 26 */ Model-check whether T ′ n ⊧ CTL ∗ ∀ ϕ if T ⊧ CTL ∗ ∀ ϕ or T ′ n ⇑⊧ CTL ∗ ∀ ϕ then return False end else /* Last case: A = ¬(︀ α dstit ∶ ϕ ⌋︀ with ϕ ∈ CTL ∗ .Similar to previous case on line 24 withobvious modifications */ end end Return True
Algorithm 1:
Model checking DAU.
We have demonstrated the use of Dominance Act Utilitarianism informalizing safety norms for autonomous vehicles. Our objectivewas to assess the feasibility and utility of doing so: we expressedsafety norms from RSS in DAU; found undesirable consequences inthese norms; and showed that system designers can automaticallyderive a formalized system’s obligations and objectives.It is desirable next to enrich the interaction between deontic andtemporal modalities, e.g. to express things like ‘In the next planningcycle the AV must see to it that it changes lanes’. This then allowsreasoning about obligation propagation through time [3]. It will beequally important to study obligation inheritance between groupsand individuals: e.g., if it is the group’s obligation to give the right-of-way, what does that imply for individual obligations? Giventhat deontic logics were developed for ethical analysis, this workalso opens the way to formally considering ethical implications ofsystem design. In our experience even framing technical specifica-tions as obligations can make explicit an implicit norm. Addressingethical considerations is necessary to build trust in autonomoussystems, and this work suggests it may be possible to formalize a asystem’s ethical constraints, and analyze the moral implications ofits design. These and other considerations will ultimately determinethe suitability of DAU for AV design and verification.
A MORE ELEMENTS OF DOMINANCE ACTUTILITARIANISM
Agent choice . The choice mapping
Choice mα in a general deonticstit model obeys ● The actions in
Choice mα partition the set H m : K ∩ K ′ = ∅ for every K , K ′ and ∪ K ∈ Choice mα K = H m . There is no loss ofgenerality in this constraint, it is a formality that allows usto maintain the useful tree structure. ● Independence of agents: given any group of agents Γ ⊆ Aдent , ∩ α ∈ Γ Choice mα ≠ ∅ . That is, the actions of one agentdo not prevent the choice of action available to any otheragent at the same moment m . ● No choice between undivided histories: If two histories arestill undivided at m (that is, they share a moment m ′ > m )then they belong to the same action K in Choice mα . B CONSTRUCTION OF ℳ T AND PROOF OFTHM. 2
We give the formal construction of stit model ℳ T from stit au-tomaton T , then prove Thm. 2. The construction is as follows (seeFig. 4). ● Initialization : set iteration i = q = q , root = S = {∐︀ q , root ̃︀} , Tree = { root } . ● Expansion : Set S ′ = ∅ . For every couple ∐︀ q , m ̃︀ ∈ S ,Exp1) set Choice mα = { K ∶ ( q , K , q ′ ) ∈ ∆ ( q ) for some q ′ } : the agenthas a choice of actions at m from the actions that label thetransitions out of q .Exp2) For every K ∈ Choice mα , and every q ′ ∈ Post ( q , K ) , add a newmoment m K ( q ′ , i ) to Tree with m K ( q ′ , i ) > m , and such thatthe history ending with the moments ( m , m K ( q ′ , i )) belongsto action K . Also, add the couple ∐︀ q ′ , m K ( q ′ , i )̃︀ to S ′ . SCC ’20, April 22–24, 2020, Sydney, NSW, Australia Colin Shea-Blymyer and Houssam Abbas
Exp3) Set the label map v ( m ⇑ h ) = L ( q ) for every history h passingthrough m . ● Update : Set S = S ′ . For the next iteration, set i = i +
1. Goto
Expansion . ● Valuation : For every history h constructed in the Expansion loop, its value is computed as
V alue ( h ) = λ ( w ( e i ) i ∈ N ) where e i ’sare the transitions taken while constructing h . ( λ must be such thatinfinite accumulation yields a finite value).Thm. 2. We first verify that ℳ T is a branching time model(Def. 1). The ordering between moments is irreflexive and tran-sitive by construction.Take 3 moments m , m and m s.t. m < m and m < m .Moments are only added in Exp2 so m = m K ( q ′ , i ) for some q ′ , i ,and by construction there is a unique moment m K ′ ( q , i − ) at level i − m K ( q ′ , i ) > m K ′ ( q , i − ) . By a simple inductive argument,there is a unique moment m K j ( q j , j ) at level j s.t. m K ( q ′ , i ) > m K j ( q j , j ) for every j < i . Thus the sequence of moments that aresmaller than m ( q ′ , i ) forms a chain (a linear order) to which mustbelong both m and m , so either m < m or m < m .The tree is rooted at 0 as can be easily established by inductionon i .The function v in Exp3 plays the role of the stit model’s labelmap.We now show that Choice mα satisfies the constraints of Appen-dix A on choices: ● The actions in
Choice mα partition H m : indeed, take a history start-ing at m = m K ( q , i − ) . It is expanded in Exp2 only, by m K ′ ( q ′ , i ) say, and the expanded history ∐︀ m , m K ′ ( q ′ , i )̃︀ is assigned to onlyone action. Thus the histories ∐︀ m , m K ′ ( q ′ , i )̃︀ , q ′ ∈ Post ( q , K ′ ) , K ′ ∈ Choice mα are partitioned among the actions at m . By definition ofthe automaton transition relation, two different actions must leadto two different states q ′ and q ′′ so the newly created moments m K ′ ( q ′ , i + ) and m K ′′ ( q ′′ , i + ) at the next iteration i +
1, andwhich expand these histories, are different. Therefore, two historiesthat were in different actions at m will never share a moment after m . Thus the actions at m partition H m . ● Independence of agents: this is automatically guaranteed by usingan automaton that models the product of all stit automata. ● No choice between undivided histories: as established in the firstbullet of the proof, histories that are in different actions at m willnever share a moment after m . Therefore, two histories that sharea moment at m ′ > m must be in the same action at m .Finally, Choice mα is finite for each moment since, as can be seenin Exp1, Choice mα is (isomorphic to) a subset of ∆ and the latter isfinite. QED. □ C PROOF OF THM. 3
Recall that by executing a stit automaton, a stit model is created(Appendix B).Lemma 1.
The histories generated by T ′ n are exactly the historiesof T whose first action is K n , modulo a re-naming of the states. Proof. Recall that T ′ n has two components, namely a copy T ren n of T n and a copy of T . See Fig. 4. T n is obtained by removing transi-tions from T , thus every history generated by T n is a valid T -history. Every history generated by T n starts with K n by construction. Soevery history h of T ′ n starts with K n , because it starts in T ren n .Case 1: h never leaves T ren n . T ren n is nothing but a renaming of T n and we’ve already established that a history of T n is a history of T , so this case is done.Case 2: h leaves T ren n . That is, a transition takes the executioninto the T copy. Up to the transition, h is a history of T as establishedin Case 1. The transition itself, say ( q , K , T . q ) , is a valid transitionof T (modulo re-naming) since it was created by replacing a T transition of the form ( T . q , T . K , T . q ) . Once in the T copy, thehistory of course continues to be a valid history of T . QED. □ Lemma 2.
The set computed at line 13 is indeed
Optimal rootα . Proof. Every history of T ′ n starts with K n so ℓ n = min { V alue ( h ) ⋃︀ h ∈ K n } and u n = max { V alue ( h ) ⋃︀ h ∈ K n } . By definition of actiondominance, K n ⪯ K ′ n in T iff u n ≤ ℓ ′ n . So (︀ ℓ n , u n ⌋︀ is un-dominatediff its action K n is un-dominated and must be optimal. QED. □ Lemma 3.
If line 21 is executed, then K n ⊈ ⋃︀ A ⋃︀ root . Proof. If T ′ n ⇑⊧ ∀ A this means some execution ˜ h of T ′ n violates A . By Lemma 1 ˜ h is also a history of T starting with the optimalaction K n , so that K n ⊈ ⋃︀ A ⋃︀ root . QED. □ Lemma 4.
If line 30 is executed, then ℳ , root ⇑ h ⇑⊧ ⊙(︀ α cstit ∶ ϕ ⌋︀ Proof. T ⊧ CTL ∗ ∀ ϕ iff every history of T satisfies ϕ and so H root = ⋃︀ ϕ ⋃︀ root ; in this case, by definition of dstit , root ⇑ h ⇑⊧ (︀ α dstit ∶ ϕ ⌋︀ . T ′ n ⇑⊧ CTL ∗ ∀ ϕ iff there exists a history ˜ h of T ′ n which violates ϕ .Again this is also a history of T which belongs to the optimal K n so that K n ⊈ ⋃︀ ϕ ⋃︀ root . QED. □ Thm. 3. We need to establish that the algorithm returns Trueiff K ⊆ ⋃︀ A ⋃︀ root for every optimal K . The set of optimal actions iscomputed at line 13 by Lemma 2. The for-loop at line 15 visits eachoptimal action in turn. Line 37 is executed iff none of the ‘returnFalse’ statements preceding it are executed; namely, iff K ⊆ ⋃︀ A ⋃︀ root by Lemma 3 in Case A is CTL ∗ , or iff H ≠ ⋃︀ ϕ ⋃︀ root and K ⊆ ⋃︀ ϕ ⋃︀ root inthe case of line 24 by Lemma 4 (and the case of line 33 is similarlytreated). These are the definition of root ⇑ h ⊧ ⊙(︀ α cstit ∶ A ⌋︀ .For the complexity, the first for-loop takes 2 ⋃︀ T ⋃︀ operations periteration to create the automata copies and 2 c λ to compute ℓ n and u n . Finding the un-dominated intervals takes m − ℓ n and m to compare each u n to max ℓ n . Thesecond for-loop does at the most two CTL ∗ model-checking runsper optimal action; each run has complexity O (⋃︀ T ⋃︀ ⋅ ⋃︀ ϕ ⋃︀ ) and thereare at most m optimal actions. The total is then O ( m (⋃︀ T ⋃︀ + c λ ) + m − + m (⋃︀ T ⋃︀ ⋃︀ ϕ ⋃︀ )) . QED. □ REFERENCES [1] Rajeev Alur, Thomas A. Henzinger, and Orna Kupferman. 2002. Alternating-timeTemporal Logic.
J. ACM
49, 5 (Sept. 2002), 672–713. https://doi.org/10.1145/585265.585270[2] Udi Boker, Krishnendu Chatterjee, Thomas A. Henzinger, and Orna Kupferman.2014. Temporal Specifications with Accumulative Values.
ACM Trans. Comput.Logic
15, 4, Article 27 (July 2014), 25 pages.[3] Jan Broersen and Julien Brunel. 2008. ‘What I fail to do Today, I Have to DoTomorrow’: A Logical Study of the Propagation of Obligations. In
ComputationalLogic in Multi-Agent Systems , Fariba Sadri and Ken Satoh (Eds.). Springer BerlinHeidelberg, Berlin, Heidelberg, 82–99.
Deontic Logic Analysis of Autonomous Systems’ Safety HSCC ’20, April 22–24, 2020, Sydney, NSW, Australia [4] Krishnendu Chatterjee, Laurent Doyen, and Thomas A. Henzinger. 2008. Quan-titative Languages. In
Computer Science Logic , Michael Kaminski and SimoneMartini (Eds.). Springer Berlin Heidelberg, 385–400.[5] B.F. Chellas. 1968.
The Logical Form of Imperatives . Department of Philosophy,Stanford University.[6] Edmund M. Clarke, Orna Grumberg, and Doron A. Peled. 1999.
Model Checking .MIT Press, Cambridge, Massachusetts.[7] Dov Gabbay, John Horty, and Xavier Parent (Eds.). 2013.
Handbook of deonticlogic and normative systems . College Publications.[8] J. Christian Gerdes and Sarah M. Thornton. 2015.
Implementable Ethics forAutonomous Vehicles . Springer Berlin Heidelberg, Berlin, Heidelberg, 87–102.[9] Laura Giordano, Alberto Martelli, and Daniele Theseider Dupré. 2013. TemporalDeontic Action Logic for the Verification of Compliance to Norms in ASP. In
Proc. of the 14th Intl. Conf. on Artificial Intelligence and Law (ICAIL ’13) . ACM,New York, NY, USA, 53–62.[10] Andrew Hawkins. [n. d.]. WaymoâĂŹs self-driving cars are now available onLyftâĂŹs app in Phoenix.
The Verge ([n. d.]).[11] Risto Hilpinen and Paul McNamara. 2013. Deontic Logic: A historical survey andintroduction.[12] John Horty. 2001.
Agency and Deontic Logic . Cambridge University Press.[13] R. Koymans. 1990. Specifying Real-Time Properties with Metric Temporal Logic.
Real-Time Systems
2, 4 (1990), 255–299.[14] Alessio Lomuscio, Hongyang Qu, and Franco Raimondi. 2017. MCMAS: an open-source model checker for the verification of multi-agent systems.
Intl. Jrnl. onSoftware Tools for Technology Transfer
19, 1 (01 Feb 2017), 9–30. [15] Zohar Manna and Amir Pnueli. 1992.
The Temporal Logic of Reactive and Concur-rent Systems — Specification . Springer.[16] Paul McNamara. 2018. Deontic Logic.
The Stanford Encyclopedia of Philosophy (Fall 2018).[17] Amir Pnueli. 1977. The Temporal Logic of Programs. In
Proceedings of the 18thIEEE Symposium Foundations of Computer Science . 46–57.[18] Cristian Prisacariu and Gerardo Schneider. 2012. A dynamic deontic logic forcomplex contracts.
The Journal of Logic and Algebraic Programming
81, 4 (2012),458 – 490. Special Issue: NWPT 2009.[19] Franco Raimondi and Alessio Lomuscio. 2004. Automatic Verification of DeonticInterpreted Systems by Model Checking via OBDD’s. In
Procs. of the 16th EuropeanConf. on Artificial Intelligence .[20] A. Rizaldi and M. Althoff. 2015. Formalising Traffic Rules for Accountability ofAutonomous Vehicles. In . 1658–1665.[21] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. 2018. On a FormalModel of Safe and Scalable Self-driving Cars. (October 2018). arXiv:1708.06374v6.[22] Wiebe van der Hoek and Michael Wooldridge. 2003. Cooperation, Knowledge, andTime: Alternating-time Temporal Epistemic Logic and its Applications.
StudiaLogica
75, 1 (01 Oct 2003), 125–157. https://doi.org/10.1023/A:1026185103185[23] Georg H. von Wright. 1951. Deontic Logic.
Mind
60, 237 (January 1951).[24] Shakiba Yaghoubi and Georgios Fainekos. 2019. Gray-box Adversarial Testingfor Control Systems with Machine Learning Components (HSCC ’19)(HSCC ’19)