Efficiently Checking Actual Causality with SAT Solving
aa r X i v : . [ c s . A I] A p r Preprint
Efficiently Checking Actual Causalitywith SAT Solving Amjad IBRAHIM a , Simon REHWALD a and Alexander PRETSCHNER a a Department of Informatics, Technical University of Munich, Germany
Abstract.
Recent formal approaches towards causality have made the conceptready for incorporation into the technical world. However, causality reasoning iscomputationally hard; and no general algorithmic approach exists that efficientlyinfers the causes for effects. Thus, checking causality in the context of com-plex, multi-agent, and distributed socio-technical systems is a significant challenge.Therefore, we conceptualize an intelligent and novel algorithmic approach towardschecking causality in acyclic causal models with binary variables, utilizing the op-timization power in the solvers of the Boolean Satisfiability Problem (SAT). Wepresent two SAT encodings, and an empirical evaluation of their efficiency andscalability. We show that causality is computed efficiently in less than 5 secondsfor models that consist of more than 4000 variables.
Keywords. accountability, actual causality, sat solving, reasoning
1. Introduction
The tremendous power of modern information technology, and cyber-physical, systemsis rooted, among other things, in the possibility to easily compose them using software.The ease of composition is due to the virtual nature of software: if a system provides anapplication programming interface, then other systems can in principle directly interactwith that system. As we will argue, it is inherently impractical to specify all “legal,” or“safe,” or “secure” interactions with a system. In turn, this means that the possibility ofillegal, unsafe or insecure interactions cannot be excluded at design time. As a conse-quence, we cannot ensure the “adequate” functioning of such open systems; hence needto be prepared for failures of the system; and therefore need accountability mechanisms that help us identify the root cause, or responsibility, for such a failure. The technicalcontribution of this article is an efficient procedure for checking a specific kind of causal-ity. Traditionally, safety and also security analysis of critical systems start by determin-ing the system’s boundary. We will concentrate on the technical boundary here. Anyaccessible software interface, or API, then is part of that boundary. In practice, these Work supported by the German National Science Foundation DFG under grant no. PR1266/3-1, DesignParadigms for Societal- Scale Cyber-Physical Systems. reprint interface descriptions are syntactic in nature. Among other things, this means that theyabstract from restrictions on the usage of the API. For instance, a collection of data mayneed to be initialized before it is used, which is a requirement that is not part of thesyntactic interface but may require additional specification in the form of sequencingrequirements.This example generalizes: Many restrictions on the potential usage of an API are leftimplicit, or come as comments in the best case. The academic community has thereforesuggested, for a long time, to provide more detailed interface descriptions. A typical ex-ample for such interfaces are contracts [26] that require the specification of preconditionsand postconditions for the usage of a specific piece of functionality.In practice, today’s software interfaces continue to be mostly syntactic, in spite ofdecades of research and impressive results on richer notions of interfaces that also in-corporate (more detailed descriptions of) the behavior of a component. We prefer not tospeculate about the lack of adoption of these ideas in practice. Instead, we would like toremind that any interface description provides a behavior abstraction—and in this sense,the syntactic interface, or API, provides such a coarse abstraction as well: data elementsof a certain type are mapped to data elements of another type.Arguably, this is the coarsest abstraction of behavior that still is useful in practice.At the other end of the spectrum of levels of abstraction, one may argue that the finestpossible abstraction is that of the code itself. However, we do share the perspective thatcode itself provides an abstraction of behavior in that in most programming languagesit does not explicitly talk about time or resource utilization. In this sense, code is notthe finest possible abstraction of behavior. Be that as it may, the above shows that thereis a huge spectrum of possible levels of abstraction for describing the behavior, or aninterface, of a system. It is important to remember that none of these levels as such is“better” than another level: this depends on the purpose of the abstraction, as is the casefor any kind of model [32].All this does not necessarily constitute novel insights. There is a consequence, how-ever, that we feel has been underestimated in the past and that is the basis for the workpresented here: Regardless of the level of abstraction of an interface that we choose, theremust, by definition, always be parts of the behavior that are left unspecified. And this,in turn, means that the boundary of a software-intensive system usually is not, shouldnot, and most importantly: cannot be specified without leaving open several degrees offreedom in terms of how to “legally” use the system. It hence cannot be excluded thata system S is used by another system S in a way that the developer of S has neverenvisaged and which violates implicit assumptions that were made when developing S ,possibly leading to a runtime failure of S and, by consequence, also of S . One consequence is that software-rooted failures for composed, or open, systemscannot be excluded by design. Because these systems are becoming ever more complex,we consider it hence mandatory to provide mechanisms for understanding the root causesof a failure, both for eliminating the underlying (technical) problem and for assigningblame. We believe that this construction helps us understand what the notion of an “open system” means: At facevalue, a software-intensive system S cannot be “open” because there is a well-specified (syntactic) interface forthe behavior of S . However, if this interface description is too coarse and cannot or does not specify restrictionson how to use S , some usage of S may violate implicit assumptions made while developing S . In this sense, every software-intensive system is then “open” if it may be used in a way that violates implicit assumptions. reprint We call a system accountable that can help answer questions regarding the rootcause of some (usually undesired) event (other notions of accountability are studied andformalized elsewhere [18]). Accountability of a system requires at least two properties:The system must provide evidence , usually in the form of logs, that helps understand thereasons of the undesired event. Moreover, there must be some mechanism that can reasonabout causality .Different forms of causality are useful in our context. Just to mention two of them,Granger causality identifies causality with a reduction of error when it comes to pre-dicting one time series on the grounds of another [10]; and model-based diagnosis com-putes candidate sets of potentially faulty components that can explain the failure of a sys-tem [30]. In this article, we focus on one technical aspect of inferring one specific kindof causality, namely Halpern-Pearl-style actual causality for counter-factual arguments:Given a failure of a system (an effect) and a potential cause, we efficiently compute if,by counterfactual argument, the potential cause is an actual cause.
Causality is a fundamental construct of human perception. Although our ability to linkeffects to causes may seem natural, defining what precisely constitutes a cause has baf-fled scholars for centuries. Early work on defining causality goes back to Hume in theeighteenth century [15]. Hume’s definition hinted at the idea of counter-factual relationsto infer the causes of effects. Informally, we argue, counter to the fact, that A is a causeof B if B does not occur if A no longer occurs. As Lewis noted with examples, this re-lation is not sufficient to deal with interdependent, multi-factorial, and complex causes[23]. Thus, the search for a comprehensive general definition of causality continues. Re-cently, in computer science, there have been some successful and influential efforts, byHalpern and Pearl, at formalizing the idea of counter-factual reasoning and addressingthe problematic examples in philosophy literature [12].The work by Halpern and Pearl covers two notions of causality, namely actual (to-ken) causality and type (general) causality. Type causality is a forward-looking linkthat forecasts the effects of causes. It is useful for predictions in different domains likemedicine [19], and machine learning applications [29]. In this paper, we focus on actualcausality that is a rather retrospective linking of effects to causes. We are chiefly inter-ested in the Halpern-Pearl (HP) definition of actual causality [11]. Causality is usefulin law [27], security [20], software and system verification [3,22], databases [24], andaccountability [9,17].In essence, HP provides a formal definition of when we can call one or more eventsa cause of another event in a way that captures the human intuition. There have beenthree versions of HP: the original (2001), updated (2005), and modified (2015) versions,the latter of which we are using. The fundamental contribution of HP is that it opens thedoor for embedding the ability to reason about causality into socio-technical systems thatare increasingly taking control of our daily lives. Among other use cases, since actualcausality can be used to answer causal queries in the postmortem of unwanted behavior,it is a vital ingredient to enable accountability. Utilizing HP in technical systems makesit possible to empower them with all the other social concepts that should also be em-bedded into the technical world, such as responsibility [6], blame, intention, and moralresponsibility [13]. reprint Causality checking, using any version of HP, is hard. For example, under the mod-ified definition, determining causality is in general D P -complete, and NP -complete forsingleton causes [11]. The computational complexity led to a domain-specific (e.g.,database queries, counter-examples of model checking), adapted (e.g., use lineage ofqueries, use Kripke structure of programs), or restricted (e.g., monotone queries, single-ton causes, single-equation models) utilization of HP (details in Section.2). Conversely,brute-force approaches work with small models (less than 30 variables [14]) only. There-fore, to the best of our knowledge, there exists no comprehensive, efficient, and scalableframework for modeling and benchmarking causality checking for binary models (i.e.,models with binary variables only). Consequently, no existing algorithm allows applyingHP on more complex examples than the simple cases in the literature.In this paper, we argue that an efficient approach for checking causality opens thedoor for new use cases that leverage the concept in modern socio-technical systems.We conceptualize a novel approach towards checking causality in acyclic binary modelsbased on the Boolean satisfiability problem (SAT). We intelligently encode the core ofHP as a SAT query that allows us to reuse the optimization power built into SAT solvers.As a consequence of the rapid development of SAT solvers (1000 X + each decade), theyoffer a promising tactic for any solution in formal methods and program analysis [28].Leveraging this power in causality establishes a robust framework for efficiently rea-soning about actual causality. Moreover, since the transformation of SAT to other logicprogramming paradigms like answer set programming (ASP) is almost straightforward,this paper establishes the ground to tackle more causality issues (e.g., causality infer-ence) using combinatorial solving approaches. Therefore, this paper makes the followingcontributions:– An approach to check causality over binary models. It includes two SAT-encodingsthat reflect HP, and two variants for optimization,– A Java library that includes the implementation of our approach. It is easilyextensible with new optimizations and algorithms.– An empirical evaluation that uses different examples to show the efficiency andscalability of our approach.
2. Related Work
To the best of our knowledge, no previous work has tackled the technical implementa-tion of the ( modified ) version of HP yet. Conversely, the first two versions were used indifferent applications. Although they use different versions, we still consider them re-lated. These applications used the definition as a refinement of other technologies. Forexample, in [24,25,4,31], a simplified HP (the updated version) was used to refine prove-nance information and explain database conjunctive query results. Theses approaches,heavily depend on the correspondence between causes and domain-specific concepts likelineage, database repairs, and denial constraints. As a simplification, the authors useda single-equation causal model based on the lineage of the query in [24,25], and no-equation model in [4,31]. The approaches also eliminate HP’s treatment of preemption.Similar simplification has been made for Boolean circuits in [7]. available at: https://github.com/amjadKhalifah/HP2SAT1.0/ reprint In the context of model verification, a concept that enhances a counterexample re-turned by a model checker with a causal explanation based on a simplified version ofthe updated HP was proposed in [3]. Their domain-specific simplification comes fromthe fact that no dependencies between variables, and hence no equations, were required.Moreover, they used the definition of singleton causes. Similarly, in [2,21], the authorsimplemented different flavors of causality checking (based on the updated HP) usingBounded Model Checking to debug models of safety-critical systems. They employedSAT solving indirectly in the course of model checking. The authors stated that this ap-proach is better in large models regarding performance than their previous work.The common ground between all these approaches and our approach is the usage ofbinary models. However, they were published before the modified HP version. Hence,they used the older versions. In contrast to our approach, the previous works adapted thedefinition for a domain specific purpose. This was reflected in restrictions on the equa-tions of the binary model (single-equation, independent variables), the cause (singleton),or dependency on other concepts (Kripke structures, lineage formula, counter-examples).On the contrary, we propose algorithms to compute causality on binary models, withoutadaptations.Similar to our aim, [14] evaluated search-based strategies for determining causalityaccording to the original HP definition. Hopkins proposed ways to explore and prunethe search space, for computing ~ W ,~ Z that were required for that version, and consideredproperties of the causal model that makes it more efficient for computation. The resultspresented are of models that consist of less than 30 variables; in contrast, we show SAT-based strategies that compute causality for models of thousands of variables.Lastly, to prove the complexity classes, Halpern [11] used the relation between theconditions and the SAT problem. However, the concrete encoding in SAT with a size thatis linear of the number of variables is still missing.
3. Halpern-Pearl Definition
In this section, we introduce the latest HP. All versions of HP use variables to describethe world. Structural equations define how these variables influence each other. The vari-ables are split into exogenous and endogenous variables. The values of the former, calleda context ~ u , are assumed to be governed by factors that are not part of the modeled world.Consequently, exogenous variables cannot be part of a cause. The values of the endoge-nous variables, in contrast, are determined by the mentioned equations. Formally, we de-scribe a causal model in Definition 1. Similar to Halpern, we limit ourselves to acycliccausal models, in which we can compute a unique solution for the equations given acontext ~ u . Definition 1
Causal Model is a tuple M = ( U , V , R , F ) , where– U, V are sets of exogenous variables and endogenous variables respectively,– R associates with Y ∈ U ∪ V a set R ( Y ) of values,– F associates with X ∈ V F X : ( × U ∈ U R ( U )) × ( × Y ∈ V \{ X } R ( Y )) → R ( X ) Here, we define the necessary notations. A primitive event is a formula of the form X = x ,for X ∈ V and x ∈ R ( X ) . A sequence of variables X , ..., X n is abbreviated as ~ X . Analo-gously, X = x , ..., X n = x n is abbreviated as ~ X = ~ x . Variable Y can be set to value y by reprint writing Y ← y (analogously ~ Y ← ~ y for vectors). ϕ is a Boolean combination of primitiveevents. ( M ,~ u ) | = X = x if the variable X has value x in the unique solution to the equa-tions in M in context ~ u . Intervention on a model is expressed, either by setting the valuesof ~ X to ~ x , written as [ X ← x , .., X k ← x k ] , or by fixing the values of ~ X in the model,written as M ~ X ← ~ x , which effectively replaces the equations for ~ X by a constant equation X i = x i . So, ( M ,~ u ) | = [ ~ Y ← ~ y ] ϕ is identical to ( M ~ Y ← ~ y ,~ u ) | = ϕ [11]. Lastly, we use to express value substitution, e.g., [ −→ V ~ v ′ ] F X j refers to the evaluation of equation F X j given that the values of other variables are set to ~ v ′ . Definition 2
Actual Cause [11] ~ X = ~ x is an actual cause of ϕ in ( M ,~ u ) if the following three conditions hold: AC1. ( M ,~ u ) | = ( ~ X = ~ x ) and ( M ,~ u ) | = ϕ AC2.
There is a set ~ W of variables in V and a setting ~ x ′ of the variables in ~ X such that if ( M ,~ u ) | = ~ W = ~ w, then ( M ,~ u ) | = [ ~ X ← ~ x ′ , ~ W ← ~ w ] ¬ ϕ . AC3. −→ X is minimal: no subset of ~ X satisfies AC1 and AC2.
The HP definition is presented in Definition 2. AC1 checks that the cause and the effectoccurred in the real world, i.e., in M given context ~ u . AC3 is a minimality check todeal with irrelevant variables. AC2 is the core; it matches the counter-factual definitionof causality. It holds if there exists a setting ~ x ′ of the variables in ~ X different from the original setting ~ x (which led to ϕ holding true) and another set of variables ~ W that weuse to fix variables at their original value, such that ϕ does not occur. Inferring ~ W isone source of the complexity of the definition. The role of ~ W becomes clearer whenwe consider the examples in [11]. Briefly, it captures the notion of preemption whichdescribes the case when one possible cause rules out the other based on, e.g., temporalfactors.Halpern [11] shows that determining causality is in general D P -complete. The familyof complexity classes D Pk was introduced, in [1], to investigate the complexity of the original and updated definitions. AC1 can be checked in polynomial time, while
AC2 is NP -complete, and AC3 is co- NP -complete. To prove this complexity, Halpern [11]showed that AC2 could be reduced to SAT, and AC3 to UNSAT. However, the concreteencoding was not specified. Example
We consider a famous example from the literature: the rock-throwing ex-ample [23], described as follows: Suzy and Billy both throw a rock at a bottle which shat-ters if one of them hits it. We know that Suzy’s rock hits the bottle slightly earlier thanBilly’s and both are accurate throwers. Halpern models this story using the followingendogenous variables: ST , BT for “Suzy/Billy throws”, with values 0 (the person doesnot throw) and 1 (s/he does), similarly, SH , BH for “Suzy/Billy hits”, and BS for “bottleshatters”. The equations are: - BS is 1 iff one of SH and BH is 1, i.e., BS = SH ∨ BH - SH is 1 iff ST is 1, i.e., SH = ST - BH = BT = SH =
0, i.e., BH = BT ∧ ¬ SH - ST , BT are set by exogenous variables, i.e., ST = ST exo ; BT = BT exo Assuming a context ~ u that sets ST = 1 and BT = 1, the original evaluation of themodel is: BS =1 SH =1 BH =0 ST =1 BT =1. Let us assume we want to find out whether ST = 1 is a cause of BS = 1. Obviously, AC1 is fulfilled. As a candidate cause, we set ST = 0.A first attempt with ~ W = /0 shows that AC2 does not hold. However, if we arbitrary take reprint ~ W = { BH } , i.e., we replace the equation of BH with BH = 0, then AC2 holds because BS = 0, and AC3 automatically holds since the cause is a singleton. Thus, ST = BS =
4. Approach
In this section, we propose our algorithmic approaches towards the HP definition. Toanswer a causal question efficiently, we need to find an intelligent way to search for a ~ W such that AC2 is fulfilled as well as to check whether AC3 holds. Therefore, we proposean approach that uses SAT-solving. We show how to encode AC2 into a formula whose(un)satisfiability and thus the (un)fulfillment of AC2 is determined by a SAT-solver. Sim-ilarly, we show how to generate a formula whose satisfying assignments obtained with asolver indicate if AC3 holds. For AC2, such a formula F has to incorporate (1) ¬ ϕ , (2) a context ~ u , (3) a setting, ~ x ′ for candidate cause, ~ X , and (4) all possible variations of ~ W , while still (5) keeping thesemantics of the underlying model M . In the following, we describe the concept and,then, the algorithm that generates such a formula F . Since we check actual causality inhindsight, we have a situation where ~ u and ~ v are determined, and we have a candidatecause ~ X ⊆ ~ V . Thus, the first two requirements are straightforward. First, the effect ϕ should not hold anymore, hence, ¬ ϕ holds. Second, the context ~ u should be set to itsvalues in the original assignment (the values ~ u of ~ U ).Since we are treating binary models only, the setting ~ x ′ (from AC2) can be tailoreddown to negating the original value of each cause variable. This is a result of Lemma 1,which utilizes the fact that we are considering binary variables to exclude other possiblesettings and define precisely the setting ~ x ′ . The proof of the Lemma is given in AppendixA. Thus, to address the third requirement, according to Lemma 1, for ¬ ϕ to hold, all thevariables of the candidate cause ~ X are negated. Lemma 1
In a binary model, if ~ X = ~ x is a cause of ϕ , according to Definition 2, thenevery ~ x ′ in the definition of AC2 always satisfies ∀ i . x ′ i = ¬ x i . To ensure that the semantics of the model are reflected in F (requirement 5), we usethe logical equivalence operator ( ↔ ) to express the equations. Particularly, to representthe endogenous variable V i and its dependency on other variables, we use this clause V i ↔ F V i . This way, we create a (sub-)formula that evaluates to true if both sides are equivalent in their evaluation. If we do so for all other variables (that are not affectedby criteria 1-3), we ensure that F is only satisfiable for assignments that respect thesemantics of the model.Finally, we need to find a possibility to account for ~ W (requirement 4) without hav-ing to iterate over the power-set of all variables. In F , we accomplish this by addinga disjunction with the positive or negative literal of each variable V i to the previouslydescribed equivalence-formula, depending on whether the actual evaluation of V i was 1or 0, respectively. Then, we can interpret (( V i ↔ F V i ) ∨ ( ¬ ) V i ) as “ V i either follows thesemantics of M or takes on its original value represented as a positive or negative literal”. reprint Algorithm 1
Check whether AC2 holds using SAT
Input: causal model M , context h U , . . . , U n i = h u , . . . , u n i , effect ϕ , candidate cause h X , . . . , X ℓ i = h x , . . . , x ℓ i , evaluation h V , . . . , V m i = h v , . . . , v m i function F ULFILLS
AC2( M , ~ U = ~ u , ϕ ,~ X = ~ x ,~ V = ~ v ) if ( M ,~ u ) | = [ ~ X ← ¬ ~ x ] ¬ ϕ then return /0 else F : = ¬ ϕ ∧ V i = ... n f ( U i = u i ) ∧ V i = ... m , j • X j = V i (cid:0) V i ↔ F V i ∨ f ( V i = v i ) (cid:1) ֒ → ∧ V i = ...ℓ f ( X i = ¬ x i ) with f ( Y = y ) = ( Y , y = ¬ Y , y = if h U = u . . . U n = u n , V = v ′ . . . V m = v ′ m i ∈ SAT ( CNF ( F )) then ~ W : = h W , . . . , W s i s.t. ∀ i ∀ j • ( i = j ⇒ W i = W j ) ∧ ( W i = V j ⇔ v ′ j = v j ) return ~ W else return not satisfiable end if end if end function By doing so for all endogenous variables, we allow for all possible variations of ~ W . It isworth noting that we exclude those variables that are in ~ X from obtaining their originalvalue, as we are already changed their setting to ¬ ~ x and thus keeping a potential causeat its original value is not reasonable. Obviously, it might not make sense to always addthe original value for all variables. We leave this as a candidate optimization for a futurework. AC2 Algorithm
We formalize the above in Algorithm 1. The evaluation, in the input,is a list of all the variables in M and their values under ~ u . The rest is self-explanatory. Weslightly change the definition of ϕ from a combination of primitive events to a combina-tion of literals. For instance, instead of writing ϕ = ( X = ∧ X = ∨ X = ) , we woulduse ϕ = ( X ∧ ¬ X ∨ X ) . In other words, we replace each primitive ( X = x ) ∈ ϕ with X if x = ¬ X if x = ϕ in a formula. Thesame logic is achieved using the function f ( Y = y ) in line 5 of the algorithm.Before we construct formula F , we check if ~ X = ~ x given ~ W = /0 (line 2) fulfills AC2.Hence, in this case, we do not need to look for a ~ W . Otherwise, we construct F (line 4)that is a conjunction of ¬ ϕ and the exogenous variables of M as literals depending on ~ u .Note that ϕ does not necessarily consist of a single variable only; it can be any Booleanformula. For example, if ϕ = ( BS = ∧ BH = ) in the notation as defined by [11], wewould represent it in F as ( BS ∧ ¬ BH ) . This consideration is handled by Algorithm 1without further modification. In addition, we represent each endogenous variable, V i ~ X with a disjunction between its equivalence formula V i ↔ F V i and its literal representation.To conclude the formula construction, we add the negation of the candidate cause ~ X = ~ x ,a consequence of Lemma 1.If F , represented in a conjunctive normal form, is satisfiable, we obtain the satisfyingassignment (line 6) and compute ~ W (line 7) as the set of those variables whose valuations reprint were not changed in order to ensure ¬ ϕ that is finally returned. If F is unsatisfiable, notsatisfiable is returned. Minimality of ~ W In Algorithm 1, we considered ~ W to consist of all the variables whoseoriginal evaluation and satisfying assignments are equal. This is an over-approximationof the ~ W set because, possibly, there are variables that are not affected by changing thevalues of the cause, and are yet not required to be fixed in ~ W . Despite this consideration,a non-minimal ~ W is still valid according to HP. However, to compute the degree of re-sponsibility [6], a minimal ~ W is required. Therefore, we briefly discuss a modificationthat yields a minimal ~ W .We need to modify two parts of Algorithm 1. First, we cannot just consider one satisfying assignment for F . Rather, we need to analyze all the assignments. Determiningall the assignments is called an All-SAT problem. Second, we have to further analyzeeach assignment of ~ W to check if we can find a subset such that F , and thus AC2, stillholds. Specifically, we check if each element in ~ W is equal to its original value becauseit was explicitly set so, or because it simply evaluated according to its equation. In thelatter case, it is not a required part of ~ W . Precisely, in Algorithm 1, everything stays thesame until the computation of F . After that, we check whether F is satisfiable, but nowwe compute all the satisfying assignments. Subsequently, for each satisfying assignment,we compute ~ W i as explained. Then, we return the smallest ~ W i , at the cost of iterating over all satisfying assignments of the variables in V . Our approach for checking AC3 using SAT is very similar to the one for AC2. We con-struct another SAT formula, G . The difference between G and F is in how the parts ofthe cause are represented. In G , we allow each of them to take on its original value or itsnegation (e.g., A ∨ ¬ A ). Clearly, we could replace that disjunction with true or 1. How-ever, we explicitly do not perform this simplification such that a satisfying assignmentfor G still contains all variables of the causal model, M .In general, the idea is as follows. If we find a satisfying assignment for G such that atleast one conjunct of the cause ~ X = ~ x takes on a value that equals the one computed fromits corresponding equation, then, we know that this particular conjunct is not requiredto be part of the cause and there exists a subset of ~ X that fulfills AC2 as well. Thesame applies if the conjunct is equal to its original value in the satisfying assignment;this would mean that it is part of a ~ W such that ¬ ϕ holds. When collecting all thoseconjuncts, we can construct a new cause ~ X ′ = ~ x ′ by subtracting them from the originalcause and then checking whether or not it fulfills AC1. If it does, AC3 is violated becausewe identified a subset ~ X ′ of ~ X for which both AC1 and AC2 hold. AC3 Algorithm
We formalize our approach in Algorithm 2. The input and the function f ( V i = v i ) remain the same as for Algorithm 1; the latter is omitted. In case ~ X = ~ x is asingleton cause or ϕ did not occur, AC3 is then fulfilled automatically (line 2). Otherwise,line 3 shows how formula G is constructed. This construction is only different from theconstruction of F in Algorithm 1 in how to treat variables ∈ ~ X . They are added as adisjunction of their positive and negative literals. Once G is constructed, we check itssatisfiability, if it is not satisfiable we return true , i.e., AC3 is fulfilled. For example, thiscan be the case if the candidate cause ~ X did not satisfy AC2. Otherwise, we check all its reprint Algorithm 2
Check whether AC3 holds using ALL-SAT
Input: causal model M , context h U , . . . , U n i = h u , . . . , u n i , effect ϕ , candidate cause h X , . . . , X ℓ i = h x , . . . , x ℓ i , evaluation h V , . . . , V m i = h v , . . . , v m i function F ULFILLS
AC3( M , ~ U = ~ u , ϕ ,~ X = ~ x ,~ V = ~ v ) if ℓ > ∧ ( M ,~ u ) | = ϕ then G : = ¬ ϕ ∧ V i = ... n f ( U i = u i ) ∧ V i = ... m , j • X j = V i (cid:0) V i ↔ F V i ∨ f ( V i = v i ) (cid:1) ֒ → ∧ V i = ...ℓ X i ∨ ¬ X i for all h ~ U = ~ u ,~ V = ~ v ′ i ∈ SAT ( CNF ( G )) do if | n j ∈ { , .., ℓ }|∃ i • V i = X j ∧ v ′ i = v i ֒ → ∧ v ′ i = [ −→ V ~ v ′ ] F X j o | < ℓ then return false end if end for end if return true end function satisfying assignments. We need to do this, as G might also be satisfiable for the original ~ X = ~ x so that we cannot say for sure that any satisfying assignment found, proves thatthere exists a subset of the cause. Instead, we need to obtain all of them. Obviously, thisis problematic and could decrease the performance if G is satisfiable for a large numberof assignments. Therefore, we plan to address this in future work.However, for now, we compute one assignment and check the count of the conjunctsin the cause that have different values in ~ v ′ than their original, and that their formula doesnot evaluate to this assignment (line 5). If the count is less than the size of the cause, thenAC3 is violated. Otherwise we check another assignment. Combining AC2 and AC3
While developing Algorithm 1 and Algorithm 2, we discov-ered that combining both is an option for optimizing our approach. In particular, we canexploit the relationship between the satisfying assignment(s) for the formulas F and G ,i.e., ~ a F ∈ A G . This holds, as we allow the variables ~ X of a cause to be both 1 or 0 in G sothat we can show that the satisfying assignment, ~ a F for F in Algorithm 1 is an elementof the satisfying assignments A G , for G . Then, instead of computing both F and G , wecould just compute G , then filter those satisfying assignments that F would have yieldedand use them for checking AC2 while we use all satisfying assignments of G to checkAC3. Recall the example from Section.3. Since the context ~ u sets ST = BT =
1, theoriginal evaluation of the model is shown in the first row of Table.1. We want to findout whether ST = BS =
1. Algorithm.1 generates the following F , that issatisfiable for one assignment (Table.1 second row): BS = SH = BH = ST = BT =
1. All the variables, except BH and BT , change their evaluation. Thus, we concludethat ST = ~ W = { BH , BT } . Notice that even though this ~ W is notminimal, it is still valid. That said, we still can calculate a minimal ~ W . reprint Table 1. F , G assignments BS SH BH ST BTM F G ~ a G ~ a F = ¬ ϕ z}|{ ¬ BS ∧ ~ u z }| { ST exo ∧ BT exo ∧ ( equation of BS z }| { ( BS ↔ SH ∨ BH ) ∨ orig. BS z}|{ BS ) ∧ ( equation of SH ( SH ↔ ST ) | {z } ∨ orig. SH SH |{z} ) ∧ (( BH ↔ BT ∧ ¬ SH ) | {z } equation of BH ∨ ¬ BH | {z } orig. BH ) ∧ ¬ ST |{z} equation of ST ∧ (( BT ↔ BT exo ) | {z } equation of BT ∨ BT |{z} orig. BT ) To illustrate checking AC3, we ask a different question: are ST = and BT = BS =
1? Note that AC2 is fulfilled with W = /0, for this cause. Obviously, if bothdo not throw, the bottle does not shatter. Using Algorithm 2, we obtain the following G formula. G = ¬ ϕ z}|{ ¬ BS ∧ ~ u z }| { ST exo ∧ BT exo ∧ ( equation of BS z }| { ( BS ↔ SH ∨ BH ) ∨ orig. BS z}|{ BS ) ∧ ( equation of SH z }| { ( SH ↔ ST ) ∨ orig. SH z}|{ SH ) ∧ (( BH ↔ BT ∧ ¬ SH ) | {z } equation of BH ∨ ¬ BH | {z } orig. BH ) ∧ ( ST |{z} orig. ST ∨ ¬ ST |{z} negated orig. ST ) ∧ ( BT |{z} orig. BT ∨ ¬ BT |{z} negated orig. BT ) As Table.1 shows, G is satisfiable with two assignments ~ a and ~ a . For ~ a , we can seethat both ST and BT have values different from their original evaluation, and that bothdo not evaluate according to their equations. Thus, we cannot show that AC3 is violated,yet. For ~ a , BT =
1, so it is equal to the evaluation of its equation. Consequently, BT isnot a required part of ~ X , because ¬ ϕ = ¬ BS still holds although we did not set BT =
5. Evaluation
In this section, we provide details on the implementation of our algorithms, and evaluatetheir efficiency.
Our implementation is a Java library. As such, it can easily be integrated into other sys-tems. It supports both the creation of binary causal models as well as solving causalityproblems. For the modeling part and the implementation of our SAT-based approach, wetake advantage of the library
LogicNG . It provides methods for creating and modify-ing boolean formulas and allows to analyze those using different SAT solvers. We use https://github.com/logic-ng/LogicNG reprint Table 2.
Evaluated causal models
Causal Model Endogenous Vars.
Abstract Models: AM , AM SMK ), 8 suspects ( SMK ) 36, 91Leakage in Sub-sea Production System (LSP) [5] 41Binary Trees of different heights (BT) 15 - 4095Abstract Model 1 Combined with Binary Tree (ABT) 4103 the implementation of MiniSAT solver [8] within
LogicNG . For the sake of this evalua-tion, we will compare the execution time and memory allocation for the following four strategies:
BRUTE FORCE -a standard brute-force implementation of HP,
SAT -SAT-basedapproach (Algorithm1, 2),
SAT MINIMAL -the minimal ~ W extension, and SAT COMBINED -optimization of the SAT-based approach by combining AC2 and AC3. All our measure-ments were performed on Ubuntu 16.04 LTS machine equipped with an Intel R (cid:13) Core TM i7-3740QM CPU and 8 GB RAM. For each benchmark, we specified 100 warmup and100 measurement iterations. In summary, we experimented with 12 different models. On the one hand, we took thebinary models from [11]. There were 5 of them in total, namely,
Rock-Throwing , For-est Fire , Prisoners , Assassin , and
Railroad . Since these examples are rather small ( ≤ SMK .We used two variants of that example, one with 3 suspects, and the other with 8 suspects.We also used one example from the safety domain that describes a leakage in a sub-seaproduction system; we refer to it as
LSP . Last, we artificially generated models of binarytrees with different heights, denoted as BT depth , and combined them with other randommodels (non-tree graphs), denoted as ABT . For a thorough description of each model,please refer to this report [16]. Table 2 shows the list of the bigger models, along withthe number of endogenous variables. In total, we analyzed 278 scenarios, however, forspace limits, we focus on a subset of the scenarios.
In this section, we discuss some representative cases from our experiments. Table 3shows the details of these cases. The first two columns show the scenario identifier,namely, the name of the model and the ID of the scenario that differs in the details of thecausal query, i.e., ~ X and ϕ . The latter are shown in the third and fourth columns. Then,the results of the three conditions are displayed in columns AC1-AC3. The size of theminimal W set is displayed in the next column. Finally, for each algorithm, the executiontime and memory allocation are shown. We write N/A in cases where the computationwas not completed in 5 minutes or consumed too much memory. As a general remark, itdoes not matter which algorithm is applied if AC2 holds for an empty ~ W and AC3 holdsautomatically, i.e., ~ X is a singleton. reprint Table 3.
Discussed scenarios as part of the analysis ID ~ X = ~ x ϕ A C A C A C (cid:12)(cid:12)(cid:12) M i n i m a l ~ W (cid:12)(cid:12)(cid:12) BR U TE S A T S A T M I N I M A L S A T C O M B I N E D S M K FSU = ∧ FNU = ∧ AU = S M K = ✓ ✓ ✓ FSU = ✓ ✗ ✓ – N/AN/A 0.6159ms0.8650MB 0.6232ms0.8651MB 0.6098ms0.8614MB24 FSU = ∧ FNU = ∧ AU = ✓ ✓ ✓ FSU = ∧ FNU = ∧ AU = ∧ ADU = ✓ ✓ ✗ AU = ∧ ADU = S D = ✓ ✓ ✗ L SP X = ∧ X = X = ✓ ✓ ✗ X = ∧ X = ✓ ✗ ✓ – N/AN/A 0.9464ms1.2348MB 1.0245ms1.2308MB 0.6984ms0.9918MB57 X = ∧ X = ✓ ✓ ✓ B T h e i gh t = n = ∧ n = n r oo t = ✓ ✗ ✓ – N/AN/A 6540.3ms2080MB 6410ms2077MB 3587.9ms1055.0MB35 n = ∧ n = ∧ n = ∧ n = ✓ ✗ ✓ – N/AN/A 6949ms2090.2MB 6618ms2090.8MB 3328.4ms1054.9MB A B T n = I = ✓ ✓ ✓ n = ∧ n = ✓ ✓ ✓ n = ∧ n = ∧ n = ✓ ✓ ✗ As expected, the Brute-Force approach (BF) works only for smaller models ( < LSP-3, and SMK-29 . Specifically, we see these situations when AC2 holds with a small orempty ~ W , and AC3 does not hold. That is, the number of iterations BF performs is smallbecause the sets, ~ W i are ordered by size. Such examples did not exhibit the major problemof BF, i.e., the generation of all possible sets, ~ W i whose number increases exponentially,and the iterations BF might, therefore, perform to check minimality in AC3.For larger models ( >
30 variables), BF did not return an answer in 5 minutes, espe-cially when AC2 does not hold. This is seen by the several N/A entries in Table 3. Forexample, in the SMK, the set of all possible ~ W i has a size of up to 2 . In the worst case,this number of iterations is required for finding out that AC2 does not hold. It is possiblethat this number of iterations multiplied by the number of subsets of the cause needs tobe executed again to check AC3. This causes BF to be extremely slow and to consume alot of memory. The SAT by contrast, always stays below 1.5 ms and allocates less than1.5 MB during the execution for all scenarios of the SMK. Even if larger models wereconsidered, SAT handles them efficiently, e.g., BT 34 - 35 , where the underlying causalmodel contains 4095 variables, executed in ≤ s . However, the latter scenarios are spe-cial because AC2 does not hold. In ABT 1, 4 and 5 , we can see that even if AC2 does hold and ~ W is not empty, SAT takes only 8 s .While obtaining a minimal ~ W using our approach showed a rather small impactrelative to the SAT approach in most of the scenarios, it showed a significant increasein some scenarios. This impact was highly dependent on the nature and semantics of theunderlying causal model. That is, we can only observe a major impact if the number ofsatisfying assignments or the size of a non-minimal ~ W is large as this will significantlyextend the analysis. For instance, in SMK-3 , the execution time increased by about 22%.Nonetheless, there are scenarios in which we observed a significant increase, such as the reprint
ABT-4 scenario. Here, the non-minimal ~ W contains more than 4000 elements, leading toan increase of more than 200% in the execution time required to determine a minimal ~ W .Finally, combining the algorithms for AC2 and AC3 is only beneficial if AC2 andAC3 need to be explicitly analyzed (AC2 does not hold for an empty W, and the causeis not a singleton). We have many such scenarios in our examples. In evaluating them,we found out that there is a positive impact in using this optimization, but it is rathersmall on the average. Larger differences can be seen, for instance, in ABT-5 where theSAT-based approach executes for 7906 ms while the current optimization takes 3803 ms .The main finding of our experiments is that actual causality can be computed effi-ciently with our SAT-based approach. Within binary models of variables, we wereable to obtain a correct answer for any query in less than 4 seconds, using a memory of .
6. Conclusions and Future Work
It is difficult to devise automated assistance for causality reasoning in modern socio-technical systems. Causality checking, according to the formal definitions, is computa-tionally hard. Therefore, efficient approaches that scale to the complexity of such sys-tems are required. In the course of this, we proposed an intelligent way to utilize SATsolvers to check actual causality in binary models in a large scale that we believe to beparticularly relevant for accountability purposes. We empirically showed that it can ef-ficiently compute actual causality in large binary models. Even with only 30 variables,determining causality in a brute force manner is incomputable, whereas our SAT-basedapproach returned a result for such cases in 1 ms. In addition, causal models consist-ing of more than 4000 endogenous variables were still handled within seconds using theproposed approach.For future work, we will consider other logic programming paradigms such as in-teger linear programming and answer set programming to develop our approach fromchecking to possibly inferring causality. Moreover, a thorough characterization of causalmodel classes that affect the efficiency of the proposed approach is a useful follow-upto this work. We plan to extend our benchmark with models of different patterns, anddifferent cardinalities of causes. reprint
References [1] Aleksandrowicz, G., Chockler, H., Halpern, J.Y., Ivrii, A.: The computational complexity ofstructure-based causality. In: Proceedings of the Twenty-Eighth AAAI Conference on Arti-ficial Intelligence, July 27 -31, 2014, Qu´ebec City, Qu´ebec, Canada. pp. 974–980 (2014), [2] Beer, A., Heidinger, S., K¨uhne, U., Leitner-Fischer, F., Leue, S.: Symbolic causality checking usingbounded model checking. In: Model Checking Software - 22nd International Symposium, SPIN 2015,Stellenbosch, South Africa, August 24-26, 2015, Proceedings. pp. 203–221 (2015)[3] Beer, I., Ben-David, S., Chockler, H., Orni, A., Trefler, R.J.: Explaining counterexamples using causality.Formal Methods in System Design (1), 20–40 (2012). https://doi.org/10.1007/s10703-011-0132-2, https://doi.org/10.1007/s10703-011-0132-2 [4] Bertossi, L.: Characterizing and computing causes for query answers in databases from database re-pairs and repair programs. In: International Symposium on Foundations of Information and KnowledgeSystems. pp. 55–76. Springer (2018)[5] Cheliyan, A.S., Bhattacharyya, S.K.: Fuzzy fault tree analysis of oil and gasleakage in subsea production systems. Journal of Ocean Engineering and Sci-ence (1), 38 – 48 (2018). https://doi.org/https://doi.org/10.1016/j.joes.2017.11.005, [6] Chockler, H., Halpern, J.Y.: Responsibility and blame: A structural-model approach. J. Artif. Intell. Res. , 93–115 (2004). https://doi.org/10.1613/jair.1391, https://doi.org/10.1613/jair.1391 [7] Chockler, H., Halpern, J.Y., Kupferman, O.: What causes a system to satisfy a specification? ACMTransactions on Computational Logic (TOCL) (3), 20 (2008)[8] E´en, N., S¨orensson, N.: An extensible sat-solver. In: Theory and Applications of Satisfiability Testing,6th International Conference, SAT 2003. Santa Margherita Ligure, Italy, May 5-8, 2003 Selected RevisedPapers. pp. 502–518 (2003)[9] Feigenbaum, J., Jaggard, A.D., Wright, R.N.: Towards a formal model of accountabil-ity. In: 2011 New Security Paradigms Workshop, NSPW ’11, Marin County, CA, USA,September 12-15, 2011. pp. 45–56 (2011). https://doi.org/10.1145/2073276.2073282, http://doi.acm.org/10.1145/2073276.2073282 [10] Granger, C.: Investigating causal relations by econometric models and cross-spectral methods. Econo-metrica (3) (1969)[11] Halpern, J.Y.: A modification of the halpern-pearl definition of causality. In: Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina,July 25-31, 2015. pp. 3022–3033 (2015), http://ijcai.org/Abstract/15/427 [12] Halpern, J.Y.: Actual causality. The MIT Press, Cambridge, Massachussetts (2016)[13] Halpern, J.Y., Kleiman-Weiner, M.: Towards formal definitions of blameworthiness, intention, and moralresponsibility. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18) (2018)[14] Hopkins, M.: Strategies for determining causes of events. In: AAAI/IAAI. pp. 546–552 (2002)[15] Hume, D.: An Enquiry Concerning Human Understanding (1748)[16] Ibrahim, A.: Efficient checking of actual causality via sat solving - benchmarked models, https://github.com/amjadKhalifah/HP2SAT1.0/blob/master/doc/models.pdf [17] Kacianka, S., Kelbert, F., Pretschner, A.: Towards a unified model of accountability infrastructures.In: Proceedings First Workshop on Causal Reasoning for Embedded and safety-critical Systems Tech-nologies, CREST ETAPS 2016, Eindhoven, The Netherlands, 8th April 2016. pp. 40–54 (2016).https://doi.org/10.4204/EPTCS.224.5, https://doi.org/10.4204/EPTCS.224.5 [18] Kacianka, S., Pretschner, A.: Understanding and formalizing accountability for cyber-physical systems.In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). pp. 3165–3170.IEEE (2018)[19] Kleinberg, S., Hripcsak, G.: A review of causal inference for biomedical informatics. Jour-nal of Biomedical Informatics (6), 1102–1112 (2011). https://doi.org/10.1016/j.jbi.2011.07.001, https://doi.org/10.1016/j.jbi.2011.07.001 [20] K¨unnemann, R., Esiyok, I., Backes, M.: Automated verification of accountability in security protocols.CoRR abs/1805.10891 (2018), http://arxiv.org/abs/1805.10891 [21] Leitner-Fischer, F.: Causality Checking of Safety-Critical Software and Systems. Ph.D. thesis, Univer-sity of Konstanz, Germany (2015), http://kops.uni-konstanz.de/handle/123456789/30778 reprint [22] Leitner-Fischer, F., Leue, S.: Causality checking for complex system models. In: Verification, ModelChecking, and Abstract Interpretation, 14th International Conference, VMCAI 2013, Rome, Italy, Jan-uary 20-22, 2013. Proceedings (2013)[23] Lewis, D.: Causation. Journal of Philosophy (17), 556–567 (1973). https://doi.org/10.2307/2025310[24] Meliou, A., Gatterbauer, W., Halpern, J.Y., Koch, C., Moore, K.F., Su-ciu, D.: Causality in databases. IEEE Data Eng. Bull. (3), 59–67 (2010), http://sites.computer.org/debull/A10sept/suciu.pdf [25] Meliou, A., Gatterbauer, W., Moore, K.F., Suciu, D.: The complexity of causalityand responsibility for query answers and non-answers. PVLDB (1), 34–45 (2010), [26] Meyer, B.: Applying ”design by contract”. IEEE Computer (10), 40–51 (1992).https://doi.org/10.1109/2.161279[27] Moore, M.S.: Causation and responsibility : an essay in law, morals, and metaphysics. Oxford Univ.Press, Oxford (2009)[28] Newsham, Z., Ganesh, V., Fischmeister, S., Audemard, G., Simon, L.: Impact of community structureon sat solver performance. In: International Conference on Theory and Applications of SatisfiabilityTesting. pp. 252–268. Springer (2014)[29] Pearl, J.: Theoretical impediments to machine learning with seven sparks from the causal revolution.arXiv preprint arXiv:1801.04016 (2018)[30] Reiter, R.: A theory of diagnosis from first principles. Artif. Intell. (1), 57–95 (1987)[31] Salimi, B., Bertossi, L.: From causes for database queries to repairs and model-based diagnosis andback. arXiv preprint arXiv:1412.4311 (2014)[32] Stachowiak, H.: Allgemeine Modelltheorie. Springer (1973), available online: https://archive.org/details/Stachowiak1973AllgemeineModelltheorie reprint A. Appendix: Lemma ProofLemma 2
In a binary model, if ~ X = ~ x is a cause of ϕ , according to HP [11] definition,then every ~ x ′ in the definition of AC2 always satisfies ∀ i • x ′ i = ¬ x i . Proof 1
We use the following notation: −→ X ( n ) stands for a vector of length n, X , . . . , X n ;and −→ X ( n ) = −→ x ( n ) stands for X = x , . . . , X n = x n . Let −→ X ( n ) = −→ x ( n ) be a cause for ϕ insome model M.1. AC1 yields ( M , −→ u ) | = ( −→ X ( n ) = −→ x ( n ) ) ∧ ( M , −→ u ) | = ϕ . (1)
2. Assume that the lemma does not hold. Then there is some index k such that x ′ k = x k and AC2 holds. Because we are free to choose the ordering of the variables, let usset k = n wlog. We may then rewrite AC2 as follows: ∃−→ W , −→ w , −→ x ′ ( n ) • ( M , −→ u ) | = ( −→ W = −→ w ) = ⇒ ( M , −→ u ) | = h −→ X ( n − ) ← −→ x ′ ( n − ) , X n ← x n , −→ W ← −→ w i ¬ ϕ . (2)
3. We will show that equations 1 and 2 give rise to a smaller cause, namely −→ X ( n − ) = −→ x ( n − ) , contradicting the minimality requirement AC3. We need to show that thesmaller cause −→ X ( n − ) = −→ x ( n − ) satisfy AC1 and AC2, as stated by equations 3and 4 below. This violates the minimality requirement of AC3 for −→ X ( n ) = −→ x ( n ) . ( M , −→ u ) | = ( −→ X ( n − ) = −→ x ( n − ) ) ∧ ( M , −→ u ) | = ϕ (3) states AC1 for a candidate “smaller” cause −→ X ( n − ) . Similarly, ∃−→ W ∗ , −→ w ∗ , −→ x ′∗ ( n − ) • ( M , −→ u ) | = ( −→ W ∗ = −→ w ∗ )= ⇒ ( M , −→ u ) | = h −→ X ( n − ) ← −→ x ′∗ ( n − ) , −→ W ∗ ← −→ w ∗ i ¬ ϕ (4) formulates AC2 for this candidate smaller cause −→ X ( n − ) .4. Let Ψ denote the structural equations that define M. Let Ψ ′ be Ψ without theequations that define the variables −→ X ( n ) and −→ W ; and let Ψ ′′ be Ψ without theequations that define the variables −→ X ( n − ) and −→ W . Clearly, Ψ ′′ = ⇒ Ψ ′ .We can turn equation 1 into a propositional formula, namelyE : = (cid:16) Ψ ∧ −→ X ( n − ) = −→ x ( n − ) ∧ X n = x n (cid:17) ∧ ϕ . (5) reprint Similarly, equation 3 is reformulated asE : = (cid:16) Ψ ∧ −→ X ( n − ) = −→ x ( n − ) (cid:17) ∧ ϕ . (6) Because equation 2 holds, we fix some −→ W , −→ w , −→ x ′ ( n ) that make it true and rewritethis equation asE : = (cid:16) Ψ ′ ∧ −→ X ( n − ) = −→ x ′ ( n − ) ∧ X n = x n ∧ −→ W = −→ w (cid:17) = ⇒ ¬ ϕ . (7) Finally, in equation 4, we use exactly these values to also fix −→ W ∗ = −→ W , −→ w ∗ = −→ w ,and −→ x ′∗ ( n − ) = −→ x ′ ( n − ) , and reformulate this equation asE : = (cid:16) Ψ ′′ ∧ −→ X ( n − ) = −→ x ′ ( n − ) ∧ −→ W = −→ w (cid:17) = ⇒ ¬ ϕ . (8) It is then a matter of equivalence transformations to show that ( Ψ ′′ = ⇒ Ψ ′ ) = ⇒ (cid:16) ( E ∧ E ) = ⇒ ( E ∧ E ) (cid:17) (9)(9)