Mutation Testing with Hyperproperties
MMutation Testing with Hyperproperties (cid:63)
Andreas Fellner , , Mitra Tabaei Befrouei , and Georg Weissenbacher AIT Austrian Institute of Technology, TU Wien
Abstract.
We present a new method for model-based mutation-driven test casegeneration. Mutants are generated by making small syntactical modifications tothe model or source code of the system under test. A test case kills a mutantif the behavior of the mutant deviates from the original system when runningthe test. In this work, we use hyperproperties—which allow to express relationsbetween multiple executions—to formalize different notions of killing for bothdeterministic as well as non-deterministic models. The resulting hyperpropertiesare universal in the sense that they apply to arbitrary reactive models and mutants.Moreover, an off-the-shelf model checking tool for hyperproperties can be usedto generate test cases. We evaluate our approach on a number of models expressedin two different modeling languages by generating tests using a state-of-the-artmutation testing tool.
Mutations—small syntactic modifications of programs that mimic typical programmingerrors—are used to assess the quality of existing test suites. A test kills a mutated pro-gram (or mutant ), obtained by applying a mutation operator to a program, if its outcomefor the mutant deviates from the outcome for the unmodified program. The percentageof mutants killed by a given test suite serves as a metric for test quality. The approachis based on two assumptions: (a) the competent programmer hypothesis [11], whichstates that implementations are typically close-to-correct, and (b) the coupling effect [26], which states that a test suites ability to detect simple errors (and mutations) isindicative of its ability to detect complex errors.In the context of model-based testing, mutations are also used to design tests. Model-based test case generation is the process of deriving tests from a reference model (whichis assumed to be free of faults) in such a way that they reveal any non-conformance ofthe reference model and its mutants, i.e., kill the mutants. The tests detect potential er-rors (modeled by mutation operators) of implementations, treated as a black box in thissetting, that conform to a mutant instead of the reference model. A test strongly kills amutant if it triggers an observable difference in behavior [11], and weakly kills a mutantif the deviation is merely in a difference in traversed program states [21].The aim of our work is to automatically construct tests that strongly kill mutantsderived from a reference model. To this end, we present two main contributions: (cid:63)
The research was supported by ECSEL JU under the project H2020 737469 AutoDrive - Ad-vancing failaware, fail-safe, and fail-operational electronic components, systems, and archi-tectures for fully automated driving to make future mobility safer, affordable, and end-useracceptable. a r X i v : . [ c s . L O ] J u l
1) A formalization of mutation killing in terms of hyperproperties [14], a formalismto relate multiple execution traces of a program which has recently gained popu-larity due to its ability to express security properties such as non-interference andobservational determinism. Notably, our formalization also takes into account po-tential non-determinism, which significantly complicates killing of mutants due tothe unpredictability of the test outcome.(2) An approach that enables the automated construction of tests by means of modelchecking the proposed hyperproperties on a model that aggregates the referencemodel and a mutant of it. To circumvent limitations of currently available modelchecking tools for hyperproperties, we present a transformation that enables thecontrol of non-determinism via additional program inputs. We evaluate our ap-proach using a state-of-the-art model checker on a number of models expressedin two different modeling languages.
Running example.
We illustrate the main concepts of our work in Figure 1. Figure 1ashows the SMV [24] model of a beverage machine, which non-deterministically serves coff (coffee) or tea after input req (request), assuming that there is still enough wtr (water) in the tank. Water can be refilled with input fill . The symbol ε representsabsence of input and output, respectively.The code in Figure 1a includes the variable mut (initialized non-deterministicallyin line 1), which enables the activation of a mutation in line 7. The mutant refills unitof water only, whereas the original model fills units.Figure 1b states a hyperproperty over the inputs and outputs of the model formal-izing that the mutant can be killed definitely (i.e., independently of non-deterministicchoices). The execution shown in Figure 1c is a witness for this claim: the test requeststwo drinks after filling the tank. For the mutant, the second request will necessarily fail,as indicated in Figure 1d, which shows all possible output sequences for the given test. Outline.
Section 2 introduces our system model and HyperLTL. Section 3 explains thenotions of potential and definite killing of mutants, which are then formalized in termsof hyperproperties for deterministic and non-deterministic models in Section 4. Sec-tion 5 introduces a transformation to control non-determinism in models, and Section 6describes our experimental results. Related work is discussed in Section 7.
This section introduces symbolic transition systems as our formalisms for representingdiscrete reactive systems and provides the syntax and semantics of HyperLTL, a logicfor hyperproperties.
A symbolic transition system (STS) is a tuple S = (cid:104)I , O , X , α, δ (cid:105) , where I , O , X arefinite sets of input, output, and state variables, α is a formula over X ∪ O (the initial i n i t ( in / out / wtr / mut ) : = ε/ε / 2 / { (cid:62) , ⊥ } next ( in ) := { ε , req , f i l l } next ( out ) : =4 i f ( in = req&wtr > { coff , t e a } e l s e : ε next ( wtr ) : =7 i f ( in = f i l l ) : ( mut ? 1 : 2)8 e l i f ( in = req&wtr > − e l s e : wtr10 next ( mut ) : = mut (a) Beverage machine with cond. mutant ∃ π ∀ π (cid:48) ∀ π (cid:48)(cid:48) (cid:50) (cid:0) ¬ mut π ∧ mut π (cid:48) ∧ ¬ mut π (cid:48)(cid:48) [ in= ε ] π ↔ [ in= ε ] π (cid:48) ↔ [ in= ε ] π (cid:48)(cid:48) ∧ [ in=req ] π ↔ [ in=req ] π (cid:48) ↔ [ in=req ] π (cid:48)(cid:48) ∧ [ in=fill ] π ↔ [ in=fill ] π (cid:48) ↔ [ in=fill ] π (cid:48)(cid:48) (cid:1) → ♦ (cid:0) ¬ ([ o= ε ] π (cid:48) ↔ [ o= ε ] π (cid:48)(cid:48) ) ∨¬ ([ o=coff ] π (cid:48) ↔ [ o=coff ] π (cid:48)(cid:48) ) ∨¬ ([ o=tea ] π (cid:48) ↔ [ o=tea ] π (cid:48)(cid:48) ) (cid:1) (b) Hyperproperty expressing killing inout εε req ε reqcoff filltea req ε reqtea ε tea (c) Definitely killing testPossible out ε ε coff / tea coff / tea ε coff / tea ε (d) Spurious test response of mutant Fig. 1: Beverage machine running exampleconditions predicate), and δ is a formula over I ∪ O ∪ X ∪ X (cid:48) (the transition relationpredicate), where X (cid:48) = { x (cid:48) | x ∈ X } is a set of primed variables representing thesuccessor states. An input I , output O , state X , and successor state X (cid:48) , respectively, isa mapping of I , O , X , and X (cid:48) , respectively, to values in a fixed domain that includes theelements (cid:62) and ⊥ (representing true and false, respectively). Y | V denotes the restrictionof the domain of mapping Y to the variables V . Given a valuation Y and a Booleanvariable v ∈ V , Y ( v ) denotes the value of v in Y (if defined) and Y [ v ] and Y [ ¬ v ] denote Y with v set to (cid:62) and ⊥ , respectively.We assume that the initial conditions- and transition relation predicate are definedin a logic that includes standard Boolean operators ¬ , ∧ , ∨ , → , and ↔ . We omit furtherdetails, as as our results do not depend on a specific formalism. We write X, O | = α and I, O, X, X (cid:48) | = δ to denote that α and δ evaluate to true under an evaluation of inputs I ,outputs O , states X , and successor states X (cid:48) . We assume that every STS has a distinctoutput O ε , representing absence of output.A state X with output O such that X, O | = α are an initial state and initial output .A state X has a transition with input I to its successor state X (cid:48) with output O iff I, O, X, X (cid:48) | = δ , denoted by X I,O −−→ X (cid:48) . A trace of S is a sequence of tuples ofconcrete inputs, outputs, and states (cid:104) ( I , O , X ) , ( I , O , X ) , ( I , O , X ) , . . . (cid:105) suchthat X , O | = α and ∀ j ≥ . X j I j ,O j +1 −−−−−→ X j +1 . We require that every state has atleast one successor, therefore all traces of S are infinite. We denote by T ( S ) the set ofall traces of S . Given a trace p = (cid:104) ( I , O , X ) , ( I , O , X ) , . . . (cid:105) , we write p [ j ] for I j , O j , X j ) , p [ j, l ] for (cid:104) ( I j , O j , X j ) , . . . , ( I l , O l , X l ) (cid:105) , p [ j, ∞ ] for (cid:104) ( I j , O j , X j ) , . . . (cid:105) and p | V to denote (cid:104) ( I | V , O | V , X | V ) , ( I | V , O | V , X | V ) , . . . (cid:105) . We lift restriction tosets of traces T by defining T | V as { p | V | t ∈ T } . S is deterministic iff there is a unique pair of an initial state and initial output andfor each state X and input I , there is at most one state X (cid:48) with output O , such that X I,O −−→ X (cid:48) . Otherwise, the model is non-deterministic .In the following, we presume the existence of sets of atomic propositions AP = { AP I ∪ AP O ∪ AP X } (intentionally kept abstract) serving as labels that characterizeinputs, outputs, and states (or properties thereof).For a trace p = (cid:104) ( I , O , X ) , ( I , O , X ) , . . . (cid:105) the corresponding trace over AP is AP ( p ) = (cid:104) AP ( I ) ∪ AP ( O ) ∪ AP ( X ) , AP ( I ) ∪ AP ( O ) ∪ AP ( X ) , . . . (cid:105) . We liftthis definition to sets of traces by defining APTr ( S ) def = { AP ( p ) | p ∈ T ( S ) } . Example 1.
Figure 1a shows the formalization of a beverage machine in SMV [24]. InFigure 1b, we use atomic propositions to enumerate the possible values of in and out .This SMV model closely corresponds to an STS: the initial condition predicate α andtransition relation δ are formalized using integer arithmetic as follows: α def = out= ε ∧ wtr=2 δ def = wtr>0 ∧ in=req ∧ out=coff ∧ wtr’=wtr-1 ∨ wtr>0 ∧ in=req ∧ out=tea ∧ wtr’=wtr-1 ∨ in=fill ∧ ¬ mut ∧ out= ε ∧ wtr’=2 ∨ in=fill ∧ mut ∧ out= ε ∧ wtr’=1 ∨ in= ε ∧ out= ε ∧ wtr’=wtr The trace p = (cid:104) ( ε, ε, , ( req , ε, ) , ( req , coff , , ( ε, tea , , . . . (cid:105) is one pos-sible execution of the system (for brevity, variable names are omitted). Examples ofatomic propositions for the system are [ in=coff ] , [ out= ε ] , [ wtr>0 ] , [ wtr=0 ] andthe respective atomic proposition trace of p is AP ( p ) = (cid:104){ [ in= ε ] , [ out= ε ] , [ wtr>0 ] } , { [ in=req ] , [ out= ε ] , [ wtr>0 ] } , { [ in=req ] , [ out=coff ] , [ wtr>0 ] } , { [ in=req ] , [ out=tea ] , [ wtr=0 ] } . . . (cid:105) In the following, we provide an overview of the HyperLTL, a logic for hyperproperties,sufficient for understanding the formalization in Section 4. For details, we refer thereader to [13]. HyperLTL is defined over atomic proposition traces (see Section 2.1) ofa fixed STS S = (cid:104)I , O , X , α, δ (cid:105) as defined in Section 2.1. Finite domains can be characterized using binary encodings; infinite domains require an ex-tension of our formalism in Section 2.2 with equality and is omitted for the sake of simplicity. yntax.
Let AP be a set of atomic propositions and let π be a trace variable from a set V of trace variables. Formulas of HyperLTL are defined by the following grammar: ψ ::= ∃ π.ψ | ∀ π.ψ | ϕϕ ::= a π | ¬ ϕ | ϕ ∨ ϕ | (cid:35) ϕ | ϕ U ϕ Connectives ∃ and ∀ are universal and existential trace quantifiers, read as ”alongsome traces” and ”along all traces”. In our setting, atomic propositions a ∈ AP expressfacts about states or the presence of inputs and outputs. Each atomic proposition issub-scripted with a trace variable to indicate the trace it is associated with. The Booleanconnectives ∧ , → , and ↔ are defined in terms of ¬ and ∨ as usual. Furthermore, we usethe standard temporal operators eventually ♦ ϕ def = true U ϕ , and always (cid:50) ϕ def = ¬♦¬ ϕ . Semantics Π | = S ψ states that ψ is valid for a given mapping Π : V →
APTr ( S ) of trace variables to atomic proposition traces. Let Π [ π (cid:55)→ p ] be as Π except that π ismapped to p . We use Π [ i, ∞ ] to denote the trace assignment Π (cid:48) ( π ) = Π ( π ) [ i, ∞ ] forall π . The validity of a formula is defined as follows: Π | = S a π iff a ∈ Π ( π )[0] Π | = S ∃ π.ψ iff there exists p ∈ APTr ( S ) : Π [ π (cid:55)→ p ] | = S ψΠ | = S ∀ π.ψ iff for all p ∈ APTr ( S ) : Π [ π (cid:55)→ p ] | = S ψΠ | = S ¬ ϕ iff Π (cid:54)| = S ϕΠ | = S ψ ∨ ψ iff Π | = S ψ or Π | = S ψ Π | = S (cid:35) ϕ iff Π [1 , ∞ ] | = S ϕΠ | = S ϕ U ϕ iff there exists i ≥ Π [ i, ∞ ] | = S ϕ and for all ≤ j < i we have Π [ j, ∞ ] | = S ϕ We write | = S ψ if Π | = S ψ holds and Π is empty. We call q ∈ T ( S ) a π -witnessof a formula ∃ π.ψ , if Π [ π (cid:55)→ p ] | = S ψ and AP ( q ) = p . In this section, we introduce mutants, tests, and the notions of potential and definitekilling. We discuss how to represent an STS and its corresponding mutant as a singleSTS, which can then be model checked to determine killability.
Mutants are variations of a model S obtained by applying small modifications to thesyntactic representation of S . A mutant of an STS S = (cid:104)I , O , X , α, δ (cid:105) (the originalmodel ) is an STS S m = (cid:104)I , O , X , α m , δ m (cid:105) with equal sets of input, output, and statevariables as S but a deviating initial predicate and/or transition relation. We assumethat S m is equally input-enabled as S , that is T ( S m ) | I = T ( S ) | I , i.e., the mutantand model accept the same sequences of inputs. In practice, this can easily be achievedby using self-loops with empty output to ignore unspecified inputs. We use standardmutation operators, such as disabling transitions, replacing operators, etc. Due to spaceimitations and the fact that mutation operators are not the primary focus of this work,we do not list them here, but refer to Appendix A and [5].We combine an original model represented by S and a mutant S m into a conditionalmutant S c ( m ) , in order to perform mutation analysis via model checking the combinedmodel. The conditional mutant is defined as S c ( m ) def = (cid:104)I , O , X ∪ { mut } , α c ( m ) , δ c ( m ) (cid:105) ,where mut is a fresh Boolean variable used to distinguish states of the original and themutated STS.Suppose S m replaces a sub-formula δ of δ by δ m , then the transition relationpredicate of the conditional mutant δ c ( m ) is obtained by replacing δ in δ by (mut ∧ δ m ) ∨ ( ¬ mut ∧ δ ) . We fix the value of mut in transitions by conjoining δ with mut ↔ mut (cid:48) . The initial conditions predicate of the conditional mutant is definedsimilarly. Consequently, for a trace p ∈ T ( S c ( m ) ) it holds that if p | { mut } = {⊥} ω then p | I∪O∪X ∈ T ( S ) , and if p | { mut } = {(cid:62)} ω then p | I∪O∪X ∈ T ( S m ) . Formally, S c ( m ) is non-deterministic, since mut is chosen non-deterministically in the initialstate. However, we only refer to S c ( m ) as “non-deterministic” if either S or S m isnon-deterministic, as mut is typically fixed in the hypertproperties in Section 4.Example 1 and Figure 1a show a conditional mutant as an STS and in SMV. Killing a mutant amounts to finding inputs for which the mutant produces outputs thatdeviate from the original model. In a reactive, model-based setting, killing has been for-malized using conformance relations [28], for example in [4,15], where an implementa-tion conforms to its specification if all its input/output sequences are part of/allowed bythe specification. In model-based testing, the model takes the role of the specificationand is assumed to be correct by design. The implementation is treated as black box,and therefore mutants of the specification serve as its proxy. Tests (i.e., input/outputsequences) that demonstrate non-conformance between the model and its mutant canbe used to check whether the implementation adheres to the specification or containsthe bug reflected in the mutant. The execution of a test on a system under test fails ifthe sequence of inputs of the test triggers a sequence of outputs that deviates from thosepredicted by the test. Formally, tests are defined as follows:
Definition 1 (Test). A test t of length n for S comprises inputs t | I and outputs t | O of length n , such that there exists a trace p ∈ T ( S ) with p | I [0 , n ] = t | I and p | O [0 , n ] = t | O . For non-deterministic models, in which a single sequence of inputs can trigger dif-ferent sequences of outputs, we consider two different notions of killing. We say thata mutant can be potentially killed if there exist inputs for which the mutant’s outputsdeviate from the original model given an appropriate choice of non-deterministic ini-tial states and transitions. In practice, executing a test that potentially kills a mutanton a faulty implementation that exhibits non-determinism (e.g., a multi-threaded pro-gram) may fail to demonstrate non-conformance (unless the non-determinism can becontrolled). A mutant can be definitely killed if there exists a sequence of inputs forwhich the behaviors of the mutant and the original model deviate independently of howon-determinism is resolved. Note potential and definite killability are orthogonal tothe folklore notions of weak and strong killing, which capture different degrees of ob-servability. Formally, we define potential and definite killability as follows:
Definition 2 (Potentially killable). S m is potentially killable if T ( S m ) | I∪O (cid:42) T ( S ) | I∪O
Test t for S of length n potentially kills S m if { q [0 , n ] | q ∈ T ( S m ) ∧ q [0 , n ] | I = t | I }| I∪O (cid:42) { p [0 , n ] | p ∈ T ( S ) }| I∪O . Definition 3 (Definitely killable). S m is definitely killable if there is a sequence ofinputs (cid:126)I ∈ T ( S ) | I , such that { q ∈ T ( S m ) | q | I = (cid:126)I }| O ∩ { p ∈ T ( S ) | p | I = (cid:126)I }| O = ∅ Test t for S of length n definitely kills S m if { q [0 , n ] | q ∈ T ( S m ) ∧ q [0 , n ] | I = t | I }| O ∩{ p [0 , n ] | p ∈ T ( S ) ∧ p [0 , n ] | I = t | I }| O = ∅ Definition 4 (Equivalent Mutant). S m is equivalent iff S m is not potentially killable. Note that definite killability is stronger than potential killabilty, though for deter-ministic systems, the two notions coincide.
Proposition 1. If S m is definitely killable then S m is potentially killable.If S m is deterministic then: S m is potentially killable iff S m is definitely killable. The following example shows a definitely killable mutant, a mutant that is onlypotentially killable, and an equivalent mutant.
Example 2.
The mutant in Figure 1a, is definitely killable, since we can force the sys-tem into a state in which both possible outputs of the original system ( coff , tea )differ from the only possible output of the mutant ( ε ).Consider a mutant that introduces non-determinism by replacing line 7 with thecode if (in=fill):(mut ? { } : 2) , indicating that the machine is filledwith either or units of water. This mutant is potentially but not definitely killable, asonly one of the non-deterministic choices leads to a deviation of the outputs.Finally, consider a mutant that replaces line 4 with if (in=req&wtr>0):(mut? coff : { coff,tea } ) and removes the mut branch of line 7, yielding a machinethat always creates coffee. Every implementation of this mutant is also correct withrespect to the original model. Hence, we consider the mutant equivalent, even thoughthe original model, unlike the mutant, can output tea . Killing with hyperproperties
In this section, we provide a formalization of potential and definite killability in termsof HyperLTL, assert the correctness of our formalization with respect to Section 3, andexplain how tests can be extracted by model checking the HyperLTL properties. AllHyperLTL formulas depend on inputs and outputs of the model, but are model-agnosticotherwise. The idea of all presented formulas is to discriminate between traces of theoriginal model ( (cid:50) ¬ mut π ) and traces of the mutant ( (cid:50) mut π ). Furthermore, we quantifyover pairs ( π, π (cid:48) ) of traces with globally equal inputs ( (cid:50) (cid:86) i ∈ AP I i π ↔ i π (cid:48) ) and expressthat such pairs will eventually have different outputs ( ♦ (cid:87) o ∈ AP O ¬ ( o π ↔ o π (cid:48) )) . To express killability (potential and definite) of a deterministic model and mutant, weneed to find a trace of the model ( ∃ π ) such that the trace of the mutant with the sameinputs ( ∃ π (cid:48) ) eventually diverges in outputs, formalized by φ as follows: φ ( I , O ) := ∃ π ∃ π (cid:48) (cid:50) ( ¬ mut π ∧ mut π (cid:48) (cid:94) i ∈ AP I i π ↔ i π (cid:48) ) ∧ ♦ ( (cid:95) o ∈ AP O ¬ ( o π ↔ o π (cid:48) )) Proposition 2.
For a deterministic model S and mutant S m it holds that S c ( m ) | = φ ( I , O ) iff S m is killable . If t is a π -witness for S c ( m ) | = φ ( I , O ) , then t [0 , n ] | I∪O kills S m (for some n ∈ N ). For potential killability of non-deterministic models and mutants, we need to find atrace of the mutant ( ∃ π ) such that all traces of the model with the same inputs ( ∀ π (cid:48) )eventually diverge in outputs, expressed in φ : φ ( I , O ) := ∃ π ∀ π (cid:48) (cid:50) (mut π ∧ ¬ mut π (cid:48) (cid:94) i ∈ AP I i π ↔ i π (cid:48) ) → ♦ ( (cid:95) o ∈ AP O ¬ ( o π ↔ o π (cid:48) )) Proposition 3.
For non-deterministic S and S m , it holds that S c ( m ) | = φ ( I , O ) iff S m is potentially killable.If s is a π -witness for S c ( m ) | = φ ( I , O ) , then for any trace t ∈ T ( S ) with t | I = s | I , t [0 , n ] | I∪O potentially kills S m (for some n ∈ N ). Appendix A covers deterministic models with non-deterministic mutants and vice-versa. o express definite killability, we need to find a sequence of inputs of the model ( ∃ π )and compare all non-deterministic outcomes of the model ( ∀ π (cid:48) ) to all non-deterministicoutcomes of the mutant ( ∀ π (cid:48)(cid:48) ) for these inputs, as formalized by φ : φ ( I , O ) := ∃ π ∀ π (cid:48) ∀ π (cid:48)(cid:48) (cid:50) (cid:0) ¬ mut π ∧ mut π (cid:48) ∧ ¬ mut π (cid:48)(cid:48) (cid:94) i ∈ AP I i π ↔ i π (cid:48) ∧ i π ↔ i π (cid:48)(cid:48) (cid:1) → ♦ (cid:0) (cid:95) o ∈ AP O ¬ ( o π (cid:48) ↔ o π (cid:48)(cid:48) ) (cid:1) In Figure 1b, we present an instance of φ for our running example. Proposition 4.
For non-deterministic S and S m , it holds that S c ( m ) | = φ ( I , O ) iff S m is definitely killable.If t is a π -witness for S c ( m ) | = φ ( I , O ) , then t [0 , n ] | I∪O definitely kills S m (for some n ∈ N ). To generate tests, we use model checking to verify whether the conditional mutantsatisfies the appropriate HyperLTL formula presented above and obtain test cases asfinite prefixes of witnesses for satisfaction.
As stated above, checking the validity of the hyperproperties in Section 4 for a givenmodel and mutant enables test-case generation. To the best of our knowledge, MCH Y - PER [17] is the only currently available HyperLTL model checker. Unfortunately, MC-H
YPER is unable to model check formulas with alternating quantifiers. Therefore, weare currently limited to checking φ ( I , O ) for deterministic models, since witnesses of φ may not satisfy φ in the presence of non-determinism.To remedy this issue, we propose a transformation that makes non-determinism con-trollable by means of additional inputs and yields a deterministic STS. The transformedmodel over-approximates killability in the sense that the resulting test cases only killthe original mutant if non-determinism can also be controlled in the system under test.However, if equivalence can be established for the transformed model, then the originalnon-deterministic mutant is also equivalent. The essential idea of our transformation is to introduce a fresh input variable that en-ables the control of non-deterministic choices in the conditional mutant S c ( m ) . Thenew input is used carefully to ensure that choices are consistent for the model and themutant encoded in S c ( m ) . W.l.o.g., we introduce an input variable nd with a domainsufficiently large to encode the non-deterministic choices in α c ( m ) and δ c ( m ) , and write While satisfiability in the presence of quantifier alternation is supported to some extent [16]. d ( X, O ) to denote a value of nd that uniquely corresponds to state X with output O .Moreover, we add a fresh Boolean variable x τ to X used to encode a fresh initial state.Let X + def = X ∪ { mut } and X + , X (cid:48) + , I, O be valuations of X + , X (cid:48) + , I , and O , and X and X (cid:48) denote X + | X and X (cid:48) + | X (cid:48) , respectively. Furthermore, ψ ( X ) , ψ ( X + , I ) , and ψ ( O, X (cid:48) + ) are formulas uniquely satisfied by X , ( X + , I ) , and ( O, X (cid:48) + ) respectively.Given conditional mutant S c ( m ) def = (cid:104)I , O , X + , α c ( m ) , δ c ( m ) (cid:105) , we define its control-lable counterpart D ( S c ( m ) ) def = (cid:104)I ∪ { nd } , O , X + ∪ { x τ } , D ( α c ( m ) ) , D ( δ c ( m ) ) (cid:105) . Weinitialize D ( δ c ( m ) ) def = δ c ( m ) and incrementally add constraints as described below. Non-deterministic initial conditions:
Let X be an arbitrary, fixed state. The uniquefresh initial state is X τ def = X [ x τ ] , which, together with an empty output, we enforce bythe new initial conditions predicate: D ( α c ( m ) ) def = ψ ( X τ , O ε ) We add the conjunct ¬ ψ ( X τ ) → ¬ x τ (cid:48) to D ( δ c ( m ) ) , in order to force x τ evaluatingto ⊥ in all states other than X τ . In addition, we add transitions from X τ to all pairs ofinitial states/outputs in α c ( m ) . To this end, we first partition the pairs in α c ( m ) into pairsshared by and exclusive to the model and the mutant: J ∩ def = { ( O, X + ) | X, O | = α c ( m ) } J orig def = { ( O, X + ) | ¬ X + (mut) ∧ ( X + , O | = α c ( m ) ) ∧ ( X + [mut] , O (cid:54)| = α c ( m ) ) } J mut def = { ( O, X + ) | X + (mut) ∧ ( X + , O | = α c ( m ) ) ∧ ( X + [ ¬ mut] , O (cid:54)| = α c ( m ) ) } For each ( O, X + ) ∈ J ∩ ∪ J mut ∪ J orig , we add the following conjunct to D ( δ c ( m ) ) : ψ ( X τ ) ∧ nd ( O, X ) → ψ ( O, X (cid:48) + ) In addition, for inputs nd ( O, X ) without corresponding target state in the model ormutant, we add conjuncts to D ( δ c ( m ) ) that represent self loops with empty outputs: ∀ ( O, X + ) ∈ J orig : ψ ( X τ [mut]) ∧ nd ( O, X ) → ψ ( O ε , X τ (cid:48) [mut]) ∀ ( O, X + ) ∈ J mut : ψ ( X τ [ ¬ mut]) ∧ nd ( O, X ) → ψ ( O ε , X τ (cid:48) [ ¬ mut]) Non-deterministic transitions:
Analogously to initial states, for each state/input pair,we partition the successors into successors shared or exclusive to model or mutant: T ∩ ( X + ,I ) def = { ( X + , I, O, X (cid:48) + ) | X I,O −−→ X (cid:48) } T orig ( X + ,I ) def = { ( X + , I, O, X (cid:48) + ) | ¬ X + (mut) ∧ ( X + I,O −−→ X (cid:48) + ) ∧ ¬ ( X + [mut] I,O −−→ X (cid:48) + ) } T mut ( X + ,I ) def = { ( X + , I, O, X (cid:48) + ) | X + (mut) ∧ ( X + I,O −−→ X (cid:48) + ) ∧ ¬ ( X + [ ¬ mut] I,O −−→ X (cid:48) + ) } A pair ( X + , I ) causes non-determinism if | ( T ∩ ( X + ,I ) ∪ T orig ( X + ,I ) ) | X ∪I∪O∪X (cid:48) | > or | ( T ∩ ( X + ,I ) ∪ T mut ( X + ,I ) ) | X ∪I∪O∪X (cid:48) | > . or each pair ( X + , I ) that causes non-determinism and each ( X + , I, O j , X (cid:48) + j ) ∈ T ∩ ( X + ,I ) ∪ T mut ( X + ,I ) ∪ T orig ( X + ,I ) , we add the following conjunct to D ( δ c ( m ) ) : ψ ( X + , I ) ∧ nd ( O j , X j ) → ψ ( O j , X (cid:48) + j ) Finally, we add conjuncts representing self loops with empty output for inputs thathave no corresponding transition in the model or mutant: ∀ ( X + , I, O j , X (cid:48) + j ) ∈ T orig ( X + ,I ) : ψ ( X + [mut] , I ) ∧ nd ( O j , X j ) → ψ ( O ε , X (cid:48) + j [mut]) ∀ ( X + , I, O j , X (cid:48) + j ) ∈ T mut ( X + ,I ) : ψ ( X + [ ¬ mut] , I ) ∧ nd ( O j , X j ) → ψ ( O ε , X (cid:48) + j [ ¬ mut]) The proposed transformation has the following properties:
Proposition 5.
Let S be a model with inputs I , outputs O , and mutant S m then1. D ( S c ( m ) ) is deterministic (up to mut ).2. T ( S c ( m ) ) | X + ∪I∪O ⊆ T ( D ( S c ( m ) ))[1 , ∞ ] | X + ∪I∪O .3. D ( S c ( m ) ) (cid:54)| = φ ( I , O ) then S m is equivalent. The transformed model is deterministic, since we enforce unique initial valuationsand make non-deterministic transitions controllable through input nd . Since we onlyadd transitions or augment existing transitions with input nd , every transition X I,O −−→ X (cid:48) of S c ( m ) is still present in D ( S c ( m ) ) (when input nd is disregarded). The po-tential additional traces of Item 2 originate from the O ε -labeled transitions for non-deterministic choices present exclusively in the model or mutant. These transitions en-able the detection of discrepancies between model and mutant caused by the introduc-tion or elimination of non-determinism by the mutation.For Item 3 (which is a direct consequence of Item 2), assume that the original non-deterministic mutant is not equivalent (i.e., potentially killable). Then D ( S c ( m ) ) | = φ ( I , O ) , and the corresponding witness yields a test which kills the mutant assumingnon-determinism can be controlled in the system under test. Killability purported by φ ,however, could be an artifact of the transformation: determinization potentially deprivesthe model of its ability to match the output of the mutant by deliberately choosing acertain non-deterministic transition. In Example 2, we present an equivalent mutantwhich is killable after the transformation, since we will detect the deviating output tea of the model and ε of the mutant. Therefore, our transformation merely allows us toprovide a lower bound for the number of equivalent non-deterministic mutants. The exhaustive enumeration of states ( J ) and transitions ( T ) outlined in Section 5.1 ispurely theoretical and infeasible in practice. However, an analogous result can often beachieved by modifying the syntactic constructs of the underlying modeling languagethat introduce non-determinism.ig. 2: Tool Pipeline of our Experiments – Non-deterministic assignments.
Non-deterministic choice over a finite set of ele-ments { x (cid:48) , . . . x (cid:48) n } , as provided by SMV [24], can readily be converted into a case-switch construct over nd . More generally, explicit non-deterministic assignments x := (cid:63) to state variables x [25] can be controlled by assigning the value of nd to x . – Non-deterministic schedulers.
Non-determinism introduced by concurrency can becontrolled by introducing input variables that control the scheduler (as proposed in[22] for bounded context switches).In case non-determinism arises through variables under-specified in transition rela-tions, these variable values can be made inputs as suggested by Section 5.1. In general,however, identifying under-specified variables automatically is non-trivial.
Example 3.
Consider again the SMV code in Figure 1a, for which non-determinism canbe made controllable by replacing line if (in=req&wtr>0): { coff,tea } withlines if (nd=0&in=req&wtr>0):coff , elif (nd=1&in=req&wtr>0):tea and adding init (nd):= { } .Similarly, the STS representation of the beverage machine, given in Example 1, canbe transformed by replacing the first two rules by the following two rules: nd=0 ∧ wtr>0 ∧ in=req ∧ out=coff ∧ wtr’=wtr-1 ∨ nd=1 ∧ wtr>0 ∧ in=req ∧ out=tea ∧ wtr’=wtr-1 ∨ In this section, we present an experimental evaluation of the presented methods. Westart by presenting the deployed tool-chain. Thereafter, we present a validation of ourmethod on one case study with another model-based mutation testing tool. Finally, wepresent quantitative results on a broad range of generic models.
Figure 2 shows the toolchain that we use to produce test suites for models encoded inthe modeling languages Verilog and SMV. Verilog models are deterministic while SMVmodels can be non-deterministic.
Variable annotation.
As a first step, we annotate variables as inputs and outputs. Theseannotations were added manually for Verilog, and heuristically for SMV (partitioningvariables into outputs and inputs).
Mutation and transformation.
We produce conditional mutants via a mutation engine.For Verilog, we implemented our own mutation engine into the open source Verilogompiler VL2MV [12]. We use standard mutation operators, replacing arithmetic op-erators, Boolean relations, Boolean connectives, constants, and assignment operators.The list of mutation operators used for Verilog can be found in Appendix A. For SMVmodels, we use the NuSeen SMV framework [5,6], which includes a mutation enginefor SMV models. The mutation operators used by NuSeen are documented in [5]. Weimplemented the transformation presented in Section 5 into NuSeen and applied it toconditional mutants.
Translation.
The resulting conditional mutants from both modeling formalisms aretranslated into AIGER circuits [9]. AIGER circuits are essentially a compact repre-sentation for finite models. The formalism is widely used by model checkers. For thetranslation of Verilog models, VL2MV and the ABC model checker are used. For thetranslation of SMV models, NuSMV is used.
Test suite creation.
We obtain a test suite, by model checking ¬ φ ( I , O ) on conditionalmutants. Tests are obtained as counterexamples, which are finite prefixes of π -witnessesto φ ( I , O ) . In case we can not find a counter-example, and use a complete modelchecking method, the mutant is provably equivalent. Case study test suite evaluation
We compare the test suite created with our methodfor a case study, with the model-based mutation testing tool MoMuT [2,15]. The casestudy is a timed version of a model of a car alarm system (CAS), which was used in themodel-based test case generation literature before [4,3,15].To this end, we created a test suite for a SMV formulation of the model. We evalu-ated its strength and correctness on an Action System (the native modeling formalismof MoMuT) formulation of the model. MoMuT evaluated our test suite by computingits mutation score — the ratio of killed- to the total number of- mutants— with respectto Action System mutations, which are described in [15].This procedure evaluates our test suite in two ways. Firstly, it shows that the tests arewell formed, since MoMuT does not reject them. Secondly, it shows that the test suite isable to kill mutants of a different modeling formalism than the one it was created from,which suggests that the test suite is also able to detect faults in implementations.We created a test suite consisting of 61 tests, mapped it to the test format acceptedby MoMuT. MoMuT then measured the mutation score of our translated test suite onthe Action System model, using Action System mutants. The measured mutation scoreis 91% on 439 Action System mutants. In comparison, the test suite achieves a mutationscore of 61% on 3057 SMV mutants. Further characteristics of the resulting test suiteare presented in the following paragraphs.
Quantitative Experiments
All experiments presented in this section were run in parallelon a machine with an Intel(R) Xeon(R) CPU at 2.00GHz, 60 cores, and 252GB RAM.We used 16 Verilog models which are presented in [17], as well as models from open-cores.org. Furthermore, we used 76 SMV models that were also used in [5]. Finally, weused the SMV formalism of CAS. All models are available in [1]. Verilog and SMVexperiments were run using property driven reachability based model checking with atime limit of 1 hour. Property driven reachability based model checking did not performwell for CAS, for which we therefore switched to bounded model checking with a depthlimit of 100. haracteristics of models.
Table 1 present characteristics of the models. For Verilogand SMV, we present average ( µ ), standard deviation ( σ ) , minimum (Min), and max-imum (Max) measures per model of the set of models. For some measurements, weadditionally present average (Avg.) or maximum (Max) number over the set of mutantsper model. We report the size of the circuits in terms of the number of inputs ( And gates ( ∆ % of Gates) of the conditional mutantand the original model, where the average is over all mutants. The last row of the tableshows the number of the mutants that are generated for the models.We can observe that our method is able to handle models of respectable size, reach-ing thousands of gates. Furthermore, ∆ Gates of the conditional mutants is relativelylow. Conditional mutants allow us to compactly encode the original and mutated modelin one model. Hyperproperties enable us to refer to and juxtapose traces from the origi-nal and mutated model, respectively. Classical temporal logic does not enable the com-parison of different traces. Therefore, mutation analysis by model checking classicaltemporal logic necessitates strictly separating traces of the original and the mutatedmodel, resulting in a quadratic blowup in the size of the input to the classical model-checker, compared to the size of the input to the hyperproperty model-checker.Table 1: Characteristics of Models
Parameters
Verilog SMV CAS µ σ
Min Max µ σ
Min Max ∆ Model checking results.
Table 2 summarizes the quantitative results of our experi-ments. The quantitative metrics we use for evaluating our test generation approach arethe mutation score (i.e. percentage of killed mutants) and the percentage of equivalentmutants, the number of generated tests, the amount of time required for generating themand the average length of the test cases. Furthermore, we show the number of times theresource limit was reached. For Verilog and SMV this was exclusively the 1 hour time-out. For CAS this was exclusively the depth limit 100.Finally, we show the total test suite creation time, including times when reaching theresource limit. The reported time assumes sequential test suite creation time. However,since mutants are model checked independently, the process can easily be parallelized,which drastically reduces the total time needed to create a test suite for a model. Theimes of the Verilog benchmark suite are dominated by two instances of the securehashing algorithm (SHA), which are inherently hard cases for model checking.We can see that the test suite creation times are in the realm of a few hours, whichcollapses to minutes when model checking instances in parallel. However, the timingmeasures really say more about the underlying model checking methods than our pro-posed technique of mutation testing via hyperporperties. Furthermore, we want to stressthat our method is agnostic to which variant of model checking (e.g. property drivenreachability, or bounded model checking) is used. As discussed above, for CAS switch-ing from one method to the other made a big difference.The mutation scores average is around 60% for all models. It is interesting to noticethat the scores of the Verilog and SMV models are similar on average, although we usea different mutation scheme for the types of models. Again, the mutation score saysmore about the mutation scheme than our proposed technique. Notice that we can onlyclaim to report the mutation score, because, besides CAS, we used a complete modelchecking method (property driven reachability). That is, in case, for example, 60% ofthe mutants were killed and no timeouts occurred, then 40% of the mutants are provablyequivalent. In contrast, incomplete methods for mutation analysis can only ever reportlower bounds of the mutation score. Furthermore, as discussed above, the 61.7% ofCAS translate to 91% mutation score on a different set of mutants. This indicates thatfailure detection capability of the produced test suites is well, which ultimately can onlybe measured by deploying the test cases on real systems.Table 2: Experimental Results
Metrics
Verilog SMV CAS µ σ
Min Max µ σ
Min Max
Mutation Score
Equivalent Mutants
Total Runtime
A number of test case generation techniques are based on model checking; a survey isprovided in [18]. Many of these techniques (such as [29,27,20]) differ in abstractionlevels and/or coverage goals from our approach.Model checking based mutation testing using trap properties is presented in [19].Trap properties are conditions that, if satisfied, indicate a killed mutant. In contrast, ourpproach directly targets the input / output behavior of the model and does not requireto formulate model specific trap properties.Mutation based test case generation via module checking is proposed in [10]. Thetheoretical framework of this work is similar to ours, but builds on module checkinginstead of hyperproperties. Moreover, no experimental evaluation is given in this work.The authors of [4] present mutation killing using SMT solving. In this work, themodel, as well as killing conditions, are encoded into a SMT formula and solved usingspecialized algorithms. Similarly, the MuAlloy [30] framework enables model-basedmutation testing for Alloy models using SAT solving. In this work, the model, as wellas killing conditions, are encoded into a SAT formula and solved using the Alloy frame-work. In contrast to these approaches, we encode only the killing conditions into a for-mula. This allows us to directly use model checking techniques, in contrast to SAT orSMT solving. Therefore, our approach is more flexible and more likely to be applicablein other domains. We demonstrate this by producing test cases for models encoded intwo different modeling languages.Symbolic methods for weak mutation coverage are proposed in [8] and [7]. The for-mer work describes the use of dynamic symbolic execution for weakly killing mutants.The latter work describes a sound and incomplete method for detecting equivalent weakmutants. The considered coverage criterion in both works is weak mutation, which, un-like the strong mutation coverage criterion considered in this work, can be encoded asa classic safety property. However, both methods could be used in conjunction with ourmethod. Dynamic symbolic execution could be used to first weakly kill mutants andthereafter strongly kill them via hyperproperty model checking. Equivalent weak mu-tants can be detected with the methods of [7] to prune the candidate space of potentiallystrongly killable mutants for hyperpropery model checking.A unified framework for defining multiple coverage criteria, including weak mu-tation and hyperproperties such as unique-cause MCDC, is proposed in [23] . Whilestrong mutation is not expressible in this framework, applying hyperproperty modelchecking to the proposed framework is interesting future work.
Our formalization of mutation testing in terms of hyperproperties enables the automatedmodel-based generation of tests using an off-the-shelf model checker. In particular, westudy killing of mutants in the presence of non-determinism, where test-case generationis enabled by a transformation that makes non-determinism in models explicit and con-trollable. We evaluated our approach on publicly available SMV and Verilog models,and will extend our evaluation to more modeling languages and models in future work. eferences
1. Mutation testing with hyperproperies benchmark models. https://git-service.ait.ac.at/sct-dse-public/mutation-testing-with-hyperproperties . Uploaded: 2019-04-25.2. B. Aichernig, H. Brandl, E. J¨obstl, W. Krenn, R. Schlick, and S. Tiran. MoMuT::UMLmodel-based mutation testing for UML. In
Software Testing, Verification and Validation(ICST), 2015 IEEE 8th International Conference on , ICST, pages 1–8, April 2015.3. Bernhard K. Aichernig, Harald Brandl, Elisabeth J¨obstl, Willibald Krenn, Rupert Schlick,and Stefan Tiran. Killing strategies for model-based mutation testing.
Softw. Test., Verif.Reliab. , 25(8):716–748, 2015.4. Bernhard K. Aichernig, Elisabeth J¨obstl, and Stefan Tiran. Model-based mutation testing viasymbolic refinement checking. 2014.5. Paolo Arcaini, Angelo Gargantini, and Elvinia Riccobene. Using mutation to assess faultdetection capability of model review.
Softw. Test., Verif. Reliab. , 25(5-7):629–652, 2015.6. Paolo Arcaini, Angelo Gargantini, and Elvinia Riccobene. Nuseen: A tool framework forthe nusmv model checker. In , pages 476–483.IEEE Computer Society, 2017.7. S´ebastien Bardin, Micka¨el Delahaye, Robin David, Nikolai Kosmatov, Mike Papadakis,Yves Le Traon, and Jean-Yves Marion. Sound and quasi-complete detection of infeasibletest requirements. In , pages 1–10, 2015.8. S´ebastien Bardin, Nikolai Kosmatov, and Franc¸ois Cheynier. Efficient leveraging of sym-bolic execution to advanced coverage criteria. In
Seventh IEEE International Conferenceon Software Testing, Verification and Validation, ICST 2014, March 31 2014-April 4, 2014,Cleveland, Ohio, USA , pages 173–182, 2014.9. Armin Biere, Keijo Heljanko, and Siert Wieringa. AIGER 1.9 and beyond, 2011. Availableat fmv.jku.at/hwmcc11/beyond1.pdf .10. Sergiy Boroday, Alexandre Petrenko, and Roland Groz. Can a model checker generate testsfor non-deterministic systems?
Electronic Notes in Theoretical Computer Science , 190(2):3–19, 2007.11. Timothy A Budd, Richard J Lipton, Richard A DeMillo, and Frederick G Sayward. Mutationanalysis. Technical report, DTIC Document, 1979.12. Szu-Tsung Cheng, Gary York, and Robert K Brayton. Vl2mv: A compiler from verilog toblif-mv.
HSIS Distribution , 1993.13. Michael R. Clarkson, Bernd Finkbeiner, Masoud Koleini, Kristopher K. Micinski, Markus N.Rabe, and C´esar S´anchez.
Temporal Logics for Hyperproperties , pages 265–284. SpringerBerlin Heidelberg, Berlin, Heidelberg, 2014.14. Michael R. Clarkson and Fred B. Schneider. Hyperproperties.
Journal of Computer Security ,18(6):1157–1210, 2010.15. Andreas Fellner, Willibald Krenn, Rupert Schlick, Thorsten Tarrach, and Georg Weis-senbacher. Model-based, mutation-driven test case generation via heuristic-guided branchingsearch. In Jean-Pierre Talpin, Patricia Derler, and Klaus Schneider, editors,
Formal Methodsand Models for System Design (MEMOCODE) , pages 56–66. ACM, 2017.16. Bernd Finkbeiner, Christopher Hahn, and Tobias Hans. Mghyper: Checking satisfiability ofHyperLTL formulas beyond the ∃ ∗ ∀ ∗ fragment. In Shuvendu K. Lahiri and Chao Wang, ed-itors, Automated Technology for Verification and Analysis (ATVA) , volume 11138 of
LectureNotes in Computer Science , pages 521–527. Springer, 2018.7. Bernd Finkbeiner, Markus N. Rabe, and C´esar S´anchez. Algorithms for model checkingHyperLTL and HyperCTL ∗ . In Daniel Kroening and Corina S. P˘as˘areanu, editors, ComputerAided Verification (CAV) , Lecture Notes in Computer Science, pages 30–48. Springer, 2015.18. Gordon Fraser, Franz Wotawa, and Paul E Ammann. Testing with model checkers: a survey.
Software Testing, Verification and Reliability , 19(3):215–261, 2009.19. Angelo Gargantini and Constance Heitmeyer. Using model checking to generate tests fromrequirements specifications. In
ACM SIGSOFT Software Engineering Notes , volume 24,pages 146–162. Springer-Verlag, 1999.20. Hyoung Seok Hong, Insup Lee, Oleg Sokolsky, and Hasan Ural. A temporal logic basedtheory of test coverage and generation. In
International Conference on Tools and Algorithmsfor the Construction and Analysis of Systems , pages 327–341. Springer, 2002.21. William E. Howden. Weak mutation testing and completeness of test sets.
IEEE Trans.Software Eng. , 8(4):371–379, 1982.22. Akash Lal and Thomas Reps. Reducing concurrent analysis under a context bound to se-quential analysis.
Formal Methods in System Design , 35(1):73–97, 2009.23. Micha¨el Marcozzi, Micka¨el Delahaye, S´ebastien Bardin, Nikolai Kosmatov, and Virgile Pre-vosto. Generic and effective specification of structural test objectives. In , pages 436–441, 2017.24. McMillan, Kenneth L. The SMV system. Technical Report CMU-CS-92-131, CarnegieMellon University, 1992.25. Greg Nelson. A generalization of dijkstra’s calculus.
ACM Transactions on ProgrammingLanguages and Systems (TOPLAS) , 11(4):517–561, October 1989.26. A. Jefferson Offutt. Investigations of the software testing coupling effect.
ACM Trans. Softw.Eng. Methodol. , 1(1):5–20, 1992.27. Sanjai Rayadurgam and Mats Per Erik Heimdahl. Coverage based test-case generation usingmodel checkers. In
Engineering of Computer Based Systems (ECBS) , pages 83–91. IEEE,2001.28. Jan Tretmans. Test generation with inputs, outputs and repetitive quiescence.
Software -Concepts and Tools , 17(3):103–120, 1996.29. Willem Visser, Corina S Psreanu, and Sarfraz Khurshid. Test input generation with javapathfinder.
ACM SIGSOFT Software Engineering Notes , 29(4):97–107, 2004.30. Kaiyuan Wang, Allison Sullivan, and Sarfraz Khurshid. Mualloy: a mutation testing frame-work for alloy. In
International Conference on Software Engineering: Companion (ICSE-Companion) , pages 29–32. IEEE, 2018.
Appendix
A.1 Verilog mutation operators
Table 3: List of supported Verilog mutation operators ( ∗ marks bit-wise operations) Type Mutation
Arithmetic Exchange binary + and − Exchange unary + and − Relations Exchange == and ! = Exchange < , ≤ , > , ≥ Boolean Exchange ! and ∼ ∗ Drop ! and ∼ ∗ Exchange && , || , & ∗ , | ∗ , XOR and
XNOR
Assignments Exchange = and < = (Blocking & Non-Blocking Assignment)Constants Replace Integer Constant c by , , c + 1 , and c − Replace Bit-Vector Constant by (cid:126) , and (cid:126) A.2 Mixed determinism casesLemma 1.
Let Π be a trace assignment with p def = Π ( π ) , q def = Π ( π (cid:48) ) , and S c ( m ) aconditional mutant.1. Π | = S c ( m ) (cid:50) ( ¬ mut π ) then p | I∪O∪X ∈ T ( S ) Π | = S c ( m ) (cid:50) ( mut π ) then p | I∪O∪X ∈ T ( S m ) Π | = S c ( m ) (cid:50) ( (cid:86) i ∈ AP I i π ↔ i π (cid:48) )) then p | I = q | I Π | = S c ( m ) ♦ ( (cid:87) o ∈ AP O ¬ ( o π ↔ o π (cid:48) )) then p | O (cid:54) = q | O Proof.
The first two statements follow directly from the definition of conditional mu-tants. The latter two statements follow directly from the fact that AP I , AP O uniquelycharacterize inputs and outputs. Proposition 6.
Let the model S with inputs I and outputs O be deterministic and themutant S m be non-deterministic. S c ( m ) | = φ ( I , O ) iff S m is potentially killable.Let p be a π -witness for S c ( m ) | = φ ( I , O ) , then there is n ∈ N , such that the test t := p [0 , n ] | I∪O potentially kills S m .Proof. Assume S m is potentially killable. Let q ∈ T ( S m ) , such that q | I ∪ O / ∈ T ( S ) | I ∪ O .Since S m is equally input-enabled, there exists a trace p ∈ T ( S ) , such that p | I = q | I .learly, p | O (cid:54) = q | O . Therefore, p and q are satisfying assignments for Φ ( I , O ) and π , π (cid:48) respectively.Assume S c ( m ) | = Φ ( I , O ) . Let p be a π -witness of Φ ( I , O ) and let q ∈ T ( S c ( m ) ) be a π (cid:48) -witness of Φ ( I , O ) without the first existential quantifier. From Lemma 1, weimmediately get p | I = q | I , and p | O (cid:54) = q | O . This shows T ( S m ) | I ∪ O (cid:42) T ( S ) | I ∪ O .Since p | O (cid:54) = q | O , there exists a smallest n ∈ N such that p [0 , n − | O = q [0 , n − | O and p [ n ] | O (cid:54) = q [ n ] | O . Clearly, p [0 , n ] | I ∪ O potentially kills S m .The following hyperproperty expresses definite killing for deterministic models andnon-deterministic mutants: φ ( I , O ) := ∃ π ∀ π (cid:48) (cid:50) ( ¬ mut π ∧ mut π (cid:48) (cid:94) i ∈ AP I i π ↔ i π (cid:48) ) → ♦ ( (cid:95) o ∈ AP O ¬ ( o π ↔ o π (cid:48) )) (1) Proposition 7.
Let the model S with inputs I and outputs O be deterministic and themutant S m be non-deterministic. S c ( m ) | = φ ( I , O ) iff S m is definitely killable.If t is a π -witness for S c ( m ) | = φ ( I , O ) , then t [0 , n ] | I∪O definitely kills S m (for some n ∈ N ).Proof. Assume that S m is definitely killable. Since S is deterministic, for every inputsequence, there is at most one trace with in T ( S ) with this input sequence. Therefore,there is an input sequence (cid:126)I and a unique trace p ∈ T ( S ) with p I = (cid:126)I and p | I ∪ O / ∈T ( S m ) | I ∪ O . Any trace assignment that maps π to p satisfies φ ( I , O ) , since either theantecedent is violated by a trace q ∈ T ( S m ) assigned to π (cid:48) with different inputs, or theconsequent is violated by a trace q ∈ T ( S m ) assigned to π (cid:48) with inputs (cid:126)I and outputsthat can only be different to p | O .Conversely, assume S c ( m ) | = φ ( I , O ) . Let q be a π -witness and p be a π (cid:48) -witnessfor which the antecedent is satisfied, which is in fact satisfiable, since S m is equallyinput-enabled. Then, from Lemma 1, we get p | I∪O∪X ∈ T ( S ) , q | I∪O∪X ∈ T ( S m ) ,and q | I = p | I . Since Π satisfies the whole formula, it must be the case that Π also sat-isfies the consequent, i.e. q | O (cid:54) = p | O (Lemma 1). Therefore, we can conclude p | I ∪ O / ∈T ( S m ) | I ∪ O , which, as argued above, is equivalent to definite killing in the determinis-tic model case.The existence of the definitely killing test can be shown analogously as in the proofof Proposition 6. Proposition 8.
Let the model S with inputs I and outputs O be non-deterministic andthe mutant S m be deterministic S c ( m ) | = φ ( I , O ) iff S m is killable . Let q be a π -witness for S c ( m ) | = φ ( I , O ) , then there is n ∈ N , such that for any trace p ∈ T ( S ) with p | I = q | I the test t := p [0 , n ] | I∪O kills S m .Proof. Potential killing directly follows from the more restricted case in Proposition 3.Since S m is deterministic, by Proposition 1 it is also definitely killable. .3 Proofs of propositionsProposition 1. If S m is definitely killable then S m is potentially killable.If S m is deterministic then: S m is potentially killable iff S m is definitely killable.Proof. Let S m be potentially killable, which implies that there is a trace q ∈ T ( S m ) ,such that there is no trace p ∈ T ( S ) with q | I ∪ O = p | I ∪ O , which implies T ( S m ) | I ∪ O (cid:42) T ( S ) | I ∪ O .Let S be deterministic and S m be potentially killable. From the definition of de-terminism, it follows that for traces q, q (cid:48) ∈ T ( S m ) with q I = q (cid:48) I it is the case that q = q (cid:48) . In other words, for every sequence of inputs (cid:126)I , we have |{ q ∈ T ( S m ) | q | I = (cid:126)I }| O | ≤ . From potential killability (i.e. T ( S m ) | I ∪ O (cid:42) T ( S ) | I ∪ O ) it followsthat there exists q ∈ T ( S m ) , such that q | O / ∈ { p ∈ T ( S m ) | p | I = q | I }| O . Sincethe set of traces in the mutant sharing inputs with q is a singleton, we have shown { q (cid:48) ∈ T ( S m ) | q (cid:48) | I = q | I }| O ∩ { p ∈ T ( S ) | p | I = q | I }| O = ∅ , i.e. definite killability. Proposition 2.
For a deterministic model S and mutant S m it holds that S c ( m ) | = φ ( I , O ) iff S m is killable . If t is a π -witness for S c ( m ) | = φ ( I , O ) , then t [0 , n ] | I∪O kills S m (for some n ∈ N ).Proof. Potential killing directly follows from the more restricted case in Proposition 6.Since S m is deterministic, by Proposition 1 it is also definitely killable. Proposition 3.
For non-deterministic S and S m , it holds that S c ( m ) | = φ ( I , O ) iff S m is potentially killable.If s is a π -witness for S c ( m ) | = φ ( I , O ) , then for any trace t ∈ T ( S ) with t | I = s | I , t [0 , n ] | I∪O potentially kills S m (for some n ∈ N ).Proof. Assume that S m is potentially killable, which implies that there is a trace q ∈T ( S m ) , such that there is no trace p ∈ T ( S ) with q | I ∪ O = p | I ∪ O . Any trace assign-ment that maps π to p satisfies φ ( I , O ) , since either the antecedent is violated by atrace q ∈ T ( S m ) assigned to π (cid:48) with different inputs, or the consequent is violated bya trace q ∈ T ( S m ) assigned to π (cid:48) with inputs (cid:126)I and outputs that can only be differentto p | O .Conversely, assume S c ( m ) | = φ ( I , O ) . Let p be a π -witness and q be a π (cid:48) -witnessfor which the antecedent is satisfied, which is in fact satisfiable, since S m is equallyinput-enabled. Then, from Lemma 1, we get p | I∪O∪X ∈ T ( S ) , q | I∪O∪X ∈ T ( S m ) ,and q | I = p | I . Since Π [ π (cid:55)→ p, π (cid:48) (cid:55)→ q ] satisfies the antecedent and the whole formula,the trace assignment also satisfies the consequent, i.e. q | O (cid:54) = p | O (Lemma 1). Since q was chosen arbitrary (besides satisfying the antecedent), we can conclude p | I ∪ O / ∈T ( S m ) | I ∪ O , i.e. S m is potentially killable.The existence of the potentially killing test can be shown analogously as in the proofof Proposition 6. roposition 4. For non-deterministic S and S m , it holds that S c ( m ) | = φ ( I , O ) iff S m is definitely killable.If t is a π -witness for S c ( m ) | = φ ( I , O ) , then t [0 , n ] | I∪O definitely kills S m (for some n ∈ N ).Proof. Let S m be definitely killable, which implies that there is a sequence of inputs (cid:126)I ∈ T ( S ) | I , such that for P (cid:126)I def = { p ∈ T ( S ) | p | I = (cid:126)I } and Q (cid:126)I def = { q ∈ T ( S ) | q | I = (cid:126)I } it is the case that P (cid:126)I | O ∩ Q (cid:126)I | O = ∅ . Since (cid:126)I is the input sequence of atrace of S , we have that P (cid:126)I (cid:54) = ∅ . Since S m is equally input-enabled, we have Q (cid:126)I (cid:54) = ∅ .We show that any p ∈ P (cid:126)I is a π -witness to S c ( m ) | = φ ( I , O ) . Let q (cid:48) ∈ T ( S m ) and p (cid:48)(cid:48) ∈ T ( S ) be arbitrary traces and consider a trace assignment that maps π to p , π (cid:48) to q (cid:48) and π (cid:48)(cid:48) to p (cid:48)(cid:48) and assume that it satisfies the antecedent (which is satisfiable, due to P (cid:126)I (cid:54) = ∅ and Q (cid:126)I (cid:54) = ∅ ). That is, q (cid:48) ∈ Q (cid:126)I and p (cid:48)(cid:48) ∈ P (cid:126)I . Since P (cid:126)I | O ∩ Q (cid:126)I | O = ∅ , we haveshown q (cid:48) | O (cid:54) = p (cid:48)(cid:48) | O . Since q (cid:48) and p (cid:48)(cid:48) were chosen arbitrarily, we have shown the traceassignment satisfies the formula, i.e. S c ( m ) φ ( I , O ) .Conversely, assume S c ( m ) | = φ ( I , O ) . Let p be a π -witness, q (cid:48) be a π (cid:48) -witness, and p (cid:48)(cid:48) be a π (cid:48) -witness for which the antecedent is satisfied, which is in fact satisfiable, since S m is equally input-enabled. Then, from Lemma 1, we get p | I∪O∪X , p (cid:48)(cid:48) | I∪O∪X ∈T ( S ) , q (cid:48) | I∪O∪X ∈ T ( S m ) , and p | I = q (cid:48) | I = p (cid:48)(cid:48) | I .Since the Π [ π (cid:55)→ p, π (cid:48) (cid:55)→ q (cid:48) , π (cid:48)(cid:48) (cid:55)→ p (cid:48)(cid:48) ] satisfies the whole formula and the an-tecedent, the trace assignment must also satisfy the consequent. That is, it must be thecase that q (cid:48) | O (cid:54) = p (cid:48)(cid:48) | O (Lemma 1). Since q (cid:48) and p (cid:48)(cid:48) were chosen arbitrarily (besides sat-isfying the antecedent), we have shown { q ∈ T ( S m ) | q | I = p | I }| O ∩ { p (cid:48)(cid:48) ∈ T ( S ) | p (cid:48)(cid:48) | I = p | I }| O = ∅ , i.e. (cid:126)I def = p | I is the input sequence showing that S m is definitelykillable.The existence of the definitely killing test can be shown analogously as in the proofof Proposition 6. Proposition 5.
Let S be a model with inputs I , outputs O , and mutant S m then1. D ( S c ( m ) ) is deterministic (up to mut ).2. T ( S c ( m ) ) | X + ∪I∪O ⊆ T ( D ( S c ( m ) ))[1 , ∞ ] | X + ∪I∪O .3. D ( S c ( m ) ) (cid:54)| = φ ( I , O ) then S m is equivalent.Proof. We show D ( S c ( m ) ) is deterministic (up to mut ). D ( S c ( m ) ) has a unique (up to mut ) initial state X τ and initial output O ε , since we fix D ( α c ( m ) ) to be satisfiable byexactly this state and output.Assume that there are state X , successor X (cid:48) (both with respect to X ∪ { mut } ∪{ x τ } ), inputs I (with respect to I ∪ { nd } ), and outputs O , O , X I,O −−−→ X (cid:48) , X I,O −−−→ X (cid:48) and either O (cid:54) = O or X (cid:48) | X (cid:54) = X (cid:48) | X . That is, there is real non-determinismremaining in D ( S c ( m ) ) . First we note that X (cid:48) ( x τ ) = X (cid:48) ( x τ ) = ⊥ due to the constraintwe add to D ( δ c ( m ) ) for x τ , so we can rule out difference in valuation of this variable.Since output O and successor state X (cid:48) are uniquely fixed with input nd ( O, I ) , it needsto be the case that X I | I\{ nd } ,O −−−−−−−−→ X (cid:48) | X + , X I | I\{ nd } ,O −−−−−−−−→ X (cid:48) | X + . This implies thathe transitions are also present in S c ( m ) . That is, ( X, I ) causes non-determinism in S c ( m ) . Therefore, in the transformation of S c ( m ) , additional constraints are added, tyingoutputs and successors of ( X, I ) to unique values of nd , which is a contradiction to thestatement that the transitions exist irrespective to the value of nd .We show p ∈ T ( S c ( m ) ) | X + ∪I∪O [0 , n ] then p ∈ T ( D ( S c ( m ) ))[1 , n + 1] | X + ∪I∪O by induction on n .First note that sets J ∩ , J orig , and J mut are pairwise disjoint and contain everyinitial state/output pair of S c ( m ) , since use the very definition of initial state/outputpairs, possibly splitting them according to values of mut . Likewise, for each ( X, I ) sets T ∩ ( X,I ) ∪ T mut ( X,I ) ∪ T orig ( X,I ) are pairwise disjoint and contain every transition for ( X, I ) asa tuple ( X, I, O, X (cid:48) ) .In the base case, n = 0 , let p = ( I , X , O ) , where X and O are initial stateand output of S c ( m ) . We need to show that there is a trace q ∈ T ( D ( S c ( m ) )) , suchthat q [1] | X m ∪I∪O = ( I , X , O ) . As noted above ( O , X ) ∈ J ∩ ∪ J orig ∪ J mut .Therefore, we add a constraint corresponding to a transition X τ nd ( O ,X ) ,O −−−−−−−−−→ X (cid:48) tothe system. Furthermore, for ( X , I ) , we add a constraint corresponding to a transition X I ,O −−−→ X (cid:48) for some output O and successor X (cid:48) . Therefore, the trace q exists.In the inductive step, assume that the statement holds for n − and consider the casefor n . Let p [ n −
1] = ( I n − , X n − , O n − ) and p [ n ] = ( I n , X n , O n ) . We need to showthat for some trace q ∈ T ( D ( S c ( m ) )) with q [ n ] | X m ∪I∪O = ( I n − , X n − , O n − ) ,which exists due to the induction hypothesis, it is the case that q | X m ∪I∪O [ n + 1] =( I n , X n , O n ) . Again, this is a consequence of ( X n − , I n − , O n , X (cid:48) n ) ∈ T ∩ ( X n − ,I n − ) ∪ T mut ( X n − ,I n − ) ∪ T orig ( X n − ,I n − ) and the transitions we add to the system, exhaustivelyconstrained by values of nd for output/successor pairs.Note also that a consequence of the above statements, and the fact that we introducetransitions for different values of nd exhaustively, is that D ( S c ( m ) ) preserves equalinput-enabledness. (cid:54)| = φ ( I , O ) then S m is equivalent is a direct consequence of the statements abouttraces, since (cid:54)| = φ ( I , O ) shows no trace in T ( D ( S c ( m ) )) is a witness to killing themutant. Since traces of S c ( m ) are included in (the projection of) this set, there can notbe a trace in T ( S c ( m ) ))