Hana Chockler

A Practical Approach to Coverage in Model Checking

In formal verification, we verify that a system is correct with respect to a specification. When verification succeeds and the system is proven to be correct, there is still a question of how complete the specification is, and whether it really covers all the behaviors of the system. In this paper we study coverage metrics for model checking from a practical point of view. Coverage metrics are based on modifications we apply to the system in order to check which parts of it were actually relevant for the verification process to succeed. We suggest several definitions of coverage, suitable for specifications given in linear temporal logic or by automata on infinite words. We describe two algorithms for computing the parts of the system that are not covered by the specification. The first algorithm is built on top of automata-based model-checking algorithms. The second algorithm reduces the coverage problem to the model-checking problem. Both algorithms can be implemented on top of existing model checking tools.

Coverage Metrics for Temporal Logic Model Checking

In formal verification, we verify that a system is correct with respect to a specification. Even when the system is proven to be correct, there is still a question of how complete the specification is, and whether it really covers all the behaviors of the system. In this paper we study coverage metrics for model checking. Coverage metrics are based on modifications we apply to the system in order to check which parts of it were actually relevant for the verification process to succeed. We introduce two principles that we believe should be part of any coverage metric for model checking: a distinction between state-based and logic-based coverage, and a distinction between the system and its environment. We suggest several coverage metrics that apply these principles, and we describe two algorithms for finding the uncovered parts of the system under these definitions. The first algorithm is a symbolic implementation of a naive algorithm that model checks many variants of the original system. The second algorithm improves the naive algorithm by exploiting overlaps in the variants. We also suggest a few helpful outputs to the user, once the uncovered parts are found.

Coverage Metrics for Formal Verification

In formal verification, we verify that a system is correct with respect to a specification. Even when the system is proven to be correct, there is still a question of how complete the specification is, and whether it really covers all the behaviors of the system. The challenge of making the verification process as exhaustive as possible is even more crucial in simulation-based verification, where the infeasible task of checking all input sequences is replaced by checking a test suite consisting of a finite subset of them. It is very important to measure the exhaustiveness of the test suite, and indeed, there has been an extensive research in the simulation-based verification community on coverage metrics, which provide such a measure. It turns out that no single measure can be absolute, leading to the development of numerous coverage metrics whose usage is determined by industrial verification methodologies. On the other hand, prior research of coverage in formal verification has focused solely on state-based coverage. In this paper we adapt the work done on coverage in simulation-based verification to the formal-verification setting in order to obtain new coverage metrics. Thus, for each of the metrics used in simulation-based verification, we present a corresponding metric that is suitable for the setting of formal verification, and describe an algorithmic way to check it.

A lower bound for testing juntas

We show an Ω(m) lower bound on the number of queries required to test whether a Boolean function depends on at most m out of its n variables. This improves a previously known lower bound for testing this property. Our proof is simple and uses only elementary techniques.

Coverage of Implementations by Simulating Specifications

In formal verification, we verify that an implementation is correct with respect to a specification. When verification succeeds and the implementation is proven to be correct, there is still a question of how complete the specification is, and whether it really covers all the behaviors of the implementation. In this paper we study coverage for simulation-based formal verification, where both the implementation and the specification are modelled by labeled state-transition graphs, and an implementation I satisfies a specification S if S simulates I. Our measure of coverage is based on small modifications we apply to I. A part of I is covered by S if the mutant implementation in which this part is modified is no longer simulated by S. Thus, “mutation coverage” tells us which parts of the implementation were actually essential for the success of the verification. We describe two algorithms for finding the parts of the implementation that are covered by S. The first algorithm improves a naive algorithm that checks the mutant implementations one by one by exploiting the significant overlaps among the mutant implementations. The second algorithm is symbolic, and it improves a naive symbolic algorithm by reducing the number of variables in the OBDDs involved. In addition, we compare our coverage measure with other approaches for measuring coverage.

ω-Regular languages are testable with a constant number of queries

We continue the study of combinatorial property testing. For a property ψ, an e-test for ψ, for 0 0, we describe an algorithm that gets as input an infinite lasso-shape word of the form x ċ yω, for finite words x and y, samples only a constant number of letters in x and y, returns yes if w ∈ L(A), and returns no with probability 2/3 if w is e-far from L(A). We also discuss the applicability of property testing to formal verification, where ω-regular languages are used for the specification of the behavior of nonterminating reactive systems, and computations correspond to lasso-shape words.

Temporal modalities for concisely capturing timing diagrams

Timing diagrams are useful for capturing temporal specifications in which all mentioned events are required to occur. We first show that translating timing diagrams with both partial orders on events and dont-care regions to LTL potentially yields exponentially larger formulas containing several non-localized terms corresponding to the same event. This raises a more fundamental question: which modalities allow a textual temporal logic to capture such diagrams using a single term for each event? We define the shapes of partial orders that are captured concisely by a hierarchy of textual linear temporal logics containing future and past time operators, as well Laroussinie et als forgettable past operator and our own unforeseen future operator. Our results give insight into the temporal abstractions that underlie timing diagrams and suggest that the abstractions in LTL are significantly weaker than those captured by timing diagrams.

Which bases admit non-trivial shrinkage of formulae?

Abstract. We show that the shrinkage exponent, under random restrictions, of formulae over a finite complete basis B of Boolean functions, is strictly greater than 1 if and only if all the functions in B are unate, i.e., monotone increasing or decreasing in each of their variables. As a consequence, we get non-linear lower bounds on the formula complexity of the parity function over any basis composed only of unate functions.

omega-Regular Languages Are Testable with a Constant Number of Queries

We continue the study of combinatorial property testing. For a property ?, an ?-test for ? for 0 0, we describe an algorithm that gets as input an infinite lasso-shape word of the form x ? y?, for finite words x and y, samples only a constant number of letters in x and y, returns yes if w ? L(A), and returns no with probability 2/3 if w is ?-far from L(A). We also discuss the applicability of property testing to formal verification, where ?-regular languages are used for the specification of the behavior of nonterminating reactive systems, and computations correspond to lasso-shape words.