A Formally Verified Protocol for Log Replication with Byzantine Fault Tolerance
aa r X i v : . [ c s . D C ] S e p A Formally Verified Protocol for Log Replicationwith Byzantine Fault Tolerance
Joel Wanner, Laurent Chuat, and Adrian Perrig
Network Security Group, Department of Computer Science, ETH Zurich, Switzerland
Abstract —Byzantine fault tolerant protocols enable state repli-cation in the presence of crashed, malfunctioning, or actively ma-licious processes. Designing such protocols without the assistanceof verification tools, however, is remarkably error-prone. In anadversarial environment, performance and flexibility come at thecost of complexity, making the verification of existing protocolsextremely difficult. We take a different approach and propose aformally verified consensus protocol designed for a specific usecase: secure logging. Our protocol allows each node to proposeentries in a parallel subroutine, and guarantees that correct nodesagree on the set of all proposed entries, without leader election.It is simple yet practical, as it can accommodate the workloadof a logging system such as Certificate Transparency. We showthat it is optimal in terms of both required rounds and tolerablefaults. Using Isabelle/HOL, we provide a fully machine-checkedsecurity proof based upon the Heard-Of model, which we extendto support signatures. We also present and evaluate a prototypeimplementation.
Index Terms —Byzantine fault tolerance, consensus algorithm,formal verification
I. I
NTRODUCTION
The problem of Byzantine consensus has been the subjectof a considerable amount of research over the past decades,giving rise to various Byzantine fault tolerant (BFT) protocols,most notably Practical Byzantine Fault Tolerance (PBFT)by Castro and Liskov [1]. In response to the publication ofPBFT, there have been many attempts to improve on theperformance and robustness of the protocol by focusing ondifferent scenarios. For instance, Zyzzyva [2] is designed tobe especially efficient in the absence of failures, whereasAardvark [3], on the contrary, is designed to react gracefullywhen failures occur.These protocols were designed for high-throughput, low-latency state-machine replication. Unfortunately, this is onlypossible at the cost of complexity [4]. The BFT-SM A R T [5]library, which implements a variant of PBFT, can serve asa benchmark with almost , lines of Java code. Evenin a benign fault model, where nodes can only crash andmessages may be lost but not modified, distributed systems arenotoriously hard to design and implement. In the presence ofpossibly malicious participants, arguing about the correctnessand security of such protocols is an even greater challenge, orin the words of Lamport et al. [6]: “We know of no area incomputer science or mathematics in which informal reasoningis more likely to lead to errors than in the study of this type ofalgorithm.” To guarantee the security of such complex systems,a formal treatment is thus essential. The traditional approach in the distributed systems community is to provide a pen-and-paper proof for the desired properties of the protocol. At best,such proofs provide some intuition about why the claimedproperties hold, but since they lack the rigor that is requiredto argue about such systems, they cannot be reasonably used asa guarantee. Past experiences, such as the Chord protocol [7],which had all of its hand-proved properties refuted by modelchecking, have shown that proofs must be machine-checked .Such proofs tend to be much longer and more detailed thantheir hand-crafted counterparts, but manual error can be ruledout conclusively using proof assistants .To the best of our knowledge, there exists no completemachine-checked proof for any authenticated BFT protocol.Most work on verifying distributed systems has instead fo-cused on consensus algorithms like Raft [8] and Paxos [9],which only tolerate benign faults. The IronFleet project [10]can serve as a benchmark for the complexity of large-scaleverification efforts, as it expended approximately . person-years for the proof of a Paxos-based distributed system andits implementation. Due to the tremendous complexity of aByzantine fault model caused by the introduction of arbitrarybehavior, it is reasonable to assume that the effort of verifyinga complex BFT protocol would require even more resources.The lack of formal verification makes general-purpose BFTprotocols unsuitable for security-critical applications, evenif they have been tested and deployed in practice. To thebest of our knowledge, the only instance of complete formalverification of a BFT protocol is by Debrat and Merz [11], whoverified two algorithms proposed by Biely et al. [12] usingthe Isabelle/HOL [13] proof assistant. However, the propertiesprovided by these very simple algorithms are too weak for usein many realistic settings.Instead of aiming to develop a general-purpose BFT sys-tem, we focus on the use case of secure logging , a criticalcomponent in a variety of systems: modern public-key infras-tructures [14, 15], online voting systems [16, 17], secure times-tamping services [18, 19], and more [20]. In this endeavor, wemake the following contributions: • We present Logres, a BFT protocol designed specificallyfor secure log replication, and provide machine-checkedproofs of all its properties using the Isabelle/HOL proofassistant. Our protocol model and proofs consist of ap-proximately and lines of code, respectively, andare available online [21]. Although the protocol is simple,our verification revealed subtle flaws in its initial design,which have since been fixed.1
We extend the Heard-Of model [22] to capture theconcept of digital signatures. Our extended model canbe used to verify other BFT algorithms that make use ofsignatures. • We evaluate the performance of a prototype implemen-tation to demonstrate that our protocol can be used inpractical scenarios.II. B
ACKGROUND : S
ECURE L OGGING
Logging can trivially be performed by a single server,but this server must then be trusted to (a) accept all validrequests, (b) not remove existing entries from the log, and (c)show the same view of the log to all clients. Verifiable datastructures based on cryptographic primitives (such as Merklehash trees) [23, 24] enable the efficient auditing of logs. Thisis, most notably, the approach employed in the CertificateTransparency (CT) framework [14]. Verifiable logging byitself is not sufficient though, as a malicious log server canstill choose to ignore requests and show different views todifferent clients [25]. A log server ignoring specific requestsis particularly problematic, because such misbehavior is hardto demonstrate and reporting it to a third party has privacyimplications [26].Relying on a single server has obvious drawbacks: weakest-link security, no resilience to failure, and no censorshipresilience. On the other hand, relying on a large collectionof non-synchronized log servers makes monitoring difficult.Indeed, a client cannot simply query one CT log server toinspect all entries related to a given domain name, for example,but must instead rely on monitors that keep entire copies ofseveral logs. In turn, monitors must be trusted to correctlydisplay all relevant entries from all trusted logs, which hasrevealed to be a challenge in practice [27]. For these reasons,we propose a protocol that allows independent entities tomaintain a single log, thus providing resilience to compromise,failure, and censorship, while facilitating the monitoring of thelog’s contents by resource-limited clients.A large majority of CT log servers accepted by GoogleChrome have a “maximum merge delay” of 24 hours at thetime of writing. This means that log servers will typicallyappend newly submitted certificates to their hash tree within24 hours. In such a context, our protocol would operate ona timescale that is perhaps unconventional for a distributedsystem, with each round of the protocol lasting several hours.However, we evaluate Logres within much smaller timeframesas well in Section VI, and find it to be able to supportsubstantial workloads even with a 1-minute period.III. P
ROBLEM D EFINITION
A. Log Replication with Byzantine Fault Tolerance
BFT protocols are commonly designed to achieve state-machine replication , where processes agree on an ordered setof incoming requests from clients, creating an input log that isequal on all processes. Running a deterministic state machineon the log then produces the same results on each node. Thedesign goals in this problem space are usually low latency and high throughput, enabling the protocol to handle a highvolume of requests quickly.This paper considers the related but slightly different prob-lem of BFT log replication . In this problem, a set of n nodes,of which at most f may fail, periodically run a distributedalgorithm to maintain a log. There is an arbitrary number ofclients in the system that can send messages to the nodes,requesting entries to be added to the log. Moreover, theclients can obtain the most recently created log along withan authenticator and verify the validity of its entries locally.In order to achieve log replication, a protocol must satisfy thefollowing properties: Agreement
All valid logs created during a run of the protocolmust be equal.
Completeness
If an entry is submitted by a client to a correctnode, the node will include it in its next log produced.
Liveness
A run of the protocol must always produce a newvalid log for every correct node.This problem is different from BFT state-machine replica-tion in three ways: • There exists an inherent total order on entries (e.g.,alphabetical or chronological). Therefore, no coordinationis required to determine an ordering, unlike in the state-machine replication problem. • Clients are not limited to obtaining the output of a statemachine. Instead, they can verify the integrity of theentire log, or parts of it. • The system aggregates entries and produces new outputsin fixed intervals, not in response to each request.Due to these differences, the log replication problem allowsfor less complex solutions, as client requests do not need tobe processed individually with low latency. Nevertheless, thisproblem appears in various real-world systems, such as public-key infrastructures.
B. Assumptions
We use the standard Byzantine fault model, where up to f of the n nodes may crash, malfunction, or even be activelymalicious (and colluding). We call these nodes faulty , andthere are at least n − f remaining nodes that we call correct .The protocol relies on the following assumptions, which arecommon for BFT protocols:(A1) There is a correct majority of nodes, i.e., the constraint n > f must hold.(A2) Messages sent between correct nodes are neither lost normodified, i.e., communication is synchronous.(A3) Every node has a key pair and knows the public keys ofall other nodes.We stress that assumption (A2) is attainable in practiceusing loose time synchronization combined with a transportprotocol that provides reliable and authenticated communica-tion. Moreover, if a link between two nodes fails, the nodes cansimply be considered faulty, and if the parameter f is chosenlarge enough, the protocol will be able to continue operatingwithout any issues.2V. T HE L OGRES P ROTOCOL
A. Definitions and Notation
A log contains an ordered sequence of entries as well as anexpiration timestamp. The makeLog ( L, X ) function creates anew log from the previous log L and a set of new entries X . A log is valid if and only if it has not expired and issigned by f + 1 different nodes. These signatures serve toverify the authenticity of a log and can be computed over adigest of it (using the getDigest ( L ) function, which may be asimple hash function, or may return the root of a hash tree,for example). Since there are at most f faulty nodes in thesystem, the requirement of f + 1 signatures ensures that theycannot collude to forge a log without obtaining a signaturefrom a correct node.We use σ i ( x ) to denote a signature for x created by process i using its private key. h x | i i is shorthand for a signed term,i.e., h x | i i := ( x, σ i ( x )) . Analogously, we use h x | W i for aterm x that is signed by a set of nodes W , which are calledwitnesses. B. Protocol Overview
The protocol is organized into rounds of communication,which can be implemented in asynchronous settings assumingloose time synchronization.A key insight into how the protocol can be kept simplewhile still achieving strong security properties is that it doesnot require leader election. This concept allows low-latencyBFT protocols to process requests quickly by designatingone process as a leader, also called primary . While this canreduce the number of messages required, electing a leader alsointroduces a high amount of complexity, as the system needsto be able to handle cases where the current primary crashes oris actively malicious. For this purpose, processes can initiate a“view change” phase to convince others to hold a new election.At the core of Logres lies the C
ONSENSUS subroutine, adistributed consensus algorithm. While this subroutine is alsobased on a primary, we avoid the process of leader electionby running n instances in parallel, such that each node is theleader of exactly one thread. The goal of this technique is toachieve interactive consistency , which Pease et al. [28] defineas follows. Each node i chooses an initial value v i , and thefollowing properties must hold:(IC1) All correct nodes agree on the same vector V of n values.(IC2) For a correct node i , all correct nodes agree on i ’s initialvalue: V i = v i .Our protocol operation is specified in detail in Algorithm 1and illustrated in Figure 1. It consists of three phases:1) Collection ( round)Clients can send requests for entries to be added to thelog, where each request should be sent to f + 1 distinctnodes to guarantee that at least one correct node receivesit. Each node stores all entries that it has received andthat have not been added locally. During this phase, nocommunication takes place between the nodes. Collection . . . . . . . . .
Consensus( f + 1 rounds) Signing L i − L i − L i Fig. 1. An overview of the protocol operation. Each white rectanglerepresents a round of communication. Logs are characterized by their va-lidity period. The exact definition of a validity period is left to applicationdevelopers. Round lengths can also be tuned to each application.
Algorithm 1
Main protocol procedure L OGRES
Code for node i Collection phase: X ← collect entries from clients Consensus phase: { X , . . . , X n } ← { C ONSENSUS ( p, X ) : 1 ≤ p ≤ n } X ← X ∪ . . . ∪ X n Signing phase: L ← log from previous epoch L ′ ← makeLog ( L, X ) h ← getDigest ( L ′ ) broadcast σ i ( h ) Σ ← receive θ − valid signatures for h publish ( L ′ , Σ ∪ { σ i ( h ) } ) Consensus ( f + 1 rounds)With the set X of all cached entries from the previousphase as input, the C ONSENSUS subroutine is run inparallel n times. In each of these executions, a differentnode acts as the primary. The C ONSENSUS subroutineis explained in more detail in Section IV-C, but for thesake of this overview, we can assume that it achievesinteractive consistency for initial values X and the resultvector is ( X , . . . , X n ) . This implies that after line 4,all correct nodes obtain the same value for X , and theircollected entries from the previous phase are containedin X .3) Signing ( round)Using the log from the previous run and the union of allnew entries, each node constructs a new log.After the new log L ′ is constructed, each node broadcastsa signature for getDigest ( L ′ ) to all other nodes. Eachnode then collects signatures from f other participantsthat have also constructed the same log and publishes italong with all signatures.In the following subsection, we describe in detail theC ONSENSUS subroutine, which is the main building block ofthe protocol, and explain how the consensus phase achievesinteractive consistency. The log needs not be sent along with the signature, as the other correctnodes will have constructed the same log L ′ . lgorithm 2 Consensus phase procedure C ONSENSUS ( p, X ) Code for node i : if i = p then ⊲ node is primary broadcast h X, p | i i return X else ⊲ node is responder P ← ∅ ⊲ witnessed values d ← ∅ ⊲ decision value for rounds r = 1 , . . . , f + 1 do M ← receive messages M ′ ← {h x, p | W i ∈ M. p ∈ W ∧ | W | ≥ r } P ′ ← { x. ∃ W. h x, p | W i ∈ M ′ } if P ′ \ P = ∅ then if | P ∪ P ′ | = 1 then d ← the only element of P ′ else d ← ∅ S ← ∅ for x ∈ P ′ \ P do h x, p | W i ← any element of M ′ S ← S ∪ {h x, p | W ∪ { i }i} multicast S to other responders P ← P ′ return d C. Consensus Phase
For the sake of abstraction, we will refer to a set of entries asa value and note that C
ONSENSUS is not limited to a specifictype of value, but can be used more generally. The algorithmmust satisfy the following two properties:(C1) All correct nodes return the same value.(C2) If the primary p is correct, all correct nodes return theprimary’s input value X .If these properties hold for all parallel executions of C ON - SENSUS , it follows that the consensus phase satisfies (IC1)and (IC2). The main challenge in designing the algorithm doesnot lie in securing executions in which the primary is correct,but in preventing a faulty primary from causing disagreementbetween correct nodes. This is especially difficult because theprimary may be actively malicious and colluding with all otherfaulty nodes.Informally, the algorithm works as follows. Only the pri-mary can propose new values, which it does by signing them.When a process receives such a value, it forwards the valueto other nodes and testifies to witnessing it by appending itsown signature. This is necessary for two reasons: (a) to informnodes about the primary’s value in case the message in theinitial round was lost, and (b) to detect equivocation by theprimary (i.e., sending different values in an attempt to createdifferent views).However, this technique alone is not enough to preventattacks: A malicious primary could collude with respondersand attempt to introduce a new value to some nodes in the last round, so the receivers will not have time to share it. Toprevent this, correct nodes also apply the following policy: Inround r , received values that have fewer than r witnesses arediscarded. Any values sent in the last round must therefore besigned by f + 1 nodes in order to be accepted, i.e., at leastone correct node must have witnessed it previously.After introducing the main concepts used in C ONSENSUS ,we now explain the subroutine in more detail and refer to thespecification in Algorithm 2. In most situations, the executionof C
ONSENSUS is simple and requires only two rounds ofcommunication. Such a normal case is depicted in Figure 2.
1) Primary:
The procedure for the primary p is simple: p broadcasts its initial value in the first round and returns. A termof the form h x, p | i i represents the assertion “node i testifiesto having witnessed value x from primary p ”. It is importantto include p in the signature, as this prevents replay attacksusing signatures received in different threads.
2) Responders:
The other nodes, which we call responders,keep in their state the set P containing all values witnessedfrom the primary, as well as their current decision value d .In the following, we describe the behavior of a respondernode with identifier i . During each round r , node i listensfor any messages of the form h x, p | W i and stores them inthe set M (discarding any invalid signatures). After a timeout, i advances to line 10 and selects from M all messages thatare (a) signed by the primary, and (b) witnessed by a totalof r nodes (including the primary). From these messages M ′ ,all distinct values are extracted and stored in P ′ , which nowcontains all values that have been accepted this round.If i has accepted any new values this round (i.e., P ′ \ P = ∅ ), it adjusts its decision based on the values it has acceptedso far (line 13). If | P ∪ P ′ | = 1 holds, i stores the new valuein P ′ as its current decision. If there are more values in thetwo sets, i has proof of misbehavior by p because a correctprimary will never send more than one value. In this case, i decides on the default value, which is the empty set.In both of these two cases, i needs to testify to havingwitnessed the new values in P ′ \ P . For this purpose, amessage m is assembled containing all new values with thecorresponding signatures (lines 17–20). For each new value x , i adds a signature of its own to the message. m is thenbroadcast to all other responders.Finally, after all f + 1 rounds have completed, i concludesthe consensus phase by returning its current decision d .V. A NALYSIS
A. Informal Security Analysis
The goal of the following informal arguments is to pro-vide intuition as to why our protocol satisfies the claimedproperties. A more rigorous treatment using formal proofs ispresented in Section VII.The security arguments are structured such that the proper-ties of C
ONSENSUS are discussed first. By building on these,we argue that L
OGRES achieves agreement, completeness, andliveness. This is also the structure that our formal proof fol-lows, as it allows for a more modular argument by treating the4 { x } { x } { x } { x } { x } Round Round Round h x, p | p i h x, p | { p, }i Fig. 2. The normal case of an execution of C
ONSENSUS with a correctprimary p . There are n = 5 nodes, of which f = 2 may be faulty, whichrequires f + 1 = 3 rounds of communication. In the first round, p broadcastsits initial value, and in the second round, each responder decides on this valueand forwards it to others. No further communication is required in the lastround. two algorithms separately, with the properties (C1) and (C2)as the only interface between them.
1) Consensus phase:
To show the validity property (C2),we can assume that the primary p is correct. In this case, allthe primary does is broadcast its value X to all responders anddecide on it. Since channels between correct nodes are reliableby Assumption (A2), all correct responders will receive X .Any other values will not be accepted by correct nodes, sincethe algorithm discards all values that are not signed by theprimary. Therefore, all correct nodes will decide on X andvalidity is satisfied. This also implies the agreement property(C1) for the case of a correct primary.We now show that agreement is also guaranteed in the pres-ence of a faulty primary, which follows from two observations:(a) When a correct node accepts a new value, all other correctnodes will have accepted it in the next round. This is impliedby a combination of (A2) with the fact that nodes sign andshare all new values they accept. (b) No new values can beintroduced into the system in the last round. This follows fromthe property that in this round, values require f + 1 signaturesto be accepted, so any new value must have been signed (andtherefore shared with all other responders) by a correct nodein an earlier round.
2) Logres:
Property (C1) ensures that all correct nodes ob-tain the same set X . Along with the fact that makeLog ( L, X ) is a deterministic function, i.e., imposes some known order onthe entries in X , this implies that all correct nodes produce thesame log L ′ . There are at least n − f ≥ f + 1 correct nodes byassumption (A1), and because (A2) implies reliable channelsbetween them, each of the correct nodes obtains signaturesfor L ′ from at least f + 1 nodes (including themselves). Thissatisfies the Liveness property.In order to satisfy the Agreement property, we not onlyneed to show that all correct nodes produce the same log,but also that there can exist no other log. This follows fromthe assumption that there are at most f faulty nodes, and since f + 1 are required for a valid log, this implies that theyneed the signature of at least one correct node to forge a log.However, since all correct nodes produce the same log, thefaulty processes will not be able to create a different log.Finally, we give some intuition about Completeness. Let i be a correct node to which the client sent its request. Then, i will include the requested entry x in its proposal X , which isgiven as input to the C ONSENSUS subroutine. Using (C2), wehave that x is contained in the set X i . When combined withthe agreement properties above, this implies that all correctnodes will include x in their new log.
3) Denial of Service:
A common problem with BFT al-gorithms is that malicious actors may attempt to introduce alarge number of values into the system in order to decreasethe effective throughput of legitimate requests. Concretely, anattacker may send an excessive number of requests to a nodesuch that it is not able to broadcast their value to all receiverswithin the duration of a round.Mitigation against such attacks is specific to the applicationdeployed on top of the protocol and is thus considered outof scope, but we envision countermeasures like rate limiting,client request authentication and duplicate suppression. Theseare all common techniques to strengthen resilience of systemsto denial-of-service attacks.
B. Relation to Theoretical Bounds
Pease et al. [28] showed that Byzantine consensus for atmost f faults requires at least f + 1 rounds of communication.An intuition for this bound is as follows. A faulty node maysend different values to different nodes (i.e., perform a split-world attack, also called equivocation). This is not necessarilya problem, since correct nodes can communicate with eachother in the next round to try to reach the same state. However,we want to avoid that faulty nodes keep proposing new valuesad infinitum. Therefore, we must require that messages besigned by more nodes in further rounds (i.e., at least i nodesin round i ). Even with the above requirement, in any round i ≤ f , faulty nodes can still make one correct node accept anew value. That correct node must be able to show the newvalue to other correct nodes in the next round, and the onlything it can do is add its own signature. Therefore, in round i + 1 , nodes should accept messages with just i + 1 signatures.As a result, in the worst case, consensus will only be reachedin round f + 1 .Our protocol attains this lower bound because consensus isreached entirely within the consensus phase, which consists of f + 1 rounds. The last round (the signing phase) only servesto collect enough signatures to publish the list that the nodeshave agreed upon.Moreover, Pease et al. [28] proved that if signatures can beused, consensus is possible for any n > f . While we assumethat n > f , the consensus phase also reaches consensus forthe weaker bound of n > f (which can be shown by modifyingthe assumptions in our Isabelle/HOL proof). However, it isintuitively clear that the secure log replication problem is notsolvable in the case of n ≤ f : Consensus can be reached5mong the servers, but clients cannot reliably obtain the correctresult in the presence of adversaries. Since a client cannotdistinguish between faulty and correct nodes, the only way toensure that the faulty participants do not forge a result fromthe protocol is to require a quorum of nodes to verify it. Thisis the purpose of requiring θ = f + 1 signatures for a logto be valid. Therefore, the system always requires a correctmajority of nodes, and the n > f bound holds.We thus conclude that our protocol is optimal with respectto both the number of rounds and the number of faulty nodestolerated, considering the theoretical bounds on the problemof secure log replication.VI. E VALUATION
We have implemented a prototype of Logres in Go con-sisting of approximately lines of code and used it in aseries of experiments to evaluate the practical performanceof the protocol. The experiments were conducted in a settingsimilar to that of Certificate Transparency (CT). In our setup,a varying number of nodes coordinate to keep an identicallog of certificate entries. The workload is simulated usingrandomly generated strings of length equal to the average sizeof a CT log entry, which we determined to be approximately bytes. The period length (duration of a protocol run)was chosen as one minute. In order to determine if Logrescan support a realistic workload, we compare our results tothe average throughput of one of the largest CT logs to date,Google Argon 2019 [29].The nodes were run in a virtual network created usingMininet on a single machine with two -core Intel XeonE5-2680 processors running at .
70 GHz and with
32 GB ofRAM. In the virtual topology, each node was connected to theother hosts by a link with capacity
100 Mb / s . A. Scaling
In this experiment, we measure the throughput of the systemas the number of nodes increases, in order to evaluate thescalability of Logres. The results are displayed in Figure 3.We observe that performance diminishes as more nodes areadded, which can be explained by the fact that messages needto be broadcast to other nodes. The uplink capacity limitsthe amount of data that can be sent during a round, andif this capacity must be distributed among more nodes, themaximum possible throughput decreases. Despite these effects,we observe that our system is able to support the load of areal-world CT log with up to nodes.We used a constant number f = 2 of tolerated faultynodes in this experiment. This parameter constitutes a typicaltrade-off between performance and security (as discussed inSection VI-B). B. Malicious Nodes
The parameter f determines how many faults can be toler-ated and may be chosen up to a maximum of ⌈ n ⌉ − . How-ever, larger values of f are also detrimental to performance.Moreover, even though we have proven the protocol secure for Number of nodes n A v e r a g e t h r oughpu t ρ [ K B / s ] f = 2 Logres prototypeGoogle Argon 2019 logFig. 3. Average throughput of our Logres prototype with f = 2 , dependingon the number of replicas n . For comparison, we indicate the averagethroughput of the (non-replicated) Google Argon 2019 CT log server, whichprocesses , certificates per hour on average at the time of writing. Number of tolerated faults f A v e r a g e t h r oughpu t ρ [ K B / s ] n = 10 0 faulty nodes f malicious nodesGoogle Argon 2019 logFig. 4. The impact of increasing the parameter f for a fixed n in the casewhere all nodes are correct, compared to a simulation of malicious behavior. these choices of f , adversaries may still be able to deterioratethe performance of the system by causing communicationoverhead. The purpose of this experiment is to measure theimpact of these factors. For this purpose, we use a setup with n = 10 nodes, where measurements are conducted both withcorrect nodes and with a simulation of malicious behavior. Theresults are displayed in Figure 4.
1) Correct nodes:
In the absence of faulty nodes, theparameter f has two effects on performance: The consensusphase consists of f + 1 rounds, so the rounds become shorteras f is increased. This limits the total size of entries thatcan be proposed with the same network bandwidth. Secondly,the clients are also affected by this parameter. In order toensure that a request is not dropped by faulty nodes, a clientshould replicate it to f + 1 nodes, which amplifies the totaldata volume that needs to be exchanged during the protocolrun.
2) Malicious behavior:
The simulation of malicious be-havior is based on a number of observations. The first isthat in executions of C
ONSENSUS with a correct primary, anadversary cannot cause the correct nodes to deviate from theirregular behavior. This is because any value that is not signedby the primary is rejected immediately. Moreover, a malicious6
ABLE IT
IME REQUIRED FOR A REQUEST TO BE PROCESSED , DEPENDING ON THENUMBER OF TOLERATED FAULTS f , WITH n = 5 NODES . f Lower Bound Measured Latency0 .
34 ms 44 ms .
51 ms 66 ms .
68 ms 88 ms primary can be blocked immediately by the responders onceequivocation is detected, since the signatures created can beused to hold the primary accountable. Finally, we have dis-cussed in Section V-A3 how denial-of-service attacks shouldbe mitigated by the application running on top of Logres.From these observations and the properties proved for theprotocol, we can rule out many behavior patterns that allowthe correct nodes to block the adversaries. The attack strategywe implemented works as follows: The faulty nodes generatesome values, share them among each other and sign them.These values are then sent to the correct nodes in a laterround. Since the messages carry more signatures than usual,this causes some communication overhead at the correct nodes.In our implementation, signatures are bytes long andthus small in comparison to the large volume of values that areexchanged. Therefore, we do not expect this attack to have agreat impact. Our hypothesis is confirmed in Figure 4, whichshows no significant difference in average system throughputbetween the normal case and the attack scenario. We thereforeconclude that Logres performs well even in the presence ofmalicious participants that actively try to sabotage the protocolexecution.
C. Latency
Finally, we investigate the minimum latency attainable bythe protocol. For the previous experiments, the system wasconfigured with a period length of one minute, but a shorterinterval can be chosen for applications that are subject tostricter latency requirements.We will determine the minimum time that is required fora request to be processed, i.e., from the moment the requestis received by a server until it is added to a new version ofthe log. A lower bound on the round length can be computedusing the network parameters between nodes and the size ofrequests. In this experiment, we choose links with a latencyof
20 ms and a bandwidth
100 Mb / s . Again, we use values, which results in a minimum bound on round length ofapproximately .
17 ms (the time required for transmission ofa single value).To measure the actual latency of the system, the clientssubmit a single request for each period, and we decrease theround length successively until the minimum value (for whichthe requested value is still processed reliably) is reached. Weexpect the measurements to exhibit slightly larger values thanthe theoretical lower bound due to the computational overheaddue to message processing and signature verification. The results from the experiment are displayed in Table I for nodes and different values of f . The results show that ourimplementation is able to achieve small latencies, close to thetheoretical lower bound.VII. V ERIFICATION
Our verification is based on the Heard-Of model, which wasintroduced by Charron-Bost and Schiper [22]. We first providea brief overview of this model and how we extended it toaccount for signatures. Finally, this section contains a briefoverview of our results and experiences with constructing theproof. We refer to the appendix for more details about the pro-tocol formalization (Appendix A) and the proof methodology(Appendix B) we used in the process.
A. The Heard-Of Model
The Heard-Of model provides a framework for modelingconsensus algorithms in a lock-step model of the system,where communication is divided into rounds. As the originalmodel only accounted for benign faults, Biely et al. [12]extended it to capture malicious behavior. Our proofs are basedon an implementation of this model in Isabelle/HOL by Debratand Merz [11].In this context, we will refer to nodes more generally asprocesses. The set of n processes is represented by Π , and M denotes the set of possible messages. We define M ⊥ := M ∪{⊥} , where ⊥ stands for the absence of a message. Each i ∈ Π is associated with a process specification P i = ( S i , S i , S i , T i ) consisting of the following components: • S i : the set of i ’s states , and S i : the set of possible initialstates of process i . • S i : N × S i × Π → M : the message sending function ,where S i ( r, s, j ) denotes the message sent by i to j inround r , given that i is in state s . • T i : N × S i × M Π ⊥ × S i → { true , false } : the next-statepredicate , where T i ( r, s, µ, s ′ ) evaluates to true if andonly if s ′ is a state that i can reach from state s in round r , given the vector µ of messages that were received by i in round r .The collection of process specifications P i is called an algo-rithm on Π .A run of such an algorithm is defined, for each process i ,by a sequence of states s i , s i , . . . that satisfies the followingconditions: • s i is a valid initial state: s i ∈ S i • For each r ∈ N , it holds that the next-state predicate T i ( r, s ri , µ ri , s r +1 i ) is true for a message vector µ ri col-lected from the messages sent by the other processes.We now define µ ri , which not only depends on the specificationof the algorithm, but also on the communication assumptions.For each process i and each round r , we define two subsetsof Π : • HO ri : the heard-of set , which places the following con-straint on the message vector: j ∈ HO ri ⇐⇒ µ ri [ j ] = ⊥ i fromeach process in HO ri , which may or may not conform tothe process specification. • SHO ri : the safe heard-of set , the set of processes whosemessage arrives unchanged: j ∈ SHO ri ⇐⇒ µ ri [ j ] = S j ( r, s rj , i ) The message received by i is therefore guaranteed to bethe one given by the protocol specification for j . B. Extension to Signatures
The standard Byzantine fault model can be encoded bydefining a subset F ⊆ Π of Byzantine faulty nodes with | F | ≤ f and choosing the heard-of sets as SHO ri = Π \ F ⊆ HO ri .This corresponds to the assumption that the messages ofcorrect nodes arrive according to process specification, andthat faulty nodes may send any message in M or none at all.However, this property is generally too weak for a protocolthat relies on digital signatures. This cryptographic primitiveis commonly used under the assumption that signatures areunforgeable, which implies that faulty nodes must not beallowed to simply send arbitrary elements from the set M .Instead, we introduce a restriction on messages sent byfaulty nodes that encodes the unforgeable nature of signatures.While this could also be achieved by modeling Dolev-Yaointruder knowledge, we chose a simpler approach that requiresfewer modifications to the original model. Let Sig denote theset of signatures and for each process i , let σ i : Π Sig be afunction that given a message m , extracts the set of signaturescontained in m that are signed by i . Furthermore, Σ ri is theset of all signatures sent by i in round r : Σ ri := [ j ∈ Π σ i ( S i ( r, s ri , j )) From this, we define M r as the set of all messages that canbe constructed from signatures sent out until round r : M r := { m ∈ M : ∀ i ∈ C r . σ i ( m ) ⊆ r [ r ′ =0 Σ ri } where C r denotes the set of processes that have not failed yet,i.e., processes that have been contained in all secure heard-ofsets up to this round. Intuitively, this definition means that anagent that has stored all messages up to round r can use itsknowledge to construct any message in M r (e.g., includingsignatures received from correct nodes). Note that maliciousnodes are not in the set C r , and therefore signatures fromthem may appear in any messages. This allows for collusionbetween attackers, which may share their private keys overout-of-band channels.Using these definitions, we can finally define the restrictionon message vectors µ ri for all rounds r and processes i and j : µ ri [ j ] ∈ M r ∪ {⊥} In summary, our extension of the HO model encodes thestandard assumption that signatures are unforgeable, while stillallowing faulty processes to sign arbitrary values and colludingwith each other.
C. Verification Results
Our Isabelle/HOL verification of all security propertiesclaimed in this paper is available online [21]. It consists ofapproximately lines of code for the protocol model and lines of code for the proofs.The verification process revealed two flaws in previousversions of the protocol. The first was based on missingisolation between instances of the C
ONSENSUS subroutine,which are invoked in parallel by Logres. Since signatures werenot bound to a specific subroutine, a sophisticated attackercould have been able to replay legitimate signatures obtainedfrom one thread in another thread, thereby preventing nodesfrom creating a valid log. This vulnerability was fixed bybinding all signatures from a C
ONSENSUS routine to thecorresponding primary.In the second attack, an adversary could cause some nodesto exit from the consensus phase prematurely and thus againprevent the creation of a valid log.These insights from our proofs provide further anecdotalevidence towards the conclusion that proofs must be machine-checked to provide strong guarantees. The vulnerabilitiesdescribed above were not discovered during pen-and-paperconstruction of the proof, but only during the process ofmechanizing it. This was detected when the proof assistantrejected some proof steps that were believed to be correctafter informal reasoning.The lessons learnt in this process and the practices we arerecommending in Appendix B have also been identified inother verification projects [8, 30]. Since the advent of large-scale mechanized proofs, significant progress has been madein the direction of systematic proof engineering.VIII. R
ELATED W ORK
Numerous protocols have been designed to solve Byzantineagreement or BFT state-machine replication [1, 2, 3, 31, 32,33, 34, 35], and the Abstract framework proposed by Guer-raoui et al. [4] represents a notable step towards making thedevelopment of these protocols more modular and systematic.The problem of Byzantine agreement has been known since1980, when it was first introduced by Pease et al. [28] as the“Byzantine Generals Problem”. Dolev et al. [36] worked onsignature-based protocols to solve this problem and introducedthe principle of requiring values to have more signatures inlater rounds, which was also used by Biely et al. [12].To the best of our knowledge, there exists no full machine-checked proof of a state-of-the-art BFT protocol. The mostnotable steps in this direction have been proofs of safetyproperties for PBFT by Lamport [37] and more recently Rahliet al. [38]. Lamport used the TLA + language in conjunctionwith the TLAPS proof system, while Rahli et al. developedan extension of EventML in the theorem prover Coq to verifythe protocol. The main issue with both of these proofs is thatthey omit any liveness properties, which are commonly morechallenging to prove than safety properties. We stress thatthese properties are just as important to the security of the8rotocol, especially when deployed in a environment that de-mands a reliable system. Another approach to verifying PBFTis based on Event-B and Rodin, but makes the assumptionthat messages cannot be forged [39]. A proof in this modelcan therefore not provide any guarantees in the Byzantine faultmodel.More success has been achieved for consensus algorithmsthat tolerate benign (i.e., non-Byzantine) faults. In a break-through effort, the IronFleet project [10] fully verified a Paxos-based distributed system and its implementation using theDafny verifier. The Raft [40] protocol is designed to achievesimilar goals as the classic Paxos algorithm, but with themain design goal of understandability. Many properties of thesystem and its implementation have been formally verified inthe Verdi framework for the Coq proof assistant, but livenessproperties were not proved [8].In the only instance of a complete proof for a BFT protocolthat we are aware of, Debrat and Merz [11] verified two simpleByzantine consensus protocols in the Heard-Of model usingthe Isabelle/HOL theorem prover. Our work builds on theseresults, showing that similar methods can be applied to a morepractical protocol that makes use of digital signatures.Model checking is another method of verifying propertiesof protocols that can be used as an alternative to formalproofs. A model checker explores the state space of a systemspecification, which can be very useful for exposing designflaws without requiring a high amount of manual effort.However, this technique is ineffective at providing assurance ofsecurity for a system due to the constraints that combinatorialexploration places on the complexity of the model. Zieli´nski[41] applied model checking to Byzantine consensus protocols,but due to the tremendous size of the state space caused byByzantine faults, the scope of the verification was limited toonly a single round of communication.In order to address problems related to logging schemessuch as Certificate Transparency, Syta et al. [20] proposedto leverage multisignatures and distribute trust among manycollective authorities or “cothorities”. However, the rathercomplex protocols used by cothorities are proposed withoutany formal verification.IX. C ONCLUSION
This paper presents Logres, a secure log replication proto-col. To the best of our knowledge, Logres is the first practicalBFT protocol to be accompanied by a complete machine-checked security proof. This makes the protocol particularlysuitable for security-critical applications that demand highresilience and reliability.We have demonstrated that our protocol can achieve suf-ficient throughput for practical applications such as certifi-cate logging, even in the presence of actively maliciousparticipants. Concretely, Logres is able to support the loadof the Certificate Transparency system for up to nodes.Our system could be further improved by using a hash-treebased data store such as Trillian [42]. A threshold signature scheme [43] would also reduce the size of the signed logsproduced by the protocol.Our extension of the Heard-Of model to signatures canbe used in formal verification of other consensus protocolsthat rely on this cryptographic primitive. Future work couldcapture other primitives such as message authentication codesand hash functions. Other interesting paths for future workinclude producing a verified implementation of Logres andconsidering application-specific optimizations.X. A CKNOWLEDGMENTS
We gratefully acknowledge support from ETH Zurich, andfrom the Zurich Information Security and Privacy Center(ZISC). Furthermore, we thank Kartik Nayak, Dahlia Malkhi,and David Kozhaya for their valuable feedback.R
EFERENCES [1] M. Castro and B. Liskov, “Practical Byzantine faulttolerance and proactive recovery,”
ACM Transactions onComputer Systems , vol. 20, no. 4, 2002.[2] R. Kotla, L. Alvisi, M. Dahlin, A. Clement, and E. Wong,“Zyzzyva: speculative Byzantine fault tolerance,” in
Pro-ceedings of the ACM Symposium on Operating SystemsPrinciples (SOSP) , 2007.[3] A. Clement, E. L. Wong, L. Alvisi, M. Dahlin, andM. Marchetti, “Making Byzantine fault tolerant systemstolerate Byzantine faults,” in
NSDI , 2009.[4] R. Guerraoui, N. Kneˇzevi´c, V. Qu´ema, and M. Vukoli´c,“The next 700 BFT protocols,” in
Proceedings of theEuropean Conference on Computer Systems (EuroSys) ,2010.[5] A. Bessani, J. Sousa, and E. E. P. Alchieri, “State ma-chine replication for the masses with BFT-SMART,” in
Proceedings of the IEEE/IFIP International Conferenceon Dependable Systems and Networks (DSN) , Jun. 2014.[6] L. Lamport, R. Shostak, and M. Pease, “The Byzantinegenerals problem,”
ACM Transactions on ProgrammingLanguages and Systems , vol. 4, no. 3, Jul. 1982.[7] P. Zave, “Using lightweight modeling to understandChord,”
ACM SIGCOMM Computer Communication Re-view , vol. 42, no. 2, Mar. 2012.[8] D. Woos, J. R. Wilcox, S. Anton, Z. Tatlock, M. D. Ernst,and T. Anderson, “Planning for change in a formal verifi-cation of the Raft consensus protocol,” in
Proceedings ofthe ACM Conference on Certified Programs and Proofs(CPP) , 2016.[9] L. Lamport, “Paxos made simple,”
ACM Sigact News ,vol. 32, no. 4, 2001.[10] C. Hawblitzel, J. Howell, M. Kapritsos, J. R. Lorch,B. Parno, M. L. Roberts, S. Setty, and B. Zill, “IronFleet:proving practical distributed systems correct,” in
Pro-ceedings of the ACM Symposium on Operating SystemsPrinciples (SOSP) , 2015.[11] H. Debrat and S. Merz, “Verifying fault-tolerantdistributed algorithms in the Heard-Of model,”
Archive f Formal Proofs , 2012. [Online]. Available: https://isa-afp.org/entries/Heard Of.html[12] M. Biely, B. Charron-Bost, A. Gaillard, M. Hutle,A. Schiper, and J. Widder, “Tolerating corrupted com-munication,” in Proceedings of the ACM Symposium onPrinciples of Distributed Computing , 2007.[13] T. Nipkow, M. Wenzel, and L. C. Paulson, Eds.,
Is-abelle/HOL . Springer Berlin Heidelberg, 2002.[14] B. Laurie, A. Langley, and E. Kasper, “Certificate trans-parency,” RFC 6962, Jun. 2013.[15] D. Basin, C. Cremers, T. H.-J. Kim, A. Perrig, R. Sasse,and P. Szalachowski, “ARPKI: attack resilient public-keyinfrastructure,” in
Proceedings of the ACM Conference onComputer and Communications Security (CCS) , 2014.[16] D. A. Gritzalis, “Principles and requirements for a securee-voting system,”
Computers and Security , vol. 21, no. 6,2002.[17] N. Chondros, B. Zhang, T. Zacharias, P. Diamantopoulos,S. Maneas, C. Patsonakis, A. Delis, A. Kiayias, andM. Roussopoulos, “A distributed, end-to-end verifiable,internet voting system,” arXiv:1507.06812 , 2015.[18] H. Massias, X. S. Avila, and J.-J. Quisquater, “Design ofa secure timestamping service with minimal trust require-ment,” in
Proceedings of the Symposium on InformationTheory in the Benelux , 1999.[19] B. Gipp, N. Meuschke, and A. Gernandt, “Decentralizedtrusted timestamping using the crypto currency Bitcoin,” arXiv:1502.04015 , 2015.[20] E. Syta, I. Tamas, D. Visher, D. I. Wolinsky, and B. Ford,“Decentralizing authorities into scalable strongest-linkcothorities,”
CoRR
Distributed Computing , Apr. 2009.[23] A. Eijdenberg, B. Laurie, and A. Cutter, “Verifiabledata structures,” https://github.com/google/trillian/blob/master/docs/papers/VerifiableDataStructures.pdf.[24] R. Dahlberg, T. Pulls, and R. Peeters, “Efficient sparseMerkle trees,” in
Proceedings of the 21st Nordic Confer-ence on Secure IT Systems (NordSec) , 2016.[25] L. Chuat, P. Szalachowski, A. Perrig, B. Laurie, andE. Messeri, “Efficient gossip protocols for verifyingthe consistency of certificate logs,” in
Proceedings ofthe IEEE Conference on Communications and NetworkSecurity (CNS) , September 2015.[26] S. Eskandarian, E. Messeri, J. Bonneau, and D. Boneh,“Certificate Transparency with privacy,” in
Proceedingson Privacy Enhancing Technologies , 2017.[27] B. Li, J. Lin, F. Li, Q. Wang, Q. Li, J. Jing, and C. Wang,“Certificate Transparency in the wild: Exploring thereliability of monitors,” in
Proceedings of the 2019 ACMSIGSAC Conference on Computer and CommunicationsSecurity (CCS) , 2019. [28] M. Pease, R. Shostak, and L. Lamport, “Reaching agree-ment in the presence of faults,”
Journal of the ACM ,vol. 27, no. 2, Apr. 1980.[29] Cloudflare, “Merkle Town,” https://ct.cloudflare.com.[30] G. Klein, M. Norrish, T. Sewell, H. Tuch, S. Winwood,K. Elphinstone, G. Heiser, J. Andronick, D. Cock, P. Der-rin, D. Elkaduwe, K. Engelhardt, and R. Kolanski, “seL4:formal verification of an OS kernel,” in
Proceedingsof the ACM SIGOPS symposium on Operating systemsprinciples (SOSP) . ACM Press, 2009.[31] M. Abd-El-Malek, G. R. Ganger, G. R. Goodson,M. K. Reiter, and J. J. Wylie, “Fault-scalable Byzantinefault-tolerant services,”
ACM Operating Systems Review ,vol. 39, no. 5, 2005.[32] J. Cowling, D. Myers, B. Liskov, R. Rodrigues, andL. Shrira, “HQ replication: a hybrid quorum protocol forByzantine fault tolerance,” in
Proceedings of the Sympo-sium on Operating Systems Design and Implementation(OSDI) , ser. OSDI ’06, Berkeley, CA, USA, 2006.[33] I. Abraham, D. Malkhi, K. Nayak, L. Ren, and M. Yin,“Sync hotstuff: Simple and practical synchronous statemachine replication,” Cryptology ePrint Archive, 2019.[34] J. Katz and C.-Y. Koo, “On expected constant-round pro-tocols for Byzantine agreement,” in
Annual InternationalCryptology Conference , 2006.[35] I. Abraham, S. Devadas, D. Dolev, K. Nayak, and L. Ren,“Synchronous Byzantine agreement with expected O(1)rounds, expected O( n ) communication, and optimalresilience,” in International Conference on FinancialCryptography and Data Security , 2019.[36] D. Dolev, M. J. Fischer, R. Fowler, N. A. Lynch, andH. R. Strong, “An efficient algorithm for Byzantineagreement without authentication,” in
Information andControl , 1982.[37] L. Lamport, “Byzantizing Paxos by refinement,” in
Lec-ture Notes in Computer Science , 2011.[38] V. Rahli, I. Vukotic, M. Vlp, and P. Esteves-Verissimo,“Velisarios: byzantine fault-tolerant protocols poweredby Coq,” in
Programming Languages and Systems , 2018.[39] R. Krenick and M. Ulbrich, “Deductive verification ofa Byzantine agreement protocol,” Karlsruhe Institute ofTechnology, Department of Computer Science, Tech.Rep., 2010.[40] D. Ongaro and J. K. Ousterhout, “In search of anunderstandable consensus algorithm,” in
USENIX ATC ,2014.[41] P. Zieli´nski, “Automatic verification and discovery ofByzantine consensus protocols,” in
Proceedings of theIEEE/IFIP International Conference on Dependable Sys-tems and Networks (DSN) , Jun. 2007.[42] A. Eijdenberg, B. Laurie, and A. Cutter, “Verifiable datastructures,” Google Research, Tech. Rep., 2015.[43] V. Shoup, “Practical threshold signatures,” in
Advancesin Cryptology (EUROCRYPT) , 2000.10
PPENDIX
A. Protocol Formalization
In the following, we show how Logres can be specifiedwithin the extension of the Heard-Of model presented inSection VII-B and provide formal specifications of the securityproperties stated in Section III.There are two types of messages that are exchanged betweennodes in Logres:1) The first and most common one is used during theconsensus phase and contains a list of value sets. Let X denote the domain of entries, and let V be the setof signatures over sets of values from X , which can bemodeled formally using the definition V := Π × (Π × X ) (consisting of the author, the identifier of the primary andthe value to be signed). We call these signatures votes .The set of possible messages of this type is then givenby M := Π × V , i.e., it carries a (possibly empty) setof votes for each primary node.2) During the signing phase, each node sends out a singlesignature over a log. We refer to the domain of logs overentries from X as L X and model the signatures over logsas M := Π × L X .These two message domains are combined in the set M := M ∪ M .Next, we specify the node states. Since every thread of theC ONSENSUS subroutine requires its own state, we first definethe possible thread states as T := 2 X × (2 X ) × V . Thiscorresponds to the definitions in Algorithm 2, which consistof (a) the current decision d ∈ X , (b) the sets of values P ∈ (2 X ) witnessed from the primary, and (c) the votes M ′ ∈ V from the current round. A node state stores sucha thread state for each primary in Π , as well as the set ofentries collected from clients, the current version of the logand some signatures (from the set Σ ) for it: S i := L threads ∈ Π × T , entries ∈ X , log ∈ L X , sigs ∈ M M for i ∈ Π To simplify the notation, this is specified as a set of records,which are tuples whose components are named. We use thesyntax s. f to access the field with name f of a state s .The initial states are defined as follows: S i := L threads = Π × { ( ∅ , ∅ , ∅ ) } ) , entries = 2 X , log = { L } , sigs = ∅ M for i ∈ Π where L is some initial log that is the same for all nodes.For the sake of brevity, we omit definitions such as the mes-sage sending functions and state transition predicates. Theseare contained in the Isabelle theories, which are availableonline [21] and were used in constructing the proofs. Using the above definitions, it is now possible to formalizethe security properties of the protocol (defined in Section III): Agreement
All valid logs created during a run of the protocolmust be equal. ∀ i, j ∈ Π \ F. s f +2 i . log = s f +2 j . log Since the protocol consists of f + 2 rounds, s f +2 i is thefinal state of the protocol run for a node i . Completeness
If an entry is submitted by a client to a correctnode, the node will include it in its next log produced. ∀ i ∈ Π \ F. s i . entries ⊆ s f +2 i . log Clients and their requests are not modeled explicitly.Instead, each node uses the entries field to store allreceived requests. We can therefore express the propertyas follows: all requested entries that a node has receivedbefore the end of the collection phase (i.e., s i . entries )will be included in the log at the end of the protocol run. Liveness
A run of the protocol must always produce a newvalid log for every correct node. ∀ i ∈ Π \ F. | s f +2 i . sigs | ≥ f + 1 A log is considered valid if it is signed by at least f + 1 distinct nodes. B. Proofs in Isabelle/HOL1) Using a Proof Assistant:
Proof assistants enable theconstruction of complex models and machine-checked proofs.Their greatest advantage over other verifications tools is gen-erality: they can be applied to practically any system. This isdone by constructing proofs that are analogous to handwrittenproofs, but provide much higher assurance since the proofassistant verifies its correctness, checking that each step ofthe proof is sound. Isabelle/HOL is a particular instance ofthis type of program, and its effectiveness has been provenin large-scale verification efforts such as that of the seL4microkernel [30].In order to provide some intuition about how proofs areconstructed with the assistance of Isabelle/HOL, we show thecode that is used to prove a simple statement about set theory,which is also used during our verification of Logres. It statesthat for any two sets A and B that are subsets of a finitedomain C , if the sum of their cardinalities is greater than thatof the domain, then A and B must overlap.Isabelle can encode statements in higher-order logic (HOL),using the functional programming language ML, which is en-riched by a large variety of standard mathematical definitions.In Isabelle’s meta-language, the lemma stated above can beformalized as follows: lemma overlap : assumes finC : finite C and subset : A ⊆ C ∧ B ⊆ C and cardAB : card A + card B > card C shows A ∩ B = {} assumes , and , shows are part of Isabelle’smeta-language. They identify the premises and conclusionsof the lemma, which can be named as in this example toimprove readability of the code. Isabelle supports reasoningusing declarative proofs in its proof language Isar, whichclosely resembles the structure of handwritten proofs. More-over, Isabelle provides a rich library of common lemmas thatcan be used in proofs.For this lemma, we can use a proof by contradiction. First,we assume that A ∩ B = {} , from which we can deduce twocontradicting results: • | A ∪ B | > | C | . Given that A and B do not intersect, | A | + | B | = | A ∪ B | holds. This follows from the built-inrule card - Un - disjoint , which is not defined here for thesake of brevity. • | A ∪ B | ≤ | C | . This follows from the fact that since A and B are both subsets of C , it holds that A ∪ B ⊆ C .The built-in rule card - mono is used to convert this intothe equation about cardinalities.From these two statements, we can show False , which is acontradiction. The code for the complete proof in Isabelle isshown below: proofassume nonempty : A ∩ B = {} from finC subset have finite A ∧ finite B using finite-subset by auto with nonempty cardAB have card ( A ∪ B ) > card C by ( simp add : card-Un-disjoint ) moreover have card ( A ∪ B ) ≤ card C using subset finC by ( simp add : card-mono ) ultimately show False by simp qed The detailed syntax and semantics of the proof are out ofscope for this brief overview of Isabelle, but thanks to thestructure that resembles the language of handwritten proofs,the steps are fairly comprehensible. For each step of the proof,an automated proof strategy (such as auto , simp , . . . ) is givenusing a by keyword, providing a hint to the proof assistantabout how the step can be automatically verified. Once themachine is able to verify all individual steps, the lemma hasbeen proved.This particular lemma was chosen as an example due to itssimplicity, but is also a part of our proofs, where we use itto show that a set of f + 1 nodes always contains at leastone correct node. Our Isabelle theory contains a total of lemmas, most of which are more complex to express and provethan the one shown above. However, this example serves asan overview of what to look for when performing high-levelinspection of the proofs.
2) Proof Methodology:
In the following, we provide a high-level description of the process of constructing our proofs for Logres. Initially, we formalized the protocol specification(given by pseudocode) in the HO model and specified thedesired properties formally. We have found it an effectivestrategy for the construction of a sophisticated proof to proceedalong the following basic steps:1) Write down a sequence of informal arguments for why theproperties hold (similar to our analysis in Section V-A).The goal of this step is to identify key intermediatelemmas around which the proof will be constructed.2) Formalize the intermediate lemmas from the previous stepto obtain a rough outline in the proof assistant. Insertplaceholders where the proof is incomplete (e.g., usingthe sorry keyword in Isabelle) to enable the assistant tocheck this outline.3) Attempt to complete the unfinished proof steps, usingexisting lemmas and creating new ones where required.Repeat this step until all statements are proved.This strategy is similar to a backward search technique, wherethe goal is inspected first and lemmas are created wheneverthey are needed to solve open goals.Naturally, this description of the process is fairly idealized,as it is often necessary in practice to modify the originaloutline or even the protocol itself upon discovering that certainsteps of the proof cannot be completed as expected. This is acommon occurrence, as it is extremely difficult for a humanto anticipate the precise outline of a proof before carrying outthe steps in detail.It is therefore necessary to ensure that the proofs are robustto changes and highly automated, such that a small changein some part of the proof does not require rewriting all ofthe remaining steps. For this purpose, we employ the designconcept of modularization, which is standard folklore in thesoftware engineering domain thanks to its benefits in creatingmaintainable code. In the case of a formal proof, this conceptimplies that the interfaces at which certain components of themodel interact must be specified clearly and kept as abstractas possible. Isabelle provides useful concepts for this, called locales and definitions , which fulfill a similar role as abstractinterfaces in software engineering. Well-designed modularproofs enable changing details of the protocol or lemmaswithout completely rewriting the proof.An example of how our proof utilizes this concept isthe C