Efficient Byzantine Fault Tolerance using Trusted Execution Environment: Preventing Equivocation is only the Beginning
Jiashuo Zhang, Jianbo Gao, Ke Wang, Zhenhao Wu, Ying Lan, Zhi Guan, Zhong Chen
TTBFT: Understandable and Efficient Byzantine FaultTolerance using Trusted Execution Environment
Jiashuo Zhang ∗† , Jianbo Gao ∗† , Ke Wang ∗† , Zhenhao Wu ∗† , Ying Lan ‡ , Zhi Guan §¶ , Zhong Chen ∗†¶∗ Department of Computer Science and Technology, EECS , Peking University , Beijing, China † Key Laboratory of High Confidence Software Technologies (Peking University), MoE , Beijing, China ‡ School of Electronic and Computer Engineering , Peking University , Shenzhen, China § National Engineering Research Center for Software Engineering , Peking University , Beijing, China ¶ Corresponding Author{zhangjiashuo, gaojianbo, wangk, zhenhaowu, ly3996, guan, zhongchen}@pku.edu.cn
Abstract —While the requirements for reliability increasingrapidly in distributed systems, byzantine fault tolerance protocolshave attracted more and more attention for the ability to toleratearbitrary failures. Although traditional BFT protocols haveachieved significant performance improvements after decades ofresearch, they still cannot meet the increasing scalability require-ments. Thanks to the recent development of trusted executionenvironment, researchers have opportunities to improve BFTprotocols with hardware assistance, including increasing the ratioof byzantine nodes that the system can tolerate and improvingthe scalability of the system. However, existing TEE-based BFTprotocols are mostly designed via improving traditional BFTprotocols and using complex operations to overcome securityissues introduced by TEE, which makes the protocols difficultto understand and implement. In fact, after introducing TEE,many byzantine failures can be eliminated naturally so thatthe adversary assumption of TEE-based BFT is more similarto CFT rather than traditional BFT. Therefore, it would bebetter to design a TEE-based BFT protocol on the basis of CFTprotocols to inherit the advantages of CFT, i.e., understandableand efficient. In this paper, we summarize the key differencesbetween TEE-based BFT and CFT protocols and propose fourprinciples to help bridge the gap between TEE-based BFT andCFT. Based on these principles, we present TBFT, a novel TEE-based BFT protocol with inspirations from CFT protocols, whichis simple and understandable enough to be applied in practice.Besides, we make comprehensive improvements to TBFT for bothperformance and security, including pipeline mechanisms, TEE-assisted secret sharing, and trusted leader election. Our evalu-ation shows that TBFT has better performance and scalabilitycompared to previous protocols.
Index Terms —State Machine Replication, Byzantine FaultTolerance, Trusted Execution Environment
I. I
NTRODUCTION
In the past few decades, byzantine fault tolerance (BFT)has mainly been considered of theoretical interest. Because ofthe large overhead to tolerate byzantine faults, the applicationof BFT protocols in large-scale systems is not as popular asthe crash fault tolerance (CFT) protocol However, the rapid de-velopment of consortium blockchain has made the demand forBFT protocols higher. As consensus mechanisms in blockchainsystems, BFT protocols need to reach consensus among a largenumber of replicas while providing high performance. Unfor-tunately, due to their poor performance and scalability, existing BFT protocols have become the performance bottleneck ofblockchain systems [1] [2].Strong adversary assumptions always cause strict restric-tions. Tolerating byzantine faults requires redundant repli-cas, extra inter-replica communication, and expensive crypto-graphic primitives. After decades of research, improving BFTprotocols is getting harder and harder. For instance, Hotstuff[3] is one of the most recent works, which still needs 3f+1replicas to run a three-phase protocol to reach consensus andleverages expensive cryptographic primitives including multi-signature. Many theoretical results [4] [5] [6] also show thattraditional BFT protocols have already been too close to theirtheoretical limits before achieving satisfying performance andscalability.Trusted Execution Environment(TEE) brings new opportu-nities. With hardware assistance, systems can protect sensitivedata and guarantee the integrity of code execution. UtilizingTEE, BFT protocols [7] [8] [9] can achieve significant im-provements or even breaking theoretical limits such as reducethe minimum number of replicas from 3f+1 to 2f+1. Moreover,to provide confidentiality, TEE has also been widely used indistributed systems, especially in blockchain platforms [10][11]. For those systems, it is natural and compatible to useTEE-based BFT to get better performance and scalability.Existing TEE-based BFT protocols focus on the same ques-tion, i.e. what TEE can help when byzantine behaviors occur.For instance, they all use a trusted monotonic counter providedby TEE to prevent leader equivocation. However, while theprotocols improve the efficiency of tolerating byzantine faultswith hardware assistance, they also have to face new securityissues introduced by TEE, such as a malicious host can arbi-trarily manipulate TEE scheduling and I/O. Consequently, theprotocols are patched on the basis of previous BFT protocols,which makes them complex and difficult to understand. Withthis drawback, they have hardly been applied to practicalimplementations.In this paper, we start from a different observation thatTEE may crash but will never give malicious executionresults, which makes TEE-based BFT is more similar to CFTrather than BFT. Since there have been several practical andunderstandable CFT protocols [12] [13] [14], a TEE-based a r X i v : . [ c s . D C ] F e b FT with inspirations from CFT protocols will be much morenatural and understandable. The key problem is to bridge thegap between TEE-based BFT and CFT. Despite the existenceof TEE, a byzantine host can still terminate TEE at any mo-ment, schedule TEE arbitrarily, or drop, replay, delay, reorder,modify the I/O messages of TEE. To solve the problem,we summarize new security issues introduced by TEE andpropose four design principles. Based on these principles, wepresent a TEE-based BFT protocol called TBFT. We also makecomprehensive improvements to TBFT for both performanceand security, including pipeline mechanisms, TEE-assistedsecret sharing, and unpredictable leader election.We summarize our contributions as follows:1) We propose an understandable and efficient TEE-basedBFT called TBFT with inspirations from CFT. We findfour extra security issues faced by TEE-based BFT com-pared with CFT and propose four principles to solvethem. TBFT is more understandable and simpler com-pared to existing TEE-based BFT protocols. To our bestknowledge, TBFT is the first TEE-based protocol with O ( n ) message complexity in both normal case and view-change. It can serve as a foundation for implementingpractical distributed systems like blockchain.2) We make comprehensive optimizations to TBFT. Weintroduce pipeline in TBFT to improve throughput. TBFTis the first TEE-based BFT with pipeline mechanisms. Wealso replace multi-signature in TBFT with TEE-assistedsecret sharing to reduce the computation overhead.3) We introduce a novel and efficient TEE-based DistributedVerifiable Random Function(DVRF) for verifiable andrandom leader election, which is one of the most con-cerning security issues for BFT recently. Compared toother complex DVRF solutions, this mechanism is morenatural and efficient.4) We have implemented and open-sourced TBFT. Theexperimental results and systematic analysis show thatTBFT has better performance and scalability comparedto existing protocols.The remaining of this paper is organized as follows: SectionII provides some preliminaries about BFT and TEE. In SectionIII, we describe new security issues faced by TEE-based BFTand propose four principles to bridge the gap. Section IV andSection V introduce the core protocol of TBFT and the electionmechanism in TBFT separately. In Section VI, we prove thesafety and liveness of TBFT. In Section VII, we evaluatesTBFT and compare it with several previous protocols. InSection VIII, we discuss about previous work related to TBFT.Finally, Section IX concludes the paper.II. B ACKGROUND
A. Byzantine Fault Tolerance(BFT)
Byzantine fault tolerance is the ability of a distributedcomputing system to tolerate byzantine failures of replicas.Byzantine means replicas can deviate from the protocol ar-bitrarily, e.g. , crash or give malicious responses. It is often mentioned in the context of State Machine Replication(SMR)which is a distributed system where the state is replicatedamong different replicas. An SMR BFT protocol needs toguarantee safety and liveness when a subset of replicas arepotentially byzantine replicas, i.e. , guarantee that all non-faultyreplicas executed clients’ requests in the same order(Safety)and clients will eventually receive execution results of theirrequests(Liveness) [15].
B. Trusted Execution Environment(TEE)
Trusted execution environment(TEE) is an isolated execu-tion environment to protect the confidentiality and integrityof the code and data. With hardware assistance, TEE canprotect sensitive data from the host and privileged system, e.g. ,operating system or hypervisor, and guarantee the correctnessof the execution. A TEE may crash, but will never give amalicious result.For TEE-based applications, a small Trusted ComputingBase(TCB) [16] is essential. This is because existing TEEs, e.g. , Intel SGX [17], ARM TrustZone [18], TPM [19] onlyhave limited resources, hence it is unrealistic to put the wholeprotocol, e.g. , existing CFT protocols, into TEE. Besides, alarge TCB means a large attack interface. Ideally, TCB shouldbe small enough to support formal verification.Intel Software Guard Extensions(SGX) [17] is one of themost popular TEEs. It is pervasive nowadays and has beendeployed on many commodity platforms, e.g. , servers, and PCs.It provides a secure container called enclave. Even if the hostis malicious, the enclave can protect the sensitive data andguarantee the correctness of the execution. Intel SGX supportsremote attestation(RA) [20]. A third party can verify whetherthe program executed inside the enclave is genuine, i.e. , therunning program and the execution result are correct have notbeen modified.
C. Prevent Equivocation Using TEE
As proved by Ben-or et al. [4], BFT protocols need 3f+1replicas to tolerate up to f faulty replicas in the asynchronoussettings. This is mainly because that a faulty replica maysend conflicting proposals to different subsets of replicas, i.e. ,equivocating. Suppose there are three replicas, where A is afaulty primary replica and B, C are non-faulty replicas. ThenA can send two different proposals P B and P C to B and Cseparately. After that, A will tell B that he votes for P B , andtell C that he votes for P C . Both B and C will receive themajority of votes so that they will execute the P B and P C separately, which contradicts the safety rules of BFT.By combining each proposal with a monotonic sequencenumber, the equivocation can be eliminated easily. Eachreplica has a trusted monotonic counter locally and keeps insync with S p ’s counter. Correct replicas never accept the mes-sage with discrete counter values. By preventing equivocation,the minimum number of replicas to tolerate f faulty nodescan be reduced from 3f+1 to 2f+1. There have already beenmany solutions using different kinds of hardware as trustedonotonic counters, e.g. , Append-Only Memory(A2M) [7],FPGA [8] and TPM [9]. D. TEE-assisted Cryptographic Primitives
TEE provides a sealed and trusted container which makesmany simple but unsafe cryptographic solutions possibleagain. For example, Jian Liu et al. [21] replace multi-signaturewith TEE-assisted lightweight secret sharing to get betterperformance. In this paper, we also use TEE-assisted secretsharing to replace multi-signature. Besides, we introduce anovel TEE-assisted distributed verifiable random function toachieve unpredictable and verifiable leader election.III. I
NSPIRATIONS FROM
CFTIn this section, we discuss new security issues faced byTEE-based BFT by revisiting popular CFT protocols. Wechoose the most popular TEE, i.e. , Intel SGX, and Raft [12],one of the most popular and understandable CFT protocolsto demonstrate those issues and propose four principles, i.e. ,one proposal, one vote, restricted commit, and restricted loggeneration to solve them.
A. TEE with Byzantine Host
The core difference between TEE-based BFT and CFT isthat the host in TEE-based BFT may be byzantine. As wementioned before, a malicious host can arbitrarily manipulateits local TEE. It can arbitrarily manipulate TEE schedulingand I/O. It can drop, replay, delay, reorder, modify the I/Omessages of TEE arbitrarily, schedule TEE in any sequence,or even terminate TEE at any time. Through these maliciousbehaviors, the byzantine host may deceive other replicas tobreak the safety and liveness of the system. The followingfour principles use TEE to tolerate those behaviors and bridgethe gap between TEE-based BFT and Raft.
B. One Proposal
In Raft, at the beginning of a log replication, the leader willchoose a client request and replicate it to other replicas as anentry. Once the entry has been safely replicated to majorityreplicas, the leader will tell other replicas to commit this entry.However, in BFT, a malicious leader may propose differentproposals to different subsets of replicas, i.e. , equivocating.As we mentioned in section II, this will lead to inconsistentstates among replicas.Similar to previous work, we use trusted monotonic countersto prevent leader equivocation. As shown in Algorithm 1, weprovide TEE primitives as a trusted monotonic counter. Theleader needs to call CREATE_COUNTER to assign a (c,v)for each proposal while other replicas will keep track of theleader’s (c,v). Upon calling CREATE_COUNTER, TEE willincrease its local monotonic counter. Hence, there will neverbe two different proposals with the same(c,v) value so thatthe equivocation can be prevented. In this paper, we calledthis One Proposal Principle, i.e. , the leader can not proposemultiple proposals in the same round.
C. One Vote
In Raft, if an entry has been safely replicated to majorityreplicas, it is committed. Raft guarantees that committedentries are durable and will be executed by all of the availablestate machines eventually. However, byzantine hosts can usethis to break safety. For instance, assume a network partitionoccurs and all f+1 non-faulty replicas are split into two subsets.The non-faulty replicas in the different subsets cannot commu-nicate with each other while all f byzantine replicas can stillcommunicate with all replicas. Due to the network partition,there may be multiple leaders, i.e. , stale leaders, and thecurrent leader, proposing different proposals at the same time.In Raft, a replica will only accept one proposal so that only oneproposal will be replicated to majority replicas successfullyand get committed. However, in TEE-based BFT, a byzantinereplica may vote for multiple proposals. Then more than oneproposal will collect f+1 votes and get committed. This willlead to inconsistent states among replicas.Hence, TEE-based BFT should guarantee that a replica cannot vote for more than one conflicting proposals. We called itOne Vote Principle. The corresponding primitives are shownin Algorithm 1. Upon calling VERIFY_COUNTER, TEE willcheck the (c,v) of the proposal. If the (c,v) is matched withits local (c,v), TEE will increase its local monotonic counter.Otherwise, it will reject to vote for it. By doing so, we solvethe problem above.
D. Restricted Commit
Raft has a strong leader model. The leader is responsiblefor notifying other replicas to commit proposals. However, inTEE-based BFT, a byzantine host may lie that a proposal hasbeen safely replicated to majority replicas to let non-faultyreplicas commit a proposal.Hence, the leader should prove that a proposal has beensafely replicated to majority replicas. Generally, this proof iscalled Quorum Cert(QC) in BFT. Upon receiving a proposal,each replica will send a vote to the leader. Upon collecting atleast f+1 votes, the leader can generate a QC and send it to allreplicas as proof of commit. As suggested in [21], we use light-weight TEE-assisted secret-sharing to replace multi-signature.The corresponding primitives are described in Algorithm 1.
E. Restricted Log Generation
In Raft, each replica needs to be honest about the entrythey have received. It is essential when a new leader electionhappens. However, this is not trivial in TEE-based BFT.TEE only has limited resources, e.g. , Intel SGX only has128MB memory currently. Hence, TEE has to store messagelogs via untrusted hosts. A byzantine host may modify thosemessage logs, e.g. , drop message logs arbitrarily. Besides,it can generate a stale message log previously and replaythis message log. In Algorithm 1, we design a TEE-basedmechanism to solve this problem. With this mechanism, werestrict the log generation and guarantee that all replicas haveto be honest about the entries they voted for, i.e. , if replica S i voted for proposal p in view v, then it can only generate valid message log L that contains p (or do nothing). Forsimplicity, we leave the proof in Section VI.IV. TBFTIn this section, we present the basic TBFT. Firstly, wedescribe the system model of TBFT. Then we introducethe basic TBFT in both normal-case and view-change caseusing TEE-assisted primitives mentioned in section III. Finally,we improve the performance of TBFT using the pipelinemechanism. A. System Model
The system is composed of n replicas in which up to f ( f + 1 ≤ n ) replicas may fail arbitrarily. Each replica isequipped with a TEE. The network may drop, delay, reorder,modify and duplicate messages. Our system guarantees safetyin an asynchronous network and guarantees liveness in apartial synchronous network. During the initialization phase,all replicas will do remote attestation to prove that their localTEEs are genuine. Besides, all TEEs will generate keys inAlgorithm 1 and exchange them with other TEEs. B. Normal-case
In the following, we describe the normal-case of the basicTBFT. The detailed protocol is shown in Algorithm 2 and themessage patterns is shown in Fig 1a. a) Prepare phase:
In the basic TBFT, each replicamaintains a (c,v) sequence to guarantee One Proposal and OneVote principles. Each (c,v) is bound to a unique request. Whenthe primary replica starts to process a new request, it willextend the tail of its (c,v) sequence with the new proposal bycalling CREATE_COUNTER first. It will also call GENER-ATE_SECRET to get n shares and send each replica the newproposal with one share and the hash of the secret. Note thatthe secret is generated in TEE so that even a byzantine primaryreplica can not know the secret without collecting f+1 shares.Upon receiving the message, non-primary replicas should callVERIFY_COUNTER to verify the proposal and get the secretshare. If the proposal is valid, the replica will send its shareto the primary replica. Note that a faulty primary replica maydeny serving specific requests. Other replicas receiving therequests will send them to the primary replica and wait forthe Prepare messages. If timeout, replicas will ask for view-change. b) Commit phase:
Upon receiving f+1 shares, the pri-mary replica can reconstruct the secret as a Commit-QC. ThisQC represents that at least f+1 replicas have accepted thisproposal. The primary replica will send the QC to all replicas.After receiving a valid Commit-QC, replicas will regard thecorresponding proposal as committed. They will execute therequest and apply it to local states. In this phase, the primarywill also generate a secret, collect f+1 votes and reconstructa secret as a QC. This QC will serve as a proof-of-execute toprove that the majority of replicas have executed the request.It can be replaced with other methods, e.g. , let all replicassend a message to clients to prove that the request has been
Algorithm 1
TEE-assisted Primitives c, v (cid:46) c is the current value of S i ’s monotonic counter, vis the current view(maintained by all replicas) c (cid:48) , v (cid:48) (cid:46) value of the last validated counter(c,v) { S i , pk i } (cid:46) all replicas and their public keys { S i , k i } (cid:46) all replicas and their symmetric keys function GENERATE _ SECRET ( c (cid:48)(cid:48) , v (cid:48)(cid:48) ) (cid:46) for S p s= rand ( ) h c (cid:48)(cid:48) = Hash (s) { s i } i ← s (cid:46) Shamir-Secret-Sharing ξ ic (cid:48)(cid:48) = E ( k i , < s ic (cid:48)(cid:48) , ( c (cid:48)(cid:48) , v (cid:48)(cid:48) ) , h c (cid:48)(cid:48) > ) < h c (cid:48)(cid:48) , ( c (cid:48)(cid:48) , v (cid:48)(cid:48) ) > σ p = Sign ( < h c (cid:48)(cid:48) , ( c (cid:48)(cid:48) , v (cid:48)(cid:48) ) > ) Return { < h c (cid:48)(cid:48) , ( c (cid:48)(cid:48) , v (cid:48)(cid:48) ) > σ p , ξ ic (cid:48)(cid:48) } i function CREATE _ COUNTER (x) (cid:46) for S p < x, ( c, v ) > σ p = Sign ( < x, ( c, v ) > σ p ) c = c + 1 Return < x, ( c, v ) > σ p function VERIFY _ COUNTER ( < x, ( c p , v p ) > σ p , ξ ic (cid:48)(cid:48) ) (cid:46) forall replicas if Vrfy( < x, ( c p , v p ) > σ p )==false thenReturn REQ-VIEW-CHANGE::Invalid_Message else < s ic (cid:48)(cid:48) , ( c (cid:48)(cid:48) , v (cid:48)(cid:48) ) , h c (cid:48)(cid:48) > = D( ξ ic (cid:48)(cid:48) ) if ( c p , v p )! = ( c (cid:48)(cid:48) , v (cid:48)(cid:48) ) or c ! = c p or v ! = v p thenReturn REQ-VIEW-CHANGE::Invalid_Counter if ( v (cid:48) == v p and c (cid:48) + 1 == c p ) or ( v (cid:48) < v and c p == 0) then c (cid:48) = c p ; v (cid:48) = v p ; c = c + 1 function MERGE _ MESSAGE _ LOGS ( {(cid:104) L, (cid:104) H ( L ) , ( c i , v i ) (cid:105) σ i } i ) (cid:46) for next primary replica S p (cid:48) for each L in {(cid:104) L, (cid:104) H ( L ) , ( c i , v i ) (cid:105) σ i } i doif vrfy(L)==true then O.merge(L)update_counter(O) (cid:46) update c, v, c’, v’ according to O (cid:104) H ( O ) , ( c, v ) (cid:105) σ p (cid:48) =TEE.create_counter(H(O)) Return (cid:104) H ( O ) , ( c, v ) (cid:105) σ p (cid:48) function GET _ MESSAGE _ LOGS (L) (cid:46) for all replicas if vrfy(L)==true then (cid:104) H ( L ) , ( c, v ) (cid:105) σ i =TEE.create_counter(H(L)) Return (cid:104) H ( L ) , ( c, v ) (cid:105) σ i elseReturn ERROR::Invalid_messgae_log function
UPDATE _ VIEW ( (cid:104) O, (cid:104) H ( O ) , ( c p (cid:48) , v p (cid:48) ) (cid:105) σ p (cid:48) , e i (cid:105) ) (cid:46) for all replicas if Vrfy( (cid:104) O, (cid:104) ( H ( O ) , ( c p (cid:48) , v p (cid:48) ) (cid:105) σ p (cid:48) (cid:105) )==true then update_counter(O) (cid:46) update c, v, c’, v’ accordingto O " ! ' " ' ($)*$+& ,($,-($ ./00 (a) Basic TBFT: Normal-Case ' ! ! ' " ' Req - New - View View - Change New - View (b) Basic TBFT: View-Change
Fig. 1: Message Patterns inTBFT
Algorithm 2
Normal-case upon reception of M= (cid:104)
Request, op (cid:105) at S p do {(cid:104) h c , ( c, v ) (cid:105) σ p , ξ ic } i =TEE.generate_secret(c,v) (cid:104) H ( M ) , ( c, v ) (cid:105) σ p =TEE.create_counter(H(M))send (cid:104) P repare, M, (cid:104) H ( M ) , ( c, v ) (cid:105) σ p , (cid:104) h c , ( c, v ) (cid:105) σ p , ξ ic (cid:105) i to each S i upon reception of f+1 (cid:104) V ote − F or − Commit, s ic (cid:105) at S p do s c ← { s ic } i (cid:46) Shamir Secret Sharing, s c is theCommit-QC res ← execute opx ← H ( M | res ) (cid:104) x, ( c + 1 , v ) (cid:105) σ p = TEE.create_counter(x) {(cid:104) h c +1 , ( c + 1 , v ) (cid:105) σ p ,ξ ic +1 } i = TEE.generate_secret ( c +1 , v ) send (cid:104) Commit, s c , res, (cid:104) x, ( c + 1 , v ) (cid:105) σ p , (cid:104) h c +1 , ( c +1 , v ) (cid:105) σ p , ξ ic +1 (cid:105) i to each replica S i upon reception of f+1 (cid:104) V ote − F or − Decide, s ic +1 (cid:105) at S p do s c +1 ← { s ic +1 } i (cid:46) s c +1 is the Decide-QCsend (cid:104) Decide, res, (cid:104) H ( M | res ) , ( c + 1 , v ) (cid:105) σ p , (cid:104) h c +1 , ( c + 1 , v ) (cid:105) σ p s c +1 , (cid:104) H ( M ) , ( c, v ) (cid:105) σ p , (cid:104) h c , ( c, v ) (cid:105) σ p , s c (cid:105) toclientsend (cid:104) Decide, s c +1 (cid:105) to all replicas upon reception of (cid:104) P repare, M, (cid:104) H ( M ) , ( c, v ) (cid:105) σ p (cid:104) h c , ( c,v ) (cid:105) σ p , ξ ic (cid:105) i at S i do (cid:104) s ic , h c (cid:105) =TEE.verify_counter( (cid:104) H ( M ) , ( c, v ) (cid:105) σ p (cid:105) , ξ ic )send (cid:104) V ote − F or − Commit, s ic (cid:105) to S p upon reception of (cid:104) Commit, s c , res, (cid:104) x, ( c + 1 , v ) (cid:105) σ p , (cid:104) h c +1 , ( c + 1 , v ) (cid:105) σ p , ξ ic +1 (cid:105) i at S i doif H( s c ) != h c or execute op != res then Request for View-Change else (cid:104) s ic +1 , h c +1 (cid:105) =TEE.verify_counter( (cid:104) H ( M ) , ( c + 1 ,v ) (cid:105) σ p , ξ ic +1 )send (cid:104) V ote − F or − Decide, s ic +1 (cid:105) to S p upon reception of (cid:104) Decide, s c +1 (cid:105) at S i doif H( s c +1 ) != h c +1 then Request for View-Change executed by at least f+1 replicas. We discuss the trade-offs inSection VII. c) Decide phase:
Upon receiving f+1 shares, the primarycan reconstruct the secret as an Execute-QC. The primaryreplica will send it to clients to prove that at least f+1 replicashave executed this request and get the same execution result.
C. View-change
In TBFT, all replicas may be byzantine, e.g. , give maliciousresponses or crash. When byzantine behaviors occur, replicaswill ask for view-change. In the following, we discuss theview-change operation in TBFT. The detailed protocol isshown in Algorithm 3 and the message pattern is shown inFig 1b. a) Request for New-View:
A non-primary replica canrequest for view change by sending Request-New-View to thenext primary replica S p (cid:48) . It will also generate its local messagelog and send it to S p (cid:48) . Recall that we use TEE primitives toguarantee Restricted Log Generation, i.e. , a replica can onlygenerate a genuine log(or do nothing). b) View-Change: Upon receiving f+1 Request-New-View messages, S p (cid:48) will call MERGE_Message_LOG to gen-erate a message history O. Then S p (cid:48) will broadcast it to allreplicas. After receiving the message, other replicas will verifyit and execute all requests in it that have not yet been executedlocally. Hence, before changing to the next view, all non-faultyreplicas will execute the same requests in the same order. Inthis progress, S p (cid:48) will also generate a secret, collect f+1 shares,generate a NEW-VIEW QC, and send it to all replicas. c) New-View: Upon receiving a valid NEW-VIEW QC,other replicas will execute all requests in it that have not yetbeen executed locally and change to the next view eventually.
D. Pipeline
In basic TBFT protocol, each request requires two phases(Prepare and Commit) to get processed. In each phase, the pri-mary replica needs to generate a secret, collect f+1 votes fromall replicas and generate a QC. Although we use lightweightTEE-assisted secret sharing to replace multi-signature, it stillhas large impact on performance. Moreover, in basic TBFT,one valid QC is used for only one phase. It is very inefficient.In chained TBFT protocol, we make full use of QCs by lgorithm 3
View-change upon reception of f+1 valid (cid:104)
Request − V iew − Change,L, (cid:104) H ( L ) , ( c i , v i ) (cid:105) σ i (cid:105) at S p (cid:48) do (cid:104) H ( O ) , ( c, v ) (cid:105) σ p (cid:48) =TEE.merge_message_logs ( { L, (cid:104) H ( L ) , ( c i , v i ) (cid:105) σ i } ) {(cid:104) h c , ( c, v ) (cid:105) σ p , ξ ic } i =TEE.generate_secret ( c, v ) send (cid:104) V iew − Change, O, (cid:104) H ( O ) , ( c, v ) (cid:105) σ p (cid:48) , e i , (cid:104) h c , ( c, v ) (cid:105) σ p , ξ ic (cid:105) to all replicas upon reception of f+1 (cid:104) V ote − F or − N ewview, s ic (cid:105) at S (cid:48) p do s c ← { s ic } TEE.update-view ( (cid:104) O, (cid:104) H ( O ) , ( c, v ) (cid:105) σ p (cid:48) , e i (cid:105) ) send (cid:104) N ew − V iew, s c , e i (cid:105) to all replicas upon invocation of Request-View-Change at S i do (cid:104) H ( L ) , ( c, v ) (cid:105) σ i =TEE.get_message_logs(L)send (cid:104) Request − V iew − Change, L, (cid:104) H ( L ) , ( c, v ) (cid:105) σ i (cid:105) to S p (cid:48) upon reception of (cid:104) V iew − Change, O, (cid:104) H ( O ) , ( c, v ) (cid:105) σ p (cid:48) , (cid:104) h c , ( c, v ) (cid:105) σ p , ξ ic (cid:105) at S i doif O is valid then (cid:104) s ic , h c (cid:105) =TEE.verify_counter ( (cid:104) H ( M ) , ( c, v ) (cid:105) σ p (cid:48) , ξ ic ) send (cid:104) V ote − F or − N ewview, s ic (cid:105) to S p (cid:48) upon reception of (cid:104) N ew − V iew, s c , { e i } i (cid:105) at S i doif s c is valid then TEE.update-view( (cid:104) O, (cid:104) H ( O ) , ( c, v ) (cid:105) σ p (cid:48) , e i (cid:105) ) QC &' ! (c,v) ( c " , v " ) Prepare&' ! Commit Decide Prepare&' " PreparePrepareDecide Decide CommitCommit DecideCommit&' &' $ QC &' ! (c,v) ( c " , v " ) QC &' ! (c,v) ( c " , v " ) QC &' ! (c,v) ( c " , v " ) Fig. 2: Chained TBFTpipelining QCs, e.g. , the Commit-QC of the current requestalso serves as the Execute-QC of the previous request. Fig 2shows the pipeline in TBFT. We leave the proof of safety andliveness in Section VI.V. U
NPREDICTABLE AND V ERIFIABLE L EADER E LECTION
In this section, we discuss the leader election mechanismin TBFT. In traditional BFT protocols, the result of leaderelection is predictable, e.g., in Hotstuff [3], each replica hasa number, and the primary replica for view v is v%n, wheren is the number of replicas. However, a predictable leaderelection mechanism leads to many serious problems, e.g., anadversary may launch a DoS(Denial of Service) attack againstthe primary replica. By predicting the next primary replica andlaunching DoS attacks against it successively, the attacker canbring the system down for a long time. To avoid such attacks, an unpredictable leader electionmechanism is needed. However, this is not trivial. Everyreplica should have a consensus on the next primary replicawhile the mechanism must be distributed(or a single-point fail-ure may bring the system down). Distributed Verifiable Ran-dom Function(DVRF) [22] can solve this problem. However,traditional DVRF solutions [23] [24] [22] rely on complexcryptographic methods, e.g. threshold BLS signatures [25]or publicly verifiable secret sharing(PVSS) [26]. To generateone random number, replicas may need to run multi-roundprotocols or do expensive computation which means largecommunication and computation overhead.Utilizing TEE, we introduce a novel and efficient solutionfor unpredictable and verifiable leader election. During ini-tialization, TEEs will share a random seed via secret sharing.All TEEs uses the same random function(Remote Attestationguarantees verifiability). So with the same input sequence, theywill generate the same random number. In detail, replicas willuse H ( s ) to generate a random number r and regard r%n asthe next primary replica where H ( s ) is the hash of the lastvalid secret in v’ and v’ is the previous view of v. Recall thatbefore changing to the next view v, all replicas will processall proposals in v’ in the same order so that every replica willhave a consensus on H ( s ) . Note that each secret is a randomnumber generated in TEE, so its value is unpredictable.By doing so, we delay the announcement of election results,i.e., only after view v-1 finishes, faulty replicas can knowthe primary replica of view v+1. It will effectively limit thewindow of time for attackers to launch attacks, e.g., DoSattacks against the primary replica. Besides, this solution onlyhas O (1) communication and computation complexity whichis much more efficient than other solutions.VI. S AFETY AND L IVENESS
A. Safety
In this section, we prove that if one non-faulty replica rexecuted a sequence of operations (cid:104) op , op , op ...op n (cid:105) , thenall non-faulty replicas executed a prefix of it(or be prefixedwith it). Definition 1.
A message log (cid:104) L, (cid:104) H ( L ) , ( c, v ) (cid:105) σ i (cid:105) is validif and only if its follows the following rules:(1) it contains valid signatures.(2) the counter value c = c v +1, where c v is the highestcounter value in L.(3) there are no holes in the (c,v) sequence in L. Lemma 1.
In a view v, if S i voted for proposal p, then S i can only generate valid message log L that contains p (ordo nothing). proof. We proof this lemma by contradiction. Assume ina view v, S i voted for proposal p(bound to (c,v)) and generateda valid message log L that does not contain p. There are thefollowing cases:(1) S i voted for p before it generated L. It means S i hasalready sent p to its local TEE for voting. Hence, TEEas already updated its ( c, v ) . When generating L, S i willsend L to its local TEE for signature. then there must be ahole in the (c,v) sequence of L and TEE will not sign forit. Otherwise, there must be another proposal p’ havingthe same (c,v) with p, which is impossible because S p ’sTEE will never sign different proposals with the same(c,v). A contradiction.(2) S i voted for p after it generated L. Once S i generated L,due to rules in TEE, S i ’s (c,v) will change( c → c + 1 )without changing (c’, v’). After that, when S i wants tovote for other requests, there will be a hole between(c’,v’) and ( c p , v p ). Hence, S i ’s TEE will refuse to votefor any proposals(including p) in this view. A contradic-tion. Lemma 2.
In a view v, no two non-faulty replicas willexecute different requests with the same (c,v). proof.
To show a contradiction, assume in view v, S i executed op and S j executed op’ with the same (c,v) pair.Then there are three cases:(1) both of S i and S j executed op and op’ in normal-case,which means they all have received a valid Generic-QCwith the same (c,v). However, S p ’s TEE guaranteed that itwill never assign different proposals with the same (c,v).Hence, there will never be two Generic-QCs with thesame (c,v), a contradiction.(2) S i executed op during normal-case execution and S j ex-ecuted op’ during view-change. It means S i has receiveda valid Generic-QC with (c,v), i.e., at least a quorumQ of f+1 replicas have received a proposal containingop( P roposal op ) and voted for it. During view-change,there were at least a quorum Q’ of f+1 replicas have sentvalid Request-View-Change and generated valid messagelogs. Hence, Q and Q’ will have at least an intersectionreplica S k . Note that S k have voted for P roposal op andgenerated a valid message log. Lemma 1 proves that S k has to add P roposal op into its message log. Then, S p (cid:48) ’sTEE must have added P roposal op to O. So non-faulty S j must have executed op before changing into the nextview. A contradiction.(3) both S i and S j executed op and op’ during view-change.Then, there must be at least two different valid messageHistory O and O’ containing P roposal op with(c,v) and P roposal op (cid:48) with (c,v) separately. However, S p ’s TEEwill never assign the same (c,v) pair to different requests.A contradiction. Lemma 3.
Before changing to the next view v+1, everynon-faulty replica will execute op if and only if there is atleast one non-faulty replica S i have executed op in view v. proof. To show a contradiction, assume S j did not executeop and changed to the next view. There are two cases:(1) S i executed op during normal-case. According toLemma2 case 2), non-faulty replicas have already exe-cuted op before changing to view v+1. A contradiction. (2) S i executed op during view-change. It means that at leastf+1 replicas have voted for a message history O thatcontains op. In view v, there is only one valid messagelog O(if not, it contradicts with Lemma 2). So S j has toexecute O or stay in v. A contradiction. Theorem 1.
If a non-faulty replica executed a sequenceof operations (cid:104) op , op , op ...op n (cid:105) , then all non-faulty replicasexecuted a prefix of it(or be prefixed with it). proof. To show a contradiction, assume S j and S i were both non-faulty replicas, but they executed (cid:104) op j , op j , op j ...op jn (cid:105) , (cid:104) op i , op i , op i ...op in (cid:105) separately. Thenthere must be a minimal k satisfying that op ik ! = op jk . If op ik and op jk are executed in the same view, then they must havethe same (c,v) (or k is not the minimum), which contradictsLemma 2. If op ik and op jk are executed in different views v i and v j separately, then it contradicts Lemma 3. A contradiction. B. Liveness
In this section, we prove that requests from clients will beeventually executed.
Lemma 4.
Suppose that, in view v, the longest validmessage log is L = (cid:104) m , m , ...., m n (cid:105) , then all valid messagelogs are the same with L or prefixes of L. proof. Assume there are one valid message logs L (cid:48) = (cid:104) m (cid:48) , m (cid:48) ..., m (cid:48) n (cid:48) (cid:105) have different prefix. There is a k that satis-fies (cid:104) m , m , ...m k − (cid:105) = (cid:104) m (cid:48) , m (cid:48) ..., m (cid:48) k − (cid:105) and m k ! = m (cid:48) k .L and L’ are both valid, hence m k and m (cid:48) k must have the same(c,v). However, S p ’s TEE will never assign the same (c,v) pairto two different messages, a contradiction. Lemma 5.
A view v will eventually be changed into aview v’ with a non-faulty primary replica if f+1 non-faultyreplicas request view-change. proof.
First, we prove that the view change will eventuallyfinish. Lemma 4 tells us that valid message logs never con-tradict each other. So a replica can always generate a valid Oafter collecting f+1 valid Request-View-Change messages. Ifthe next primary replica is faulty or network partitions occur,a new view-change will be triggered out. Otherwise, we havea view with a non-faulty primary replica. lemma 6.
If all non-faulty replicas remain synchronizedin view v during a bounded time t, and the primary replica inview v is non-faulty, then a consensus can be reached. proof.
Under the weak synchrony assumption, after GST,all non-faulty replicas are synchronized in view v. Therefore,all non-faulty replicas will vote for a QC. And the primaryreplica is S p is non-faulty, so all non-faulty replicas willreceive a valid QC. Eventually, all non-faulty replicas willexecute the request and response to the client. Theorem 2.
The request from clients will be eventuallyexecuted. proof.
During the processing of a request, if no replicarequests for view-change, the request will be executed. Oth-erwise, there are two cases:) If at least f+1 non-faulty replicas request view-change,Lemma 5 proves the view will eventually be changedinto a view v’ with a non-faulty primary replica. Lemma6 proves that the request will be executed.2) If less than f+1 non-faulty replicas request view-change,no view-change will be triggered out if some faultyreplicas follow the protocol. Then the request will beexecuted. Otherwise, all non-faulty replicas will requestfor view-change and this case will fall into case 1.VII. E
VALUATION AND D ISCUSSION
A. Evaluation
In this part, we evaluate TBFT via systematic analysis andexperiments. Firstly, we compared TBFT with several popularBFT protocols in several aspects, including the performance,scalability, resilience, and security of BFT. In Table I, latencymeans how many times the network latency is needed tohandle a single request in normal-case(including client sendingrequests and receiving replies), and resilience means howmany non-primary faults will corrupt normal execution. AsTable I shows, TBFT has better properties in many aspectscompared to those protocols.We have published the implementation of TBFT online .It is based on an open-source library provided by MinBFTand use Intel SGX as TEE. Please note that although theimplementation uses Intel SGX, TBFT can be implementedon other standard TEE platforms, e.g. , GlobalPlatform [27].Our implementation has roughly 11000 lines of untrusted Gocodes and 600 lines of trusted C codes. The TCB is smallenough to be formally verified.We choose MinBFT as the baseline to evaluate the perfor-mance and scalability of TBFT. To the best of our knowledge,MinBFT is the only TEE-based BFT protocol listed in TableI which has an official and open-source implementation. Theexperiments use a server with 2 Intel E5-2680 v4 CPU (14cores and 28 threads each), 384G RAM, Ubuntu 18.04 as OS,and the Intel SGX is running in SIM mode. For each exper-iment, we use 16 clients sending requests concurrently. Eachclient will send 1000 requests to simulate a real workload.Fig 3a shows the peak throughput of TBFT and MinBFTwith varying number of replicas. We use two different payloadsizes, i.e. , 1KB and 1MB to simulate different block sizes.The results show that TBFT has much better performance andscalability than MinBFT. Especially when the payload size is1MB, the throughput of MinBFT decreases very fast whileTBFT decrease much slower. Assume the average transactionsize is about 200bytes(as same as the average transaction sizeon Ethereum [31], TBFT can process up to 550K transactionsper second.Fig 3b shows the average end-to-end latency of TBFT andMinBFT with varying number of replicas. Clients start thetimer after sending requests and stop it after receiving results.The results show that TBFT has a much smaller latency thanMinBFT. With 1MB payload, when the number of replicas https://github.com/TBFT/Protocol increasing to 17, the latency of MinBFT is even 9x latency ofTBFT. B. Discussion
In this part, we discuss some trade-offs we met whendesigning TBFT.
1) Why 2f+1? :
Unlike TBFT, there are also some TEE-based BFT protocols having only f+1 active replicas. In thoseprotocols, passive replicas only need to update their states pas-sively rather than participate in the consensus all time. Whenactive replicas deviate from the protocol, passive replicas willbe awakened. Those protocols are resource-friendly. However,they have poor resilience. To tolerate even single-point non-primary faults, those protocols may need to trigger a view-change or transit to a fallback protocol. Unfortunately, thosefaults may occur frequently and faulty nodes may also usethem to reduce the performance of the system. TBFT focuseson general scenarios so that we believe 2f+1 is a more suitablesolution.
2) Why 2 phases? :
As we mentioned before, the secondphase in TBFT is a proof-of-execute. The Execute-QC provesthat more than f+1 replicas have executed the request. It canbe generated in other ways, e.g. , let all replicas send the resultsto clients. In that case, a client needs to collect messages fromat least f+1 replicas to know that its request has already beenexecuted. We keep the Execute-QC for the following reasons:1) The client may have limited resources, e.g., the clientmay be an edge device in IoT with limited computingpower and network bandwidth. It may not have theability to receive f+1 packets and validate the signaturesseparately especially when f is large. Besides, replicasusually share a stable network with high speed whilethe network situation between replicas and clients is farmore complicated and unstable. It may take even moretime( i.e. , latency) for clients to collect f+1 messagesthan introducing one more phase. Hence, it is natural tolet resourceful replicas take on more computation andcommunication tasks.2) In TBFT, we replace inefficient multi-signature withlightweight secret sharing. To validate a reply, a clientonly needs to verify two secrets with O(1) computationcomplexity. It is much more efficient than collecting f+1replies and verifying f+1 signatures( i.e. , O(n) computa-tion complexity).3) In TBFT, phases are pipelined. Although we introduceone more phase in TBFT, the throughput of the systemis almost the same.VIII. R
ELATED W ORK
The advent of commodity TEEs has brought many oppor-tunities to distributed systems. Many previous works focus onTEE-based consensus protocol for blockchain. Some of themfocus on proof-of-X consensus protocol. REM [32] proposesa variant of PoW [33] to transform meaningless mining intouseful work. PoET [34] use a random timer provided by TEEto represent the work of miners in PoW. PoLK [35] use trusted rotocols Active/All Replicas Normal-CaseComplexity View-ChangeComplexity Latency Resilience Pipeline DVRF TEEFastBFT [21] f+1/2f+1 O ( n ) O ( n ) O ( n ) O ( n ) O ( n ) O ( n ) O ( n ) O ( n ) O ( n ) O ( n ) O ( n ) O ( n ) O ( n ) O ( n ) TBFT 2f+1/2f+1 O ( n ) O ( n ) TABLE I: TBFT vs. Previous Protocols
Number of Replicas T h r o u g h p u t ( o p / s ) TBFT-1kMinBFT-1kTBFT-1MMinBFT-1M (a) Throughput vs. n
Number of Replicas L a n t e n c y ( m s ) TBFT-1kMinBFT-1kTBFT-1MMinBFT-1M (b) Latency vs. n
Fig. 3: Evaluation Resultsrandom number to elect the leader and use trusted monotoniccounter to prohibit concurrent invocations of TEEs.Other works focus on TEE-based BFT protocols. MinBFT[9] uses a trusted counter service to improve PBFT. It has O ( n ) message complexity. CheapBFT [36] is a resource-friendly TEE-based BFT protocol that requires only f+1 activereplicas to reach consensus. If byzantine behaviors occur,CheapBFT will awake all passive replicas and switch toMinBFT. FastBFT [21] is one of the most recent work.It also requires f+1 active replicas to reach consensus. Itintroduces a novel message aggregation technique to improveits performance. It has O ( n ) message complexity in view-change. After a threshold number of failure detections, itwill transit to a fallback protocol. Hybster [28] is a TEE-based BFT protocol that leverages parallelization to improveits performance. It is orthogonal to our work.Except for consensus protocols, TEE has also been appliedin other research areas of distributed systems, especiallyblockchain. It has been used to replace complex cryptographicsolutions, e.g. zero knowledge proof [37] or multi-party com-putation [38] in many areas, e.g. , payment channel [39], mixer[40], oracle [41], light-weight client [42], off-chain computa-tion [10], sharding [43], and cross-chain [44]. It is also used toprotect privacy and confidentiality for the execution of smartcontract [45] [46] [47]. Combined with those works, TBFT canhelp to build a complete and comprehensive research systemfor TEE-based blockchain. IX. C ONCLUSION
In this paper, we propose TBFT, an understandable andefficient TEE-based BFT protocol. It is the first TEE-basedBFT protocol with O ( n ) message complexity in both view-change and normal-case. Different from the previous work,the design of TBFT is inspired by CFT protocol to get moreunderstandability and simplicity. Besides, we make compre-hensive improvements for both performance and security, in-cluding pipeline mechanisms, TEE-assisted secret sharing, andunpredictable leader election. Our experiments demonstratethat TBFT has better performance and scalability comparedto previous protocols. R EFERENCES[1] H. Sukhwani, J. M. Martínez, X. Chang, K. S. Trivedi, and A. Rindos,“Performance modeling of pbft consensus process for permissionedblockchain network (hyperledger fabric),” in , 2017, pp. 253–255.[2] J. Zhang, J. Gao, Z. Wu, W. Yan, Q. Wo, Q. Li, and Z. Chen, “Perfor-mance analysis of the libra blockchain: An experimental study,” in . IEEE, 2019, pp. 77–83.[3] M. Yin, D. Malkhi, M. K. Reiter, G. G. Gueta, and I. Abraham,“Hotstuff: Bft consensus with linearity and responsiveness,” in
Proceedings of the 2019 ACM Symposium on Principles of DistributedComputing , ser. PODC ’19. New York, NY, USA: Associationfor Computing Machinery, 2019, p. 347–356. [Online]. Available:https://doi.org/10.1145/3293611.3331591[4] M. Ben-Or, “Another advantage of free choice (extended abstract):Completely asynchronous agreement protocols,” in
Proceedings ofthe Second Annual ACM Symposium on Principles of DistributedComputing , ser. PODC ’83. New York, NY, USA: Associationor Computing Machinery, 1983, p. 27–30. [Online]. Available:https://doi.org/10.1145/800221.806707[5] D. Dolev and R. Reischuk, “Bounds on information exchange forbyzantine agreement,”
Journal of the ACM (JACM) , vol. 32, no. 1, pp.191–204, 1985.[6] M. J. Fischer, N. A. Lynch, and M. S. Paterson, “Impossibility ofdistributed consensus with one faulty process,”
Journal of the ACM(JACM) , vol. 32, no. 2, pp. 374–382, 1985.[7] B.-G. Chun, P. Maniatis, S. Shenker, and J. Kubiatowicz, “Attestedappend-only memory: Making adversaries stick to their word,”
ACMSIGOPS Operating Systems Review , vol. 41, no. 6, pp. 189–204, 2007.[8] R. Kapitza, J. Behl, C. Cachin, T. Distler, S. Kuhnle, S. V. Mohammadi,W. Schröder-Preikschat, and K. Stengel, “Cheapbft: Resource-efficientbyzantine fault tolerance,” in
Proceedings of the 7th ACM EuropeanConference on Computer Systems , ser. EuroSys ’12. New York,NY, USA: Association for Computing Machinery, 2012, p. 295–308.[Online]. Available: https://doi.org/10.1145/2168836.2168866[9] G. S. Veronese, M. Correia, A. N. Bessani, L. C. Lung, and P. Verissimo,“Efficient byzantine fault-tolerance,”
IEEE Transactions on Computers ,vol. 62, no. 1, pp. 16–30, 2011.[10] R. Cheng, F. Zhang, J. Kos, W. He, N. Hynes, N. Johnson, A. Juels,A. Miller, and D. Song, “Ekiden: A platform for confidentiality-preserving, trustworthy, and performant smart contracts,” in . IEEE, 2019,pp. 185–200.[11] Y. Yan, C. Wei, X. Guo, X. Lu, X. Zheng, Q. Liu, C. Zhou, X. Song,B. Zhao, H. Zhang et al. , “Confidentiality support over financial gradeconsortium blockchain,” in
Proceedings of the 2020 ACM SIGMODInternational Conference on Management of Data , 2020, pp. 2227–2240.[12] D. Ongaro and J. Ousterhout, “In search of an understandable consensusalgorithm,” in et al. , “Paxos made simple,”
ACM Sigact News , vol. 32,no. 4, pp. 18–25, 2001.[14] T. D. Chandra, R. Griesemer, and J. Redstone, “Paxos made live: Anengineering perspective,” in
Proceedings of the Twenty-Sixth AnnualACM Symposium on Principles of Distributed Computing , ser. PODC’07. New York, NY, USA: Association for Computing Machinery,2007, p. 398–407. [Online]. Available: https://doi.org/10.1145/1281100.1281103[15] B. Alpern and F. B. Schneider, “Recognizing safety and liveness,”
Distributed computing , vol. 2, no. 3, pp. 117–126, 1987.[16] G. Nibaldi, “Specification of a trusted computing base (tcb),” MITRECORP BEDFORD MA, Tech. Rep., 1979.[17] F. McKeen, I. Alexandrovich, I. Anati, D. Caspi, S. Johnson,R. Leslie-Hurd, and C. Rozas, “Intel® software guard extensions(intel® sgx) support for dynamic memory management inside anenclave,” in
Proceedings of the Hardware and Architectural Supportfor Security and Privacy 2016 , ser. HASP 2016. New York, NY,USA: Association for Computing Machinery, 2016. [Online]. Available:https://doi.org/10.1145/2948618.2954331[18] B. Ngabonziza, D. Martin, A. Bailey, H. Cho, and S. Martin, “Trustzoneexplained: Architectural features and use cases,” in , 2016, pp. 445–451.[19] R. Perez, R. Sailer, L. van Doorn et al. , “vtpm: virtualizing the trustedplatform module,” in
Proc. 15th Conf. on USENIX Security Symposium ,2006, pp. 305–320.[20] G. Coker, J. Guttman, P. Loscocco, A. Herzog, J. Millen, B. O’Hanlon,J. Ramsdell, A. Segall, J. Sheehy, and B. Sniffen, “Principles of remoteattestation,”
International Journal of Information Security , vol. 10, no. 2,pp. 63–81, 2011.[21] J. Liu, W. Li, G. O. Karame, and N. Asokan, “Scalable byzantineconsensus via hardware-assisted secret sharing,”
IEEE Transactions onComputers , vol. 68, no. 1, pp. 139–151, 2019.[22] D. Galindo, J. Liu, M. Ordean, and J.-M. Wong, “Fully distributedverifiable random functions and their application to decentralised randombeacons.”
IACR Cryptol. ePrint Arch. , vol. 2020, p. 96, 2020.[23] Y. Dodis, “Efficient construction of (distributed) verifiable randomfunctions,” in
International Workshop on Public Key Cryptography .Springer, 2003, pp. 1–17. [24] V. Kuchta and M. Manulis, “Unique aggregate signatures with ap-plications to distributed verifiable random functions,” in
InternationalConference on Cryptology and Network Security . Springer, 2013, pp.251–270.[25] D. Boneh, B. Lynn, and H. Shacham, “Short signatures from the weilpairing,”
Journal of cryptology , vol. 17, no. 4, pp. 297–319, 2004.[26] M. Stadler, “Publicly verifiable secret sharing,” in
International Con-ference on the Theory and Applications of Cryptographic Techniques .Springer, 1996, pp. 190–199.[27] J.-E. Ekberg, K. Kostiainen, and N. Asokan, “Trusted execution envi-ronments on mobile devices,” in
Proceedings of the 2013 ACM SIGSACconference on Computer & communications security , 2013, pp. 1497–1498.[28] J. Behl, T. Distler, and R. Kapitza, “Hybrids on steroids: Sgx-based highperformance bft,” in
Proceedings of the Twelfth European Conferenceon Computer Systems , ser. EuroSys ’17. New York, NY, USA:Association for Computing Machinery, 2017, p. 222–237. [Online].Available: https://doi.org/10.1145/3064176.3064213[29] M. Castro, B. Liskov et al. , “Practical byzantine fault tolerance,” in
OSDI , vol. 99, no. 1999, 1999, pp. 173–186.[30] G. G. Gueta, I. Abraham, S. Grossman, D. Malkhi, B. Pinkas, M. Reiter,D.-A. Seredinschi, O. Tamir, and A. Tomescu, “Sbft: a scalable anddecentralized trust infrastructure,” in .IEEE, 2019, pp. 568–580.[31] G. Wood et al. , “Ethereum: A secure decentralised generalised trans-action ledger,”
Ethereum project yellow paper , vol. 151, no. 2014, pp.1–32, 2014.[32] F. Zhang, I. Eyal, R. Escriva, A. Juels, and R. V.Renesse, “REM: Resource-efficient mining for blockchains,”in
Proceedings of the 2016 ACM SIGSAC Conferenceon Computer and Communications Security , ser. CCS ’16. NewYork, NY, USA: Association for Computing Machinery, 2016, p. 3–16.[Online]. Available: https://doi.org/10.1145/2976749.2978341[34] L. Chen, L. Xu, N. Shah, Z. Gao, Y. Lu, and W. Shi, “On securityanalysis of proof-of-elapsed-time (poet),” in
Stabilization, Safety, andSecurity of Distributed Systems , P. Spirakis and P. Tsigas, Eds. Cham:Springer International Publishing, 2017, pp. 282–297.[35] M. Milutinovic, W. He, H. Wu, and M. Kanwal, “Proof of luck: Anefficient blockchain consensus protocol,” in
Proceedings of the 1stWorkshop on System Software for Trusted Execution , ser. SysTEX ’16.New York, NY, USA: Association for Computing Machinery, 2016.[Online]. Available: https://doi.org/10.1145/3007788.3007790[36] T. Distler, C. Cachin, and R. Kapitza, “Resource-efficient byzantine faulttolerance,”
IEEE Transactions on Computers , vol. 65, no. 9, pp. 2807–2819, 2016.[37] A. De Santis, S. Micali, and G. Persiano, “Non-interactive zero-knowledge proof systems,” in
Advances in Cryptology — CRYPTO ’87 ,C. Pomerance, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg,1988, pp. 52–72.[38] O. Goldreich, “Secure multi-party computation,”
Manuscript. Prelimi-nary version , vol. 78, 1998.[39] J. Lind, I. Eyal, P. Pietzuch, and E. G. Sirer, “Teechan: Payment channelsusing trusted execution environments,” arXiv preprint arXiv:1612.07766 ,2016.[40] M. Tran, L. Luu, M. S. Kang, I. Bentov, and P. Saxena, “Obscuro: Abitcoin mixer using trusted execution environments,” in
Proceedings ofthe 34th Annual Computer Security Applications Conference , 2018, pp.692–701.[41] A. Beniiche, “A study of blockchain oracles,” arXiv preprintarXiv:2004.07140 , 2020.[42] S. Matetic, K. Wüst, M. Schneider, K. Kostiainen, G. Karame, andS. Capkun, “ { BITE } : Bitcoin lightweight client privacy using trustedexecution,” in { USENIX } Security Symposium ( { USENIX } Security19) , 2019, pp. 783–800.[43] H. Dang, T. T. A. Dinh, D. Loghin, E.-C. Chang, Q. Lin, and B. C. Ooi,“Towards scaling blockchain systems via sharding,” in
Proceedings ofhe 2019 international conference on management of data , 2019, pp.123–140.[44] I. Bentov, Y. Ji, F. Zhang, L. Breidenbach, P. Daian, and A. Juels,“Tesseract: Real-time cryptocurrency exchange using trusted hardware,”in
Proceedings of the 2019 ACM SIGSAC Conference on Computer andCommunications Security , 2019, pp. 1521–1538.[45] M. Bowman, A. Miele, M. Steiner, and B. Vavala, “Private data objects:an overview,” arXiv preprint arXiv:1807.05686 , 2018.[46] M. Brandenburger, C. Cachin, R. Kapitza, and A. Sorniotti, “Blockchainand trusted computing: Problems, pitfalls, and a solution for hyperledgerfabric,” arXiv preprint arXiv:1805.08541 , 2018.[47] M. Russinovich, E. Ashton, C. Avanessians, M. Castro, A. Chamayou,S. Clebsch, M. Costa, C. Fournet, M. Kerner, S. Krishna et al. , “Ccf:A framework for building confidential verifiable replicated services,”