A Generic Sharding Scheme for Blockchain Protocols
AA Generic Sharding Scheme forBlockchain Protocols
Research Thesis
Submitted in partial fulfillment of the requirementsfor the degree of Master of Science in Computer Science
Zuphit Fidelman
Submitted to the Senateof the Technion — Israel Institute of TechnologySivan 5779 Haifa June 2019his research was carried out under the supervision of Prof. Roy Friedman, in theFaculty of Computer Science.The generous financial help of the Technion is gratefully acknowledged. ontents
Abstract 1Abbreviations and Notations 31 Introduction 5
Applying the Framework 25 bstract
Bitcoin was the first successful attempt at creating a decentralized cryptographic pay-ment system. Overcoming the trusted third party hurdle, which plagued earlier solu-tions for decades, it paved the way for a whole family of blockchain protocols. However,its fully replicated design prevents Bitcoin from scaling. The scalability crisis is notunique to Bitcoin as this issue extends to most of its descendant blockchain protocols.A new paradigm that enables to systematically scale blockchain protocols seems due.This thesis introduces a formal general framework for scaling blockchain protocols bysharding. The framework is modular and it can be adjusted for different needs or setsof assumptions. We prove that sharded protocols obtained by following our scheme(with correct modules in place) live up to the same safety and liveness guarantees astheir non-sharded counterparts. The proof is general and relies on well-defined speci-fications of certain components. This lays the ground for simple proofs of correctnessfor sharded protocols obtained by following the proposed scheme.The framework is not left as an obscure specification of some high level structure;explicit use is demonstrated by applying it to shard Algorand. As part of this concreteconstruction, a tamper-proof mechanism to assign nodes to shards is introduced. Thismechanism is constructed by using verifiable random functions and can safely withstanda powerful adaptive adversary. 1 bbreviations and Notations n : the number of users in the system m : the number of shards in the system r : an arbitrary round index T X : transaction space
UTXO : unspent transactions output
PKS : public-key space PK r : the set of all public keys in the system at time rC r : the state of an unsharded system when round r begins B r : a block of transactions, approved by an unsharded system in round rC ri : the state of the i th shard when round r begins C ( r ) : the state of a sharded system when round r begins B ri : a sub-block of transactions, approved by the i th shard in round rB ( r ) : a global block of transactions, approved by a sharded system in round rP : an arbitrary blockchain protocol [ i ] : { , , . . . , i }⊥ : the empty block ◦ : concatenation (e.g. a ◦ b ◦ c = abc )3 hapter 1 Introduction
Attempts to create cryptographic payment systems date back to the 1980s and 1990s.Those early endeavors relied on trusted third parties, deeming such solutions securebut centralized. In 2008 a whitepaper titled “Bitcoin: A Peer-to-Peer Electronic CashSystem” [16] introduced the original Bitcoin protocol and coined the term blockchain .The paper was subsequently followed by the Bitcoin network’s launch in early 2009.Bitcoin was the first truly decentralized electronic payment system, bypassing financialinstitutions, mediators and other trusted third parties. Nodes in the Bitcoin networkstore a full copy of a shared transactions ledger. Every new block of approved trans-actions is verified by all participating nodes, as they append it. Nodes compete overfinding the next block, along with a solution to a cryptographic riddle. Overall, thecollective computational power across the network takes the trusted third party’s role.This design marked a breakthrough, overcoming the trusted third party hurdle. Itinspired the creation of an entire new category of blockchain protocols, modeled afterBitcoin. At the same time, this design prevents Bitcoin from scaling, marking its daysas a global payment network numbered. The scalability crisis is not unique to Bitcoin,as this issue extends to most blockchain protocols that followed.A new formal scheme that enables to systematically scale a whole category ofblockchain protocols is the subject of this thesis.
Blockchain Protocols and Their Usage
Blockchain is a data model that keeps track of ordered sequences of transactions ina decentralized and persistent manner. It is known first and foremost as a platformfor implementation of cryptocurrencies, such as Bitcoin [16], Ripple [17], Litecoin andothers. Several other applications adopted this technology as well, most notably smartcontracts [6] and supply chains [1, 12].Different blockchain protocols propose different recipes on how to create an agreedupon shared transaction ledger, often under different sets of assumptions.5 roof-of-Work and Proof-of-Stake
In order to withstand adversarial behavior,blockchain protocols employ various defense mechanisms, suited for a decentralizedsetting. Most notable are proof-of-work (PoW) and proof-of-stake (PoS) approaches.In PoW protocols, a user’s influence on the system is proportional to its computingpower. Such protocols provide safety as long as the majority of computing power inthe network is held by honest participants. In PoS protocols, it is proportional to auser’s stake (e.g. money) in the system and safety depends on the majority of overallwealth being held by honest parties.
Permissioned and Permissionless
Loosely speaking, blockchain protocols can becategorized as permissioned vs. permissionless . In permissioned protocols, such asRipple [17] and Hyperledger Fabric [2], only known nodes, who have permission, cantake part in the system. In permissionless protocols, such as Bitcoin [16], Ethereum [6]and Algorand [7], any node may participate at will.
Forks
Some blockchain protocols, such as Bitcoin [16] and Ethereum [6], may fork ,i.e., there may temporarily exist multiple conflicting versions of the chain until theconflict is finally resolved. Other protocols, such as Algorand [7], do not fork , or mayfork only with negligible probability.
In most existing blockchain protocols the system behaves as one fully replicated statemachine, regardless of the network size. The state progresses by appending one blockat a time to the end of the chain. This raises two main scalability concerns:1.
Bounded throughput, waste of available computing power:
Once additional participants join the network, overall computing power increasesand (presumably) more transactions pend approval. Assuming that the networkhas already been approving transactions at its maximum capacity, the addedcomputing power could theoretically be used to serve those added transactions’approval. However, the intervals between block approvals must remain unchangedand still only a single block at a time, whose capacity is bounded by the protocol,could be appended to the chain. Overall, the network’s throughput remainsbounded by some fixed maximum capacity.This problem is most prominent with Bitcoin and similar PoW protocols. As thecollective computing power increases, nodes have to work proportionally harderin order to simply maintain the same old throughput, let alone increase it.2.
Waste of space:
The entire global state must be locally stored by all nodes that wish to participatein the protocol. In relatively small networks this indeed appears to be essential.6owever, surpassing some desired redundancy threshold, this simply wastes space.Moreover, requiring nodes to store the global state, which could be sizable, in itsentirety leaves lighter nodes with limited storage out of the game, thus impedingdecentralization.In a theoretical global network with millions of participating nodes, it seemsabsurd to store millions of identical copies of an entire enormous global state.Still, most current protocols are designed that way. This distances them frombecoming those idealized decentralized networks used by the masses.
Both scalability issues could be ameliorated through sharding.
Sharding is a design pattern for obtaining scalability of computing systems, by dividingthe computational tasks and the data space into multiple entities, that act mostlyindependently. Sharding blockchain protocols intuitively means dividing nodes in alarge network into smaller groups, where each group is in charge of approving a distinctsubset of pending transactions and of storing a subset of the global state.
This work presents a formal generic framework for sharding blockchain protocols.The proposed framework is formal as it carefully defines all that must be defined andcorrectness is asserted thoroughly. The framework is generic as it follows a modulardesign which enables key components, including the blockchain protocol itself, to beeasily swapped and independently created. The framework is general purposed andsuits both the UTXO model and the account balance model. Moreover, it suits bothpermissioned and permissionless protocols and it mostly targets PoS protocols.The framework executes in two-phased rounds, within a strongly synchronous net-work. During the first phase, disjoint sub-blocks of approved transactions are simulta-neously committed by all shards. During the second phase, relevant cross-shard data issynchronized. Throughout this two-phased execution a specified workflow is followed,while maintaining three invariants. Each invariant is associated with a correspondinginterface that defines a specific module. Applying the framework can be done simplyby following our scheme while providing concrete implementations to all these inter-faces. We provide a formal proof that as long as our scheme is correctly followed, i.e.all interface implementations maintain the desired invariants, then the new shardedprotocol is indeed correct - satisfying safety and liveness. A useful result of this proof,whose only assumption is that supplied modules match their interface specifications,is that any subsequent protocols following our scheme, need only prove their modules’correctness to establish the overall correctness.7e demonstrate the applicability of our framework by sharding Algorand [7, 14]. Aconcrete implementation of all required interfaces is provided and the implementation’scorrectness is fully proved. Then, correctness of the sharded Algorand immediatelyfollows. The methods used to implement the sharding interface are quite general. Aswe already established their correctness, these specific modules could be reused to shardprotocols other that Algorand too, without having to re-establish their correctness.
Resiliency
A major standing issue with sharded protocols is resiliency in presence ofadversarial behavior. A potential attack vector by an adaptive adversary is to move allByzantine nodes into a single shard, thereby controlling that shard despite controllingonly a small minority of nodes overall.The implementation presented in chapter 4 solves this issue by assigning partici-pants time-bounded leases to uniformly selected shards. The assignment mechanism isdecentralized and the generated assignments are both uniformly random and tamper-proof by using verifiable random functions (VRFs) to implement it. These nodes-into-shards assignments are iteratively shuffled, at a shuffling rate corresponding to theadversary’s adaptability, as determined by the environment assumptions model. Theprovided proof establishes that as long as the adversary controls no more than / of allnodes, then it never controls more than / of the nodes in each shard with extremelyhigh probability, even for extraordinarily long executions (e.g. the probability of everencountering failure < − , totalled over a million years of continuous execution).Thus, the suggested implementation maintains safety despite Byzantine mobility. The rest of this thesis is organized as follows:In chapter 2 a formal model expressing the behavior of blockchain protocols is laid out.In chapter 3 a sharding scheme is introduced and its correctness is established, based onthe definitions detailed in chapter 2. In chapter 4 a concrete construction implementingthe sharding scheme is given and two sharded Algorand variants are created. In chap-ter 5 we survey recent work regarding sharding blockchain protocols. We also discussthe different approaches taken towards handling cross-shard transactions and comparethem to the approach taken in this work. In chapter 6 we discuss the applicability ofour framework and relaxation of some its assumptions.8 hapter 2
Preliminaries, Model andDefinitions
We consider a distributed system in which nodes communicate through a stronglysynchronous gossip network. Nodes are computational entities that execute some pro-tocol. Nodes may join and leave the network, or the set of nodes could remain constantthroughout the execution. Each user owns a pair of public and secret keys. Whena user joins the system for the first time, their public key is recorded and this log ispublicly available. Each node is owned and operated by a specific user. We use theterms “node” and “participant” interchangeably when referring to the machine used toparticipate in the protocol. By “user”, we shall refer to the client operating the nodeand using the system (to send payments, for example).The end goal of the system is to run some protocol, denoted P . In our case P would be a blockchain protocol, that can be either permissioned or permissionless. P isassumed to be a PoS protocol where users hold voting power in proportion to their stakein the system. Alternatively, if all nodes are guaranteed to possess equal computingpowers, then P can also be a PoW protocol. In order to simplify matters, from thispoint forward we regard all participants as equally powerful and assume that wealthierplayers have more distinct nodes under their control. Notations: n denotes the number of nodes, ⊥ denotes the empty block, ◦ means concatenation (e.g. a ◦ b ◦ c = abc ) and [ i ] = { , , . . . , i } . Every user owns a pair of public and secret keys. Let u denote some arbitrary userin the system, then P K u denotes u ’s public key and SK u denotes u ’s secret key . P K u is known to all participants and identifies u uniquely. SK u is known only to u and isused for authentication. 9 KS denotes the public-key space . It is the set of all strings that could be used asacceptable public keys. For any u in the system, it holds that P K u ∈ PKS . P K r refers to the set of all public keys in the system at time r . We assume the existence of a unique digital signature scheme
SIG , as defined in [8, 11,14].
SIG is used to generate ( P K u , SK u ) pairs in the system. It provides Sign SK u and Ver
P K u functionality. Specifically:• SIG is a digital signature scheme: – Sign SK u ( x ) ≜ Sign ( SK u , x ) returns a value σ . – Ver
P K u ( x, σ ) ≜ Ver ( P K u , x, σ ) returns True if σ = Sign SK u ( x ) .It is hard to find σ for x that returns True without knowing SK u .It is hard to find x ′ ̸ = x such that Ver ( PK u , x, σ ) = Ver ( PK u , x ′ , σ ) = True .• SIG is unique: – It is hard to find
P K u , x, σ and σ ′ ̸ = σ such that: Ver ( P K u , x, σ ) = Ver ( P K u , x, σ ′ ) = True .This holds when SK u is known, even for maliciously selected ( P K u , SK u ) , σ = σ P K u ( x ) is the unique signature for x and P K u .We assume the existence of an efficiently computable ideal hash function, H .• H is available to all participants and they can run it locally.• H is modeled as a random oracle. It takes an arbitrarily long input and returnsa hashlen bits long binary string. When x is chosen uniformly at random: – H ( x ) is regarded as a uniformly distributed integer between 0 and hashlen − . – . H ( x ) is regarded as a uniformly distributed binary between 0 and 1.It should be pointed out that we adhere to the same cryptographic assumptions Micaliet al. make in Algorand [7, 14], as we extend some of their constructions. The adversary is a polynomially bounded entity, able to attack any target user’s nodeand attain total control over their actions. It cannot successfully forge digital signaturesby uncorrupted users and cannot simultaneously control or attack more than a fraction ≤ b < of active users; in chapter 4 we set b = 1 / .A participant under adversarial control is considered Byzantine . A Byzantine par-ticipant may follow the protocol or it may deviate arbitrarily. All non-Byzantine par-ticipant are honest and meticulously follow the protocol. Gaining control over a set10f target participants takes t takeover rounds to complete. During these rounds targetednodes are still honest. Once an attack has been set in motion, it cannot be altered oraborted; the adversary must see it to completion.We sometimes refer to an adversary as being either static or adaptive. A staticadversary can introduce malicious participants to the network, but it can never gaincontrol over honest ones. Equivalently, for a static adversary t takeover → ∞ . Anadaptive adversary is mobile, it can actively attack honest participants and gain controlwithin a finite t takeover ∈ N + number of rounds. If an adaptive adversary is at capacity(simultaneously controls or attacks b of all participants) it must retire control over aparticipant in order to initiate an attack on another. We discuss blockchain protocols that execute in a sequence of consecutive rounds.Round r ∈ { , , ... } results in a block of approved transactions, denoted B r , appendedto the chain. The initial state of the system is determined by a genesis block, B .Any P we consider requires an honest majority of . < h P ≤ of all participants,i.e. ≤ b < − h P . Then, it provides safety and liveness guarantees: all admittedblocks must be admissible (soon to be defined) and some of them non empty. Welimit the discussion to protocols that do not fork . When round r completes, B r isfinalized and its content is indisputable.Let us introduce some definitions in order to dive into a more rigorous discussion: Definition 2.1
Transaction Space : Let P be some blockchain protocol. TX P denote the transaction space of P , or simply TX since P is known. A transaction is an atomic operation, defined per P ’s logic.Any transaction tx that could be created at some point in some execution of P sat-isfies tx ∈ TX . Transactions contains various data fields, dictated by P . At the bareminimum, a tx in all protocols we consider is assumed to include the following fields:• tx.from : stores the identity of the user who created tx .This field must include exactly one public key, denoted PK from ∈ PKS . – In a payment system PK from would be the public key of a user sending apayment, or transferring some resource it owns.• tx.to : stores identity of the user who receives the output of tx .This field must include at least one PK to ∈ PKS , more if tx has multiple outputs. – In a payment system PK to would be the public key of a user receiving apayment, or some other resource.• tx.sig : stores Sign SK from signature on this unique tx , used for authentication. This restriction could later be lifted, see discussion section. It simplifies matters for the time being. if PK from is a multisig address, this requires m-of-n signatures.Any tx ′ with either of these fields is missing or invalid satisfies tx ′ / ∈ TX . Note that asyntactically incorrect transaction is easily detectable and tx ′ is immediately discarded. Definition 2.2
Block : A block of transactions is any set X such that X ⊆ TX . Usually (and unless statedotherwise), blocks are viewed as lists or ordered sets of transactions and then X ∈ TX N .Sometimes, blocks can be viewed as unordered sets, in which case X ∈ TX . Definition 2.3
State, Context : C r denotes the context of the protocol when round r begins. The admissibility of anyblock to be added to the chain in round r is determined by P with respect to C r .The state of the system is defined by an ordered list of approved blocks. It is normallyassumed that the context and the state equal, therefore we usually use C r to denoteboth. The state at time r is recursively defined by C r = C r − ◦ B r − with C = ⊥ .In settings where blocks are ordered sets, the state is totally ordered , C r ∈ ( TX N ) N + .In settings where blocks are unordered sets, the state is weakly ordered , C r ∈ (2 TX ) N + .Explicitly: take some tx i ∈ B i , tx j ∈ B j . If i < j then tx i ’s approval predates tx j ’s. If i = j and blocks are ordered sets, then tx i , tx j approval order is induced by the internalorder in that block. If i = j and blocks are unordered sets, then tx i , tx j are consideredto be simultaneously approved.We refer to B as the genesis block and to C = ⊥ ◦ B as the genesis state, bothare assumed to be valid. Applying an admissible block on a valid state transitions thesystem into a new valid state. This leads us to the next definition. Definition 2.4
Admissible Blocks, Valid States, Legal Executions : A block is admissible at time r if all transactions in that block, in order of appearance,could be appended to the chain and admitted to the system, in compliance with theprotocol’s logic. P is assumed to include a well defined deterministic function named Verify . It takes an ordered set of transactions along with a context C r and returns True if the set is admissible with regards to C r or False otherwise. Formally:
Verify : ( TX N , ( TX N ) N + ) → { True , False } C = ⊥ is a valid state , the rest is inductively defined such that: Verify ( B r , C r ) = True ⇐⇒ C r +1 = C r ◦ B r is valid , B r is admissible at time r An execution is legal if and only if all of its states are valid. P must never permit invalid intermediate states. Any prefix of an admissible blockmust also be admissible on its own, with respect to the same state. Take B r such that Verify ( B r , C r ) = True and any prefix of B r , denoted X . Then, Verify ( X, C r ) = True always holds. Additionally,
Verify ( B r \ X, C r ◦ X ) = True should also hold.12 emark.
In practice, blocks contain additional bookkeeping records (such as headers,PoW etc.) on top of the list of approved transactions. However, our interest lies inthe state of the ledger, as determined by those ordered sets of transactions. Therefore,in order to keep this focus, we abstract. Any additional bookkeeping that might berequired throughout this work will be handled separately and explicitly. Alternatively,consider
Verify as one component of a more comprehensive
Validate method.
Definition 2.5
Intra-Block Transactions : tx ∈ B r is considered an intra-block transaction if its input depends on the output of an-other transaction tx ′ ∈ B r preceding it within the same block: B r = [ . . . , tx ′ , . . . , tx, . . . ] such that Verify ( B r , C r ) = True and
Verify ( B r \ { tx ′ } , C r ) = False . Block Permutation Effect:
The existence of intrablock transactions gives rise tocases where two blocks, B r and B rπ , consist of equal sets of transactions in differentorderings, such that Verify ( B r , C r ) = True while
Verify ( B rπ , C r ) = False . Take, forexample, B r = [ . . . , tx ′ , . . . , tx, . . . ] from definition 2.5 such that Verify ( B r , C r ) = True and
Verify ( B r \ { tx ′ } , C r ) = False . Define B rπ = [ . . . , . . . , tx, . . . , tx ′ ] = [ B r \ { tx ′ } ] ◦ tx ′ . Verify ( B rπ , C r ) = False follows from definition 2.4, as states are totally ordered.Such cases may cause ambiguity and raise additional difficulties when parallelizing
Verify or transforming a system that outputs one fully replicated block at a time intoone producing several sub-blocks, separately and simultaneously. For these reasons,from this point forward, we consider only protocols (or variants of protocols) that donot permit intrablock transactions . Specifically, we require
Verify to always return
Verify ( B r , C r ) = Verify ( B rπ , C r ) for all B r , C r and any possible permutation π . Lemma 2.2.1. If P does not permit intra-block transactions, then any subset X ⊆ B r of an admissible B r also makes an admissible block, applicable instead of B r at time r .Formally: Verify ( B r , C r ) = True ⇒ ∀ X ⊆ B r : Verify ( X, C r ) = True .Proof.
For any permutation π it holds that Verify ( B r , C r ) = Verify ( B rπ , C r ) = True .Consider a permutation π where X appears as the first | X | transaction within B rπ , insome internal order. Any prefix of B rπ , in particular one of length | X | , must also beadmissible. We do not permit intra-block transactions , i.e. we do not allow a resourcereceived within the round to be immediately spent. All resources being put to use inround r must already be admitted when the round began, appearing in C r . Then, theinternal order of B r plays no role. Funds or resources received during round r becomeavailable to spend starting round r + 1 , as they appear in C r +1 . This strategy appearsreasonable when rounds are frequent enough, it resolves such ambiguity altogether. Removing this assumption, sharded protocols created by the framework still maintain safety. Seediscussion section for more details.
Verify takes the form:
Verify : (2 TX , (2 TX ) N + ) → { True , False } . Definition 2.6
Conflicting Sets, Competing Transactions : Let tx , tx ∈ TX be two distinct transactions. If there exists some X ⊆ TX and a state C ⊆ (2 TX ) N + such that Verify ( X ∪ { tx } , C ) = True and
Verify ( X ∪ { tx } , C ) = True but
Verify ( X ∪ { tx , tx } , C ) = False then tx , tx are competing transactions and X ∪ { tx , tx } is a conflicting set (note that this definition generalizes the notion ofdouble spending ).Let X = { tx , tx , ..., tx k } be a conflicting set s.t. ∀ tx i ∈ X : X \ { tx i } is anadmissible set, then X is minimal . Observe that all distinct pairs tx i , tx j ∈ X are competing transactions . Definition 2.7
Support Sets : The support set of X ⊆ TX at time r is the minimal subset of C r that is required tocorrectly determine whether any X ′ ⊆ X could be accepted or must be rejected, per P ’s logic. Denoted Supp r ( X ) ≜ Supp ( X, C r ) ⊆ C r , this is a weakly ordered set. Theorder between elements is induced by their order in C r . Supp ( X, C r ) satisfies ∀ X ′ ⊆ X : Verify ( X ′ , C r ) = Verify ( X ′ , Supp ( X, C r )) Observe that:1. ∀ X ′ ⊆ X : Supp r ( X ′ ) ⊆ Supp r ( X ) ∀ C r ⊇ Y ⊇ Supp r ( X ) : Verify ( X , Supp r ( X )) = Verify ( X , Y ) are both implied.Note that Supp definitions are modeled after and rely on internal P behavior.Since minimality is a requirement and definitions are P -specific, Supp is somewhatillusive. Therefore, we also define
Supp ( X ) ≜ Supp ( X, TX ) ⊆ TX which splits TX into transactions that may appear in X ’s support and those that never will. Then, Supp ( X, C r ) ⊆ Supp ( X ) ∩ C r ⊆ C r always holds and is easier to use. Under the UTXO model, if tx , tx compete over the same utxo , they certainly conflict. Take, forexample, X = ∅ then X ∪ { tx , tx } is a conflicting set (it is also minimal). Generally, a resource maybe used or accessed more than once. Consider a venue that can hold up to 200 distinct visitors at atime. Visitors compete over the same resource (entrance to the venue), yet as long as fewer than 201of them coexist, no conflict arises. Let X be a set of 201 distinct visitors asking to enter an empty venue that is capped at 200. Then X is a conflicting set, any proper subset of X (in particular one of size 200) is admissible and therefore X is minimal. The same analysis applies to the sum of monetary transactions under the account balancemodel. Building on top of these definitions makes our scheme general enough to be useful under boththe UTXO and account balance models. .3 Sharded Blockchain Protocols When dealing with sharded protocols, we slightly adjust some terminology. Let S denote a sharded blockchain protocol and m denote the number of shards. Then S ’sparticipants are scattered between those m shards. Nodes that participate in the i th shard run some intra-shard sub-protocol, denoted S i , that sequentially outputs blocks ofthe form B i , B i , . . . , B ri , . . . . We generally assume that S is synchronized. Specifically,if S i hasn’t decided on B ri within the r th time frame, then B ri = ⊥ . Definition 2.8
Global Blocks and Local Blocks : B ( r ) denotes the r th global block . B ( r ) is a distributed block rather than a fully replicatedone. We define B ( r ) = B r ◦ . . . ◦ B rm or simply B ( r ) = S mi =1 B ri , since we consider B ( r ) to be simultaneously approved. We refer to B r , . . . , B rm as sub-blocks or local blocks .Normally, B ri is locally stored within S i ’s nodes. The sharded protocol S is assumedto allow its participants to query any S j regarding the content of any B rj . Definition 2.9
Global State and Local Contexts : Given an environment in which the system is not fully replicated, some participantsview of it may differ from others. This requires us to make a clear distinction betweencontexts and the state of the system. C ( r ) denotes the global state when round r begins. The global state of a shardedsystem follows the same definition a state in an unsharded system follows. Specifically,it is recursively defined by C ( r ) = C ( r − ◦ B ( r − with C (0) = ⊥ . The global state isthe full ledger, defined by an ordered list of approved global blocks. C ri denotes the local context (sometimes called local state) of the i th shard attime r . This is some subset of the global state C ( r ) , stored in S i nodes. Its partici-pants execute S i and produce B ri under that local context C ri . We specifically require Verify ( B ri , C ri ) = True to hold, the rest of C ri definition is dependant on the protocol S at hand.We use this different notation so to clearly distinct between sharded and unshardedsystems. Any definition relating to the state (e.g. support sets) remains unchanged inthe sharded setting, other than using C ( r ) instead of C r .156 hapter 3 Shardder: The ShardingFramework
This chapter presents Shardder, a framework for transforming an overcrowded networkrunning a single instance of P into a network that runs several specialized sub-instances,working together in cooperation. These sub-protocols, denoted P , · · · , P m , are simul-taneously executed and produce m disjoint sub-blocks in parallel. The overall capacityof the network increases by up to m -fold, while preserving P ’s original logic (as definedby its Verify predicate). Following a careful division of labor, P i executes correctly fromboth local and global perspectives. This is achieved despite the fact that participantsstore only a fraction of the global state locally.Shardder is generic in a sense that while the framework follows a preset workflow,several key components are black-boxed into well defined interfaces. Each interfacelays out a set of requirements that any adequate implementation must fulfill. Theframework-user, i.e. the deployer , must provide implementations of three interfaces, aconcrete protocol P and a granularity parameter m ∈ N + . Plugging these componentsinto the framework results in defining SP , a Sharded P protocol. Swapping someinterface implementation results in creating a different sharded variant of P . Differentvariants might be better suited for different settings, as demonstrated in chapter 4.Any sharded variant created by applying the framework lives up to original protocol’ssafety and liveness guarantees, provided that all interface implementations (provably)meet all specified requirements, this we prove in section 3.4.17 .1.1 Assumptions P is assumed to be some blockchain protocol that:1. Does not permit intra-block transactions.2. Does not fork. Block approval is final and cannot be overturned.3. Provides safety and liveness, as long as at least . < h P ≤ of its participantsare honest.See discussion in section 6.1 regarding relaxing assumptions 1 and 2. n - denotes the initial number of participants. m - denotes the number of shards set by the deployer. m ∈ N + s.t. n ≫ m . h - denotes the honesty rate the resulting SP assumes. It must hold that h P ≤ h ≤ .The precise value of h is set by the deployer, in accordance with their concreteinterface implementations, acceptable values for m and adversarial assumptions.We assume that the number of participants remains θ ( n ) throughout the execution anddoesn’t fluctuate too wildly. Permissionless environments in which specific users mayjoin or leave are suitable, as long as the number of active users remains stable. Partition TX is partitioned into TX , . . . , TX m according to the Partition interface.By definition, the sole authority regarding approval of pending transactions fromwithin TX i is P i . Membership
Participants are divided into m groups by the Membership interface.It must ensure that at least h P out of every P i ’s participants are honest, thusenabling them to execute the underlying P . Sync
Global changes are tracked and local states are updated, using the
Sync interface. Adjusting the framework for networks with significant fluctuations requires to dynamically adjust m for n/m as n deviates. This extension is left for future work, see discussion section. .2 Algorithms and Workflow To construct the system, concrete implementations of all abstractions used by theframework are required. This includes the code to a blockchain protocol P and tonon-abstract classes implementing the sharding interfaces. Additionally, genesis datais needed to set the genesis state. The framework takes a standard valid P -formattedgenesis block, B , for this purpose.During bootstrap, m specialized instances of P , referred to by P , . . . , P m , are cre-ated. The instances are then initialized so to reflect the appropriate genesis state: B data is broken into m shards, according to Partition . Thus, transforming B intoan equivalent distributed block B (0) s.t. tx ∈ B ⇔ tx ∈ B (0) . Algorithm 1:
Bootstrap
Shardder
Input :
P, B , nodes , m ∈ N + , Partition , Membership , Sync
Output :
Bootstrapped System define Part m = Partition . Part ( T X, m ) = { T X , ..., T X m } ; init Membership and
Sync ; forall i ∈ [ m ] do C i = ⊥ ; B i = B ∩ T X i ; RS i = Sync. collectSupport i ( B ) ; construct P i and set C i = C i ◦ B i ◦ RS i ; end scatter nodes between P , . . . , P m according to Membership ; return P , . . . , P m , Part m , B (0) = S mi =1 B i ; After bootstrap, the framework follows Algorithm 2 and executes in sequential twophased rounds. Every round yields a new global block of approved transactions, whichtransitions the system into its next state.During the first phase, the P phase, P , . . . , P m simultaneously execute a singleblock creation round. Every P i is essentially an instance of P with protection againstunauthorized participants, as described in Algorithm 3. Each P i executes under itslocal context C ri and takes as input pending transactions from T X i , its designatedpartition. Consequently, P i outputs a locally admissible block B ri ⊆ T X i such that Verify ( B ri , C ri ) = T rue . An empty sub-block is decided by default if P i times out. Empty blocks are always admissible. B ( r ) is globally known and the nextglobal state C ( r +1) = C ( r ) ◦ B ( r ) is determined. Then, during the sync phase and beforetransitioning to the next round, shards utilize the Sync interface in order to updatetheir local states to include the up-to-date
Supp ( TX i , C ( r ) ◦ B ( r ) ) . Algorithm 2:
Shardder Workflow
Input : a stream of pending transactions
Output : B (1) , B (2) , ... Bootstrap ; r = 1 ; while safety holds do // protocol phase execute P , . . . , P m on TX , . . . , TX m respectively; set B ( r ) = S mi =1 B ri and make public; // sync phase forall i ∈ [ m ] do RS ri = Sync. collectSupport i ( B ( r ) ) ; C r +1 i = C ri ◦ B ri ◦ RS ri ; end adjust nodes’ memberships; r = r + 1 ; endAlgorithm 3: P i block creation round, as executed by P K u Input : a set of pending transactions T ⊆ TX i Output : an approved block B ri ⊆ T let σ u be a proof binding P K u and i , generated by Membership . getMembership ; while running round r of P with context C ri do attach ( P K u , σ u , i, r ) to all messages directed to the underlying P ; foreach received protocol message from node w do if Membership . verifyMember ( P K w , σ w , i, r ) = False then discard message; else pass message to underlying P ; end end return B ri ; Take notice of the fact that P i ’s synchronization with B ( r ) does not imply that every node mustrun or acquire every transaction. While this is a technical design issue, sub-blocks could batch togethertransactions that belong in distinct shards’ support. Then, nodes need to download nothing more thanthe relevant sections of each sub-block. .3 Interfaces, Methods and Invariants The sharded protocol’s correctness relies on three invariants being kept throughout itsexecution. In order to plug each invariant into the framework, an interface abstractionis used. The interfaces specify a set of methods that must be implemented, as thosemethods are invoked by the framework. As long as concrete implementations suppliedby the deployer satisfy the following requirements, SP ’s correctness immediately followsfrom the proof in Section 3.4. Interface Name:
Partition
Role:
Splits the workload into well defined pools, each allotted to a specific shard.
Key Methods: Part : (2 TX , N + ) → (2 TX ) N + Part m ( TX ) = Part ( TX , m ) = { TX , ..., TX m } partitions TX into m subsets.2. whichPart : TX → N + A companion function to
Part m s.t. ∀ tx ∈ TX i : whichPart m ( tx ) = i . Invariant : Partition Function:
Part is a partition function. ∀ m ∈ N + it must satisfy: Part ( TX , m ) = { TX , ..., TX m } such that TX = S mi = TX i , ∀ i ∈ [ m ] : TX i ̸ = ∅ and ∀ i ̸ = j : TX i ∩ TX j = ∅ .2. Conflict Preservation / Aggregation:
Part must assign any pair of competing transactions into the same partition. tx, tx ′ ∈ TX are competing transactions ⇒ whichPart m ( tx ) = whichPart m ( tx ′ ) . Definition 3.1
Remote Support : The remote support of TX i in B ( r ) is RemoteSupp i ( B ( r ) ) ≜ { Supp ( TX i ) ∩ B ( r ) } \ B ri . Interface Name:
Sync
Role:
Enables participants to update local states so to reflect global changes.
Key Method: collectSupport : (2 TX , N + ) → TX collectSupport i ( B ( r ) ) = collectSupport ( B ( r ) , i ) return RS ri ⊆ { B ( r ) \ B ri } such that RemoteSupp i ( B ( r ) ) ⊆ RS ri always holds.21 nvariant : Self Containment must hold throughout the first phase of every round.
Self Containment holds ⇔ ∀ i ∈ [ m ] : Supp ( TX i , C ( r ) ) ⊆ C ri . Theorem 3.1.
If a collectSupport function that returns RS ri such that ∀ i ∈ [ m ] , r ∈ N : tx ∈ RemoteSupp i ( B ( r ) ) ⇒ tx ∈ RS ri is used, then self containment holds throughout theexecution.Proof. By induction:
Base: C (0) = C i = ⊥ . Closure ( r > ): consider progress made by the system: C ri = C r − i ◦ B r − i ◦ RS r − i as determined by Algorithm 2. C ( r ) = C ( r − ◦ B ( r − as defined in 2.9.Take any x ∈ Supp ( TX i , C ( r ) ) . Either x ∈ C ( r − or x ∈ B ( r − .• If x ∈ C ( r − then x ∈ Supp ( TX i , C ( r − ) .By induction hypothesis, Supp ( TX i , C ( r − ) ⊆ C r − i and we get x ∈ C r − i .• When x ∈ B ( r − we have two cases to consider: – if x ∈ B ( r − ∩ TX i then x ∈ B r − i . – otherwise, x ∈ { B ( r − \ B r − i } meaning x ∈ RemoteSupp i ( B ( r − ) ⊆ RS r − i Overall, x ∈ C r − i ◦ B r − i ◦ RS r − i = C ri ⇒ Supp ( TX i , C ( r ) ) ⊆ C ri Interface Name:
Membership
Role:
Assigns nodes to shards in a tamper-proof manner.
Key Methods: getMembership : (( PK , SK ) , N + , ... ) → ( N + , { , } ∗ ) getMembership (( PK u , SK u ) , r, . . . ) → ⟨ i, σ u ⟩ where i ∈ [ m ] .generates σ u that uniquely binds PK u with P i verifyMember : ( PK , { , } ∗ , N + , N + , ... ) → { True , False } verifyMember ( P K u , σ u , i, r ) → True getMembership (( PK u , SK u ) , r ) = ⟨ i, σ u ⟩ False otherwiseverifies σ u authenticity, for PK u ∈ PKS , i ∈ [ m ] , r ∈ N + such thatany σ ′ u not issued by getMembership returns False with extremely high probability.
Definition 3.2 PK ri − the set of P i participants at time r : PK ri = { PK u ∈ PK r | verifyMember ( PK u , σ u , i , r ) = True } Invariant :
Honest majority must hold at all times with extremely high probability.honest majority holds ⇔ ∀ i ∈ [ m ] : at least h P · | PK ri | of P i ’s participants are honest.22 .4 Correctness In this section we prove that any SP created by properly following the algorithmspresented in section 3.2 is indeed correct. Namely, SP satisfies the same safety andliveness guarantees that an instance of P would have. SP is assumed to be correctly deployed, with all interface implementations satisfy-ing the requirements listed in section 3.3. We examine the course of execution in twoparallel P and SP networks that are executing under equal terms. Specifically, bothnetworks are bootstrapped with the same genesis block, B , and are running with n participants, out of which at least h · n are honest. Block-size limitations are disregardedand comparing SP with an unbounded version of P , we demonstrate equivalent safetyand liveness. Recall that:• An execution e of P is legal ⇔ every B r outputted by e satisfies Verify ( B r , C r ) = True .• An execution e ′ of SP is legal ⇔ every B ri outputted by e ′ satisfies Verify ( B ri , C ri ) = True . Lemma 3.4.1 ( P ⊆ SP ) . For any legal execution e of P that runs with input T ⊆ T X and outputs B , B , B , ... , there exists an equivalent execution e ′ of SP that runs forthe same number of rounds, with the same input T and outputs B (0) , B (1) , B (2) , ... s.t. ∀ r ≥ B ( r ) = B r .Proof Sketch. Given e executed by P , we show the existence of a legal execution of SP ,denoted e ′ , that always decides B ( r ) = B r . We inductively construct e ′ , round by round. r = 0 : Let’s explicitly demonstrate that Algorithm 1 always returns with B (0) = B :This is indeed the case because ∀ i ∈ [ m ] : B i = B ∩ T X i , B (0) = S i ∈ [ m ] B i and since Part m is a partition function B = S i ∈ [ m ] B ∩ T X i = S i ∈ [ m ] B i = B (0) . r > 0 : Take the inductively constructed e ′ that agrees with e up to round r − .Having B (0) = B , . . . , B ( r − = B r − ⇒ C ( r ) = C r we need only to extend e ′ by oneadditional round where B ( r ) = B r . P does not allow intra-block transactions, therefore: Verify ( B r , C r ) = True ⇒ ∀ X ⊆ B r : Verify ( X, C r ) = True
The LHS holds since e is a legal execution. Then the RHS holds too, it holds inparticular for ∀ i ∈ [ m ] : X = B r ∩ T X i . Clearly B r ∩ T X i ⊆ T X i , we have establishedthat C ( r ) = C r and since self containment holds Supp ( T X i , C r ) ⊆ C ri . This gives Verify ( B r ∩ T X i , C ri ) = Verify ( B r ∩ T X i , C r ) = True , which makes B ri = B r ∩ T X i alegal P i output for round r , for every P i . Overall B ( r ) = S i ∈ [ m ] B ri = S i ∈ [ m ] B r ∩ T X i = B r makes legal round r output in e ′ . 23 emma 3.4.2 ( SP ⊆ P ) . For any legal execution e ′ of SP that runs with input T ⊆ T X and outputs B (0) , B (1) , B (2) , ... , there exists an equivalent execution e of P that runsfor the same number of rounds, with the same input T and outputs B , B , B , ... s.t. ∀ r ≥ B r = B ( r ) .Proof. Given e ′ executed by SP , we show that an execution of P that always decides B r = B ( r ) is legal. If we can find some legal e ′ where Verify ( B ( r ) , C ( r ) ) = False forsome r , then the inductively constructed e satisfies Verify ( B r , C r ) = False , deeming itillegal. By showing this can never be the case, we prove e is legal and SP is safe.We have shown in Lemma 3.4.1 that B = B (0) and the genesis block is assumed to bevalid. Let us assume, for the sake of contradiction, that an execution e ′ of SP in whichthere is some round r > where Verify ( B ( r ) , C ( r ) ) = False exists. If several such roundsexist, let r denote the earliest one ⇒ ∀ r ′ ≤ r − Verify ( B ( r ′ ) , C ( r ′ ) ) = True holds.Therefore, we can inductively construct e , an execution of P that reaches round r with C r = B ◦ ... ◦ B r − = B (0) ◦ ... ◦ B ( r − = C ( r ) and ∀ r ′ ≤ r − Verify ( B r ′ , C r ′ ) = True . Claim 3.4.3. ∀ i ∈ [ m ] : Verify ( B ri , C ( r ) ) = True
Proof of claim 3.4.3 : Verify ( B ri , C ri ) = True must hold, because B ri is the output of aninstance of P running with honest majority under context C ri . Additionally, self containment holds, ergo Supp ( B ri , C ( r ) ) ⊆ C ri ⇒ Verify ( B ir , C ri ) = Verify ( B ir , C ( r ) ) = True .Back to the main proof, by assumption
Verify ( B ( r ) , C ( r ) ) = False . Therefore, thedistributed block B ( r ) makes a conflicting set. Let X ⊆ B ( r ) be a minimal conflictingset. From Claim 3.4.3 it must be that X contains items from at least two distinctblocks, B ri , B rj and i ̸ = j .Let us pick tx i , tx j ∈ X such that tx i ∈ B ri , tx j ∈ B rj and i ̸ = j . Since X is minimal , tx i , tx j are competing transactions and must belong to the same partition,as Part is conflict preserving . However, we specifically selected tx i ∈ B ri ⊆ T X i and tx j ∈ B rj ⊆ T X j where i ̸ = j ⇒ T X i ∩ T X j = ∅ , as Part is a partition function .This is a contradiction.
Theorem 3.2 ( SP = P ) . For any legal execution of P there exists a legal equivalent execution of SP and viceversa.Proof. Lemma 3.4.1 above gives P ⊆ SP and Lemma 3.4.2 gives SP ⊆ P .24 hapter 4 Applying the Framework
In this chapter the framework is put to its first use. We demonstrate the steps requiredto properly apply the framework, create a sharded protocol and to assert correctness:First, an adequate protocol to be sharded is selected. Next, implementations ofall required interfaces are specified and their correctness is proved. Finally, a specificvalue for h is set and the values m can take are analyzed. By following these steps, thesharded protocol is well defined and its correctness is fully asserted.The selected blockchain protocol for our demonstration is Algorand. Algorand, by Micali et al., [7, 14] is an efficient PoS blockchain protocol with negligibleprobability of creating forks. Its efficiency stems mostly from blocks being admittedto the chain by means of cryptographic sortition, rather than a network-wide race tofind PoW. Adjusted for their stake in the system, users are randomly sampled eitheras block proposers or as committee members, once per round. Only a handful of usersmay be selected as block proposers. Each selection comes with a priority and thehighest priority proposer is the round leader. The expected committee size is largerand committee members run a Byzantine Agreement protocol named
BA⋆ [5] to agreeon the leader’s identity, the proposed block and its admissibility. If any fails, consensuson the empty block is reached.Algorand’s cryptographic sortition mechanism is implemented by verifiable randomfunctions (VRFs)[15]. A selection procedure, executed locally and non-interactivelyby the user itself, returns a result res and a proof for that result π ru . Selected usersmay claim their role as block proposers or committee members by revealing ⟨ res , π ru ⟩ as credentials. Specifically: ⟨ res , π ru ⟩ ← Sortition (( PK u , SK u ) , seed r , role ) requires aglobal randomness seed along with the user’s cryptographic keys in order to create π ru ,a proof tying a randomly sampled res with P K u , role and round r . The authenticityof any claimed ⟨ res, π ru ⟩ pair may be verified by VerifySort ( PK u , seed r , role , res , π ru ) .25ote that ⟨ res , π ru ⟩ is publicly verifiable, anyone can execute VerifySort , as it takesonly public parameters. Since
Sortition is non-interactive and takes privately keptdata, known only to the user ( SK u ), selection is discrete . This helps to mitigatetargeted attacks. Algorand’s sortition is considered tamper-proof since a ⟨ res , π ru ⟩ pairgenerated in any way other than invoking Sortition (( PK u , SK u ) , seed r , role ) properly,will not pass VerifySort ( PK u , seed r , role , res , π ru ) with non-negligible probability. The following components are assumed to be globally known and publicly available:• H - a cryptographic hash function that doubles as a random oracle.• SIG - a unique digital signature scheme.• seed r - a random seed, available at time r .Then: π ru = Sign SK u (( seed r , role )) and res = H ( π ru ) .More specifically: a user u , whose PK u ∈ PK r , can generate a unique π ru by signing thestring ( seed r , role ) using SIG . Then, res is created simply by hashing π ru . Clearly only u can generate a valid π ru , but anyone can verify π ru by calling Ver
P K u (( seed r , role ) , π ru ) .Then, res is easily asserted by re-hashing π ru . PK u is selected as a block proposer if res = H ( Sign SK u ( seed r , proposer )) ≤ τ proposer where τ proposer is a known threshold. The leader is the user with minimal res value.Committee members are similarly selected. Since H is a random oracle and seed r isunpredictable, the output of each H ( Sign SK u ( seed r , role )) is uniformly distributed andusers are randomly sampled.The initial seed is bootstrapped using some distributed random number generationalgorithm. The algorithm is collectively carried by the initial set of users, once genesistime users register their public keys. Succeeding rounds’ random seeds are efficientlygenerated by setting either seed r +1 ← V RF SK u ( seed r ◦ r + 1) when PK u is the roundleader and proposed a valid B r or seed r + ← H ( seed r ◦ r + ) if BA⋆ decided B r = ⊥ . Algorand can withstand an adaptive poly-time adversary that controls users holdingup to / of overall funds in the network. This adversary is seen as a single entity withperfect coordination and total control over the actions of all malicious participants. Itcan initiate an attack and turn an honest participant malicious within a single round( t takeover = 1 ). In order for Algorand to operate properly, at least / stake in thesystem must always be under honest ownership, i.e. h Algorand = 2 / .26 .2 Sharding Algorand - Introducing Algoshard Two sharded variants of Algorand,
Algoshard0 and
Algoshard1 , are introduced in thischapter. This section provides a high level description and intuition regarding eachprotocol. Actual implementations appear in section 4.3, in which both versions ofAlgoshard are presented side by side, followed by proofs of their correctness.We set h = 3 / in both variants, suited for h Algorand = 2 / . The same Partition module is reused by both variants, who split the workload between shards identically.Each variant has its own
Sync and
Membership implementations, suited for differentadversarial assumptions, as detailed below.
Algoshard0 abides Algorand’s strict adversarial model, specifically setting t takeover = 1 .To withstand such a powerful adversary, one able to corrupt any participants withina single round, membership leases are ephemeral. Implemented in Eager Membership ,every new round sees a new uniform assignment of all participants into all shards. Assignments are randomly drawn, new assignments are independent of previous ones.Since Algoshard0 nodes continuously hop from one random shard to another, neverstanding still, swift and unpredictable shard transitions must be accommodated. Thisis achieved by storing the entire global state locally, as implemented in
Eager Sync .In that sense, Algoshard0 is logically sharded rather than physically sharded. Theblock creation process is successfully parallelized, yet the global state remains fullyreplicated by all nodes. Overall, Algoshard0 can scale throughput whilst resisting anextremely able adversary.
Algoshard1 caters for a more relaxed adversary, with a considerable t takeover , spanningan order of one day for example. This setting makes way for longer lasting membershipleases, as attacks take longer to complete. Implemented by Lazy Membership , the pro-tocol’s execution begins with a uniform assignment of all participants into all shards.Then, existing assignments gradually shift, as a small set of participants is required toattain new (uniformly selected) memberships in order to continue participating in theprotocol. Memberships are sticky and cannot be manipulated by an adversary. Uponjoining the system, each participant is assigned a sort of “personalized membershipepoch” timeframe that sprawl specific rounds. Repeated invocations of getMembership within the same epoch always return the same result. New users must sit out their firstepoch, before being permitted to participate. This helps to mitigate join-leave attacks. To be precise, a different certificate could be set for every step in every round of Algorand. Thisfollows immediately from Algorand’s player replaceability property.
27y definition, members of P i need nothing more than the (global) support of TX i in order to execute P i correctly. Since Lazy Membership puts lengthy memberships inplace, swiftness of transition and thrashing are no longer much of a concern. Thus, localstorage of the entire global state becomes redundant. This is where
Lazy Sync comesinto play, as it collects only the missing remote support items of TX i , those needed tomaintain Supp ( TX i ) . Then, local states become proper subsets of the global state, forany value of m > . Moreover, our selection of Part suggests that larger values of m reduce the relative size of local states, since each shard stores fewer historical records .Moving shards essentially means re-joining the system, a task that may take a fewrounds to complete. Still, this task is infrequent and randomly distributed between thenodes (epochs are uniformly assigned), therefore should not be obstructing operability.Overall, Algoshard1 is both physically and logically sharded. The block creationprocess is parallelized while also local states are proper subsets of the global state.Adversarial assumptions are somewhat relaxed in comparison to Algorand, still theadversary is extremely apt.The analysis of Algoshard’s Membership implementation, provided in section 4.3.3,shows that networks with more participants support larger values of m . In denselypopulated networks, this implies both increased throughput and reduced local contextssizes. Assuming that networks with heavier activity and heftier ledgers also containmore participants, Algoshard1 truly scales. Algoshard0 and
Algoshard1 implementations are introduced side by side.•
Partition (section 4.3.1) is shared by both protocols. This section is not divided.•
Eager Sync (in 4.3.2) and
Eager Membership (in 4.3.3) implement Algoshard0. – Both appear on the left side of the relevant section.•
Lazy Sync (in 4.3.2) and
Lazy Membership (in 4.3.3) implement Algoshard1. – Both appear on the right side of the relevant section.Proof of each interface correctness follows its implementation.In section 4.3.3 we analyze how to properly select adequate values for m . Remark.
Provided implementations (and proofs) for both
Partition and
Sync aregeneric and may be reused as-is with any suitable protocol. However,
Membership ’simplementation is specifically tailored to Algorand, as it piggybacks on the publicrandomness Algorand already creates. A generic version for this implementation may becreated too, for example by incorporating a standalone public randomness mechanism. Assuming that transactions are uniform between all shards’ clients, each local state is roughly /m − /m the size of the global state. .3.1 Partition For ease of exposition we transform
PKS , the public-key space, into the interval (0 , . Algorithm 4:
Partition implementation // ∀ i ∈ [ m ] : define PKS i = ( i − m , im ] Part m ( T ⊆ TX ) = Part ( T , m ) = { T , . . . , T m } s.t. T i = { tx ∈ T | tx . from ∈ PKS i } whichPart m ( tx ) = ⌈ ( tx . from ) · m ⌉ Intuitively,
PKS is lexicographically split into m disjoint intervals of equal sizes, PKS , . . . , PKS m . Transactions sent from users whose public keys reside in PKS i fallwithin T X i and therefore pend P i ’s approval.Going forward, we refer to users whose public keys falls in PKS i as P i ’s clients. Proposition 4.3.1.
Part is a partition functionProof.
According to definition 2.1, any tx ∈ TX must include exactly one PK from ∈ PKS under tx.from . Take any PK u ∈ PKS , it must fall within the bounds of exactly onesubinterval. This gives
T X = S i ∈ [ m ] T X i and ∀ i ̸ = j : T X i ∩ T X j = ∅ . Additionally, ∀ i ∈ [ m ] : ∃ PK u ∈ PKS i ⇒ ∃ tx ∈ T X s.t. tx . from = P K u which means T X i ̸ = ∅ andcompletes the proof. Proposition 4.3.2.
Part is conflict preservingProof.
In both the
UTXO model and the account balance model, an object is used asinput to a transaction. This object has a single owner authorized to sign for it, theowner of that account or of that UTXO set. All transactions sent by u always fallwithin the same T X i partition, where i = ⌈ ( PK u ) · m ⌉ . This aggregates all conflictsthat may arise when several transactions clash over the same asset. Formally:Let X ⊆ T X be some set of transactions such that tx , tx / ∈ X and Verify ( X ∪{ tx } , C ) = True , Verify ( X ∪ { tx } , C ) = True but
Verify ( X ∪ { tx , tx } , C ) = False .Then, by definition, tx , tx are competing transactions. Let us prove that Part assignsboth tx , tx into the same partition, in both schemes we consider possible: UTXO scheme:
Since either one of the pair tx , tx could be admitted, but not both,there must be some utx ∈ UTXO that doubles as an input to both transactions.The owner of utx is the only user authorized to sign and use it, which implies tx . from = tx . from = utx . to = PK u . This immediately gives tx , tx ∈ TX ⌈ PK u · m ⌉ Balance scheme:
Similar analysis leads to concluding that
P K u has sufficient fundsfor either tx or tx but (when combined with X ) it cannot afford both ⇒ tx , tx ∈ TX ⌈ PK u · m ⌉ Recall, we assume that each transaction has a single sender. We implicitly assume that transactionsfrom multiple senders / public keys are aggregated into a multisig address (requiring m-of-n signatures). .3.2 Sync Eager collectSupport reconstructs a global block locally by collecting missing sub-blocks.
Lazy collectSupport collects only transactions sent to P i ’s clients and missing from itslocal block. Both Sync implementations are suitable for
Partition presented in 4.3.1.
Algorithm 5:
Eager collectSupport
Input : B ( r ) ⊆ T X, i ∈ [ m ] Output : { B ( r ) \ B ri } = { tx ∈ B ( r ) | tx . from / ∈ PKS i } RS ri = ∅ ; foreach tx ∈ B ( r ) do if tx . from / ∈ PKS i then RS ri = RS ri ∪ { tx } ; return RS ri ; Algorithm 6:
Lazy collectSupport
Input : B ( r ) ⊆ T X, i ∈ [ m ] Output : { tx ∈ B ( r ) | tx . from / ∈ PKS i ∧ tx . to ∈ PKS i } RS ri = ∅ ; foreach tx ∈ B ( r ) do if tx . from / ∈ PKS i ∧ tx . to ∈ PKS i then RS ri = RS ri ∪ { tx } ; return RS ri ; Algoshard0 uses
Eager collectSupport as it enables participants to maintain the fullglobal state. This makes swift and unpredictable transitions between shards possible.
Proposition 4.3.3.
Eager collectSupport guarantees self containment throughout theentire execution: ∀ i ∈ [ m ] , r ∈ N : Supp ( T X i , C ( r ) ) ⊆ C ri Proof.
By definition,
Eager collectSupport returns RS ri = { B ( r ) \ B ri } , for all i ∈ [ m ] .It follows by induction that C ri = C ( r ) : r = 0 : C i = ⊥ = C (0) r > C ri = C r − i ◦ B r − i ◦ RS r − i = C r − i | {z } C ( r − ) ◦ B r − i ◦ { B ( r − \ B r − i } | {z } B ( r − ) = C ( r ) ⇒ ∀ X ⊆ T X and in particular ∀ X ⊆ T X i : Supp ( X, C ( r ) ) ⊆ C ( r ) = C ri .Algoshard1 uses Lazy collectSupport, a construction that returns a more compact set,resulting in lighter local states. The following proof establishes that when local contextsare kept up-to-date by using
Lazy collectSupport , self containment is preserved.
Proposition 4.3.4.
Lazy collectSupport guarantees that self containment is kept.It holds that:
RemoteSupp i ( B ( r ) ) ⊆ { tx ∈ B ( r ) | tx . from / ∈ PKS i ∧ tx . to ∈ PKS i } = RS ri .Proof. We establish
Supp ( TX i , C r ) ⊆ { tx ∈ TX | tx . from ∈ PKS i ∨ tx . to ∈ PKS i } ∩ C r first. Then, we show RemoteSupp i ( B ( r ) ) ⊆ { tx ∈ B ( r ) | tx . from / ∈ PKS i ∧ tx . to ∈ PKS i } . Claim 4.3.5. for any X ⊆ T X and any legal C r :Supp ( X, C r ) ⊆ { tx ∈ C r | ∃ t ∈ X s.t. tx . from = t . from ∨ tx . to = t . from } roof of claim 4.3.5 : Define
Supp ′ ( X ) = { tx ∈ T X | ∃ t ∈ X s.t. tx . from = t . from ∨ tx . to = t . from } .It suffices to show that Verify ( X, C r ) = Verify ( X, Supp ′ ( X ) ∩ C r ) always holds.Let L u = { tx ∈ TX | tx . from = PK u ∨ tx . to = PK u } . Then, HIST u ( C r ) ≜ L u ∩ C r is the set of all approved transactions sent by or sent to P K u up to time r .Any bookkeeping method used to determine either P K u ’s balance or set of UTXO must return identical results with either C r or HIST u ( C r ) as input. Now, let usexpand L u ’s definition to sets of users by L U ≜ S u ∈ U L u .For any X ⊆ T X let us define PK X = { PK u ∈ PKS | ∃ t ∈ X s.t. t . from = PK u } ,the set of users that send payments in X . Then, HIST
P K X ( C r ) = L P K X ∩ C r isthe complete log of all sent and received transactions by members of P K X sincegenesis. This log suffices to deduce the complete set of their up-to-date resources,as appearing in C r . Therefore X , as well as any other set of transactions sentfrom P K X ’s members, satisfies Verify ( X, C r ) = Verify ( X, HIST
P K X ( C r )) . Notethat HIST PK X ( C r ) = Supp ′ ( X ) ∩ C r .Overall, Verify ( X, C r ) = Verify ( X, Supp ′ ( X ) ∩ C r ) holds. Corollary 4.1.
Supp ( T X i , C r ) ⊆ { tx ∈ T X | tx . from ∈ PKS i ∨ tx . to ∈ PKS i } ∩ C r .It is a direct result of claim 4.3.5, simply by plugging in X = T X i . Let us define:
LSupp ( TX i ) = { tx ∈ TX | tx.from ∈ PKS i } RSupp ( TX i ) = { tx ∈ TX | tx.from / ∈ PKS i ∧ tx.to ∈ PKS i } LSupp ( TX i ) ∩ RSupp ( TX i ) = ∅ and LSupp ( TX i ) ∪ RSupp ( TX i ) = { tx ∈ TX | tx . from ∈ PKS i ∨ tx . to ∈ PKS i } = Supp ′ ( TX i ) both hold. Observe that B ( r ) ∩ LSupp ( TX i ) = B ri , by 4.3.1. Corollary 4.1 gives: Supp ′ ( T X i ) ∩ B ( r ) ⊆ { tx ∈ B ( r ) | tx . from ∈ PKS i ∨ tx . to ∈ PKS i } .These give way to RemoteSupp i ( B ( r ) ) = Supp ( T X i ) ∩{ B ( r ) \ B ri } ⊆ RSupp ( TX i ) ∩ B ( r ) = { tx ∈ B ( r ) | tx . from / ∈ PKS i ∧ tx . to ∈ PKS i } which completes the main proof. Generally speaking, this makes the set of all resources attained and released by
P K u up to time r . .3.3 Membership A key part of Algoshard’s construction, as well as the reason for surveying Algorand’sVRFs in-depth earlier, is implementing
Membership by a VRF construction very similarto Algorand’s sortition procedure.The general idea behind Algoshard’s
Membership is the following:A global random seed, common to all participants and all shards, is maintained through-out the execution. Users locally generate their own shard assignments and membershipcertificates. Users propagate their certificates, for others to verify, whenever necessary.A certificate σ u is created by computing σ u = Sign SK u ( seed ) . Then, a uniform shard isselected by using σ u as an input to . H , thereby binding u with P i where i = ⌈ . H ( σ u ) · m ⌉ .Assuming that seed is indeed random, SIG is a unique signature scheme and . H is acryptographic hash function implies that . H ( σ u ) yields a uniformly drawn (binary)value between 0 and 1. This, in turn, implies that i ∈ [ m ] is indeed randomly drawn.Any user-claimed assignment is easily verifiable, simply by executing Ver PK u ( seed , σ u ) and recalculating ⌈ . H ( σ u ) · m ⌉ .The binding between participants and assigned shards is time-restricted, therebyaverting targeted attacks on specific shard(s). Restriction details are variant dependant:In EagerMembership , used by Algoshard0, certificates are valid for a single round. LazyMembership , used by Algoshard1, begins by bootstrapping a uniform assignmentof all initial participants into all shards. It then proceeds by slowly shuffling currentassignments. A global parameter ≤ t lease ≤ t takeover is set by the deployer. Uponjoining the network, PK u is publicly and randomly assigned a slot t shuffle u ∈ [0 , t lease − .Every t lease rounds, starting from t shuffle u , its previous certificate expires and PK u mustacquire a new membership lease in order to continue participating in the protocol. Newparticipants must wait t lease rounds before their t shuffle u slot is set and their first shardassignment is known and comes into effect. This resolves join-leave attacks.Throughout Algoshard’s execution, participants essentially invoke one VRF in orderto select which second VRF they should call. ⟨ i u , σ u ⟩ ← getMembership (( PK u , SK u ) , r ) followed by Algorand’s ⟨ res , π ru ⟩ ← Sortition (( PK u , SK u ) , seed ri u , role ) are both needed.That is because P i ’s members simply discard u ’s actions, even those sent with ⟨ res , π ru ⟩ that allows u to act as block proposer or verifier, unless σ u that permits u to participatein P i is also provided. EagerMembership is actually a special, simpler, case of
LazyMembership where t lease = 1 . EagerMembership ’s algorithms are explicitly included, as it might be easier to follow this much simplervariant before proceeding to the general version. ootstrapping Membership
Algorithm 7:
EagerMembership init
Input : m ∈ N + , PK ⊆ PKS
Output : participants bookkeeping seed = AlgorandInitRandomness ( PK ) ; foreach i ∈ [ m ] do // init P i 's public randomness seed ← H ( seed + i ) end seed = H ( seed ◦ . . . ◦ seed ) ; foreach P K u ∈ P K do // publicly logged by all nodes t join P K u = 0 ; // privately invoked by eachparticipant ⟨ i u , σ u ⟩ = getMembership (( PK u , SK u ) , ) ; join P i u ; end return { ( PK u , t joinPK u ) | PK u ∈ PK } ; Algorithm 8:
LazyMembership init
Input : m ∈ N + , PK ⊆ PKS , t lease ∈ N + Output : participants bookkeeping seed = AlgorandInitRandomness ( PK ) ; foreach i ∈ [ m ] do // init P i 's public randomness seed ← H ( seed + i ) end seed = H ( seed ◦ . . . ◦ seed ) ; foreach P K u ∈ P K do // publicly logged by all nodes t joinPK u = 0 ; t shufflePK u = H ( PK u ◦ seed ) ( mod t lease ) ; // privately invoked by each participant ⟨ i u , σ u ⟩ = getMembership (( PK u , SK u ) , ) ; join P i u ; end return { ( PK u , t joinPK u , t shufflePK u ) | PK u ∈ PK } ; Once genesis time participants register their public keys, one initial random seed is selected. This seed is generated by genesis time users calling the same distributed random numbers generation algorithmAlgorand employs. seed is then used to kick-start m independent instances of Algorand, rather thanonly one. This is efficiently done by setting P i ’s initial public randomness to be seed = H ( seed + i ) ,which is indeed random since seed is randomly selected and H is a random oracle.Both Membership variants begin with a uniform assignment of all participants into all shards.The initial assignment depends on seed , whose value is unknown at the time participants registertheir public keys. In LazyMembership , each PK u is publicly and randomly assigned a permanent t shuffle u ∈ [0 , t lease − slot. Note that t shuffle u is publicly verifiable and it is always determined by arandom seed that is still unknown when u registers PK u .The following m + 1 public randomness seeds are maintained throughout the execution:• seed r , . . . , seed rm - local public randomness seeds. seed ri is the public randomness used by P i . Its progression follows Algorand’s standard definition: seed r +1 i is either V RF SK u ( seed ri ◦ r + 1) or H ( seed ri ◦ r + 1) , depending on B ri .• seed r - a global public randomness seed, serves as input to Membership . seed r ≜ H ( seed r1 ◦ . . . ◦ seed rm ) is created by bundling all m local public seeds.33 RF construction
Algorithm 9:
Eager getMembership
Input : ( P K u , SK u ) , r Output : ⟨ i ru , σ ru ⟩ σ ru = Sign ( SK u , seed r ) ; h ru = .H ( σ ru ) ; i ru = ⌈ h ru · m ⌉ ; return ⟨ i ru , σ ru ⟩ ; Algorithm 10:
Lazy getMembership
Input : ( P K u , SK u ) , r Output : ⟨ i ru , σ ru ⟩ slot = r % t lease ; diff = slot − t shuffle u ( mod t lease ) ; r ′ = r − diff ; σ ru = Sign ( SK u , seed r ′ ) ; h ru = .H ( σ ru ) ; i ru = ⌈ h ru · m ⌉ ; return ⟨ i ru , σ ru ⟩ ; To participate in Algoshard, u must acquire membership in some P i and a proof thereof. This isdone by executing getMembership (( PK u , SK u ) , r ) → ⟨ i ru , σ ru ⟩ locally and non-interactively. Since seed r is unpredictable and H is a random oracle, h ru = . H ( σ ru ) is uniform between 0 and 1 which makes i ru ∈ [ m ] a uniformly selected shard index. Algorithm 11:
Eager verifyMember
Input :
P K u , σ ru , i, r Output :
True σ ru proves PK u ’s P i membership in round r False otherwise if ⌈ .H ( σ ru ) · m ⌉ ̸ = i then return False ; if P K u / ∈ P K r − then return False ; return Ver ( P K u , seed r , σ ru ) ; Algorithm 12:
Lazy verifyMember
Input :
P K u , σ ru , i, r Output :
True σ ru proves PK u ’s P i membership in round r False otherwise if ⌈ .H ( σ u ) · m ⌉ ̸ = i then return False ; if P K u / ∈ P K max { ,r − t lease } then return False ; attain t shuffle P K u ; slot = r % t lease ; diff = slot − t shuffle P K u ( mod t lease ) ; r ′ = r − diff ; return Ver ( P K u , seed r ′ , σ u ) ; By adopting Algorand’s cryptographic assumptions regarding
SIG and H , it follows that ⟨ i ru , σ ru ⟩ is unforgeable by a poly-time adversary.It remains to prove the resulting assignments are indeed safe, even when facing an adaptive adversary.This is done in proposition 4.3.6. 34 dditional functionalities Algorithm 13:
Eager newNodesArrival
Input : r ∈ N + , P K r new ⊆ PKS
Output : new participants bookkeeping // publicly logged by all nodes foreach PK u ∈ PK rnew do t joinPK u = r ; end return { ( PK u , t joinPK u ) | PK u ∈ PK rnew } ; Algorithm 14:
Lazy newNodesArrival
Input : r ∈ N + , PK rnew ⊆ PKS , t lease ∈ N + Output : new participants bookkeeping // publicly logged by all nodes foreach PK u ∈ PK rnew do // recorded at time r t joinPK u = r ; // set after benching PK u t lease rounds t shufflePK u = H ( PK u ◦ seed t joinPKu + t lease ) ( mod t lease ) ; end return { ( PK u , t joinPK u , t shufflePK u ) | PK u ∈ PK rnew } ; Algorithm 15:
Eager endOfRound
Input : r , seed r + , . . . , seed r + Output : seed r + , updated membership // locally executed by each node seed r + = H ( seed r + ◦ . . . ◦ seed r + ) ; ⟨ i r +1 u , σ r +1 u ⟩ = getMembership (( PK u , SK u ) , r + ) ; leave P i ru and join P i r +1 u ; return seed r + ; Algorithm 16:
Lazy endOfRound
Input : r , seed r + , . . . , seed r + Output : seed r + , updated membership // locally executed by each node seed r + = H ( seed r + ◦ . . . ◦ seed r + ) ; if t shufflePK u ≡ r + ( mod t lease ) then ⟨ i r +1 u , σ r +1 u ⟩ = getMembership (( PK u , SK u ) , r + ) ; leave current P i and join P i r + ; end return seed r + ; Proposition 4.3.6.
Given n participants, of which at least n are honest, an appropriately setparameter m and faced with an adaptive adversary that is able to corrupt participants in t takeover rounds: both EagerMembership and
LazyMembership guarantee an honest majority of h Algorand = 2 / in all shards with extremely high probability. embership CorrectnessTransitioning into balls and bins
Consider representing an honest participant bya blue ball, a Byzantine participant by a red ball and each P i by a bin. Then the anal-ysis of failure could be seen as a model with n blue balls and n red balls placed into m bins, according to Membership ’s logic. If a state in which the number of blue balls insome bin is less than twice the number of red balls in that bin is reached, we lose, hon-est majority breaks. We bound this probability throughout the execution. Let us begin:The initial uniform assignment of both
Membership variants goes as follows:All n balls, regardless of color, are independently and uniformly tossed into all m bins. Lemma 4.3.7.
Let us define failure as any of the bins having a red to overall ballsratio greater than or equal to / . In a setting with n blue balls and n red balls, alluniformly assigned into m bins, it holds that P r ( failure ) < me − n m .Proof. Let us examine the situation in the i th bin (analysis is symmetric for all binsand we union bound it later). From the bin’s perspective, any tossed ball is a Bernoullitrial with success probability p = P r ( X = 1) = m and all tosses are independent. Wedenote Y as the number of blue balls assigned to our bin and Z as the number of redballs assigned to our bin. We get that Y = Y + Y + ... + Y n where Y j representsthe j th blue Bernoulli experiment and Z = Z + Z + ... + Z n where Z j represents the j th red Bernoulli experiment, alternatively Y ∼ Bin ( n , m ) , Z ∼ Bin ( n , m ) . In bothcases we have that E [ Y ] = n m and E [ Z ] = n m . We define X = Y + Z as the totalnumber of balls in the bin. Clearly Y, Z are independent (as all tosses are i.i.d.) andthen X ∼ Bin ( n, m ) with E [ X ] = nm . Complying with the law of large numbers, as n → ∞ and while keeping m constant or sufficiently small, Y → n m and Z → n m . Theexpected ratio of red to overall balls in the bin is E [ ZY + Z ] = E [ Z ] E [ Y ]+ E [ Z ] = . We maydeviate somewhat from the mean and still reach an acceptable (non-failed) state. Inorder to estimate how likely it is to deviate too far, we now bound the tail probability P r ( ZY + Z ≥ ) of states in which the system breaks, as we cannot safely run P i withtoo many Byzantine players.1. P r ( Y ≤ . n m ) ≤ e − n m – The Chernoff bound P r ( X ≤ (1 − δ ) µ ) ≤ e − δ µ applied on Y with δ = and µ = E [ Y ] = n m gives P r ( Y ≤ . n m ) = P r ( Y ≤ (1 − ) n m ) ≤ e − ( 16 )2 · n m = e − n m .2. P r ( Z ≥ . n m ) ≤ e − n m – Similarly, using the bound P r ( X ≥ (1 + δ ) µ ) ≤ e − δ µ δ on Z with δ = and µ = E [ Z ] = n m gives P r ( Z ≥ . m ) ≤ e − ( 14 )2 · n m
2+ 14 = e − n m .36. P r ( ZY + Z ≥
13 ) < e − n m .– Following (1) and (2), the probability of failure from the i th bin perspective satisfies: P r ( ZY + Z ≥ ) ( ∗ ) z}|{ ≤ − (1 − e − n m )(1 − e − n m ) = e − n m + e − n m − e − n m · e − n m ≤ e − n m + e − n m < e − n m .(*) Let us define A = { all assignments where Y ≤ . n m } and B = { all assignmentswhere Z ≥ . n m } . Then, A = { all assignments with Y > . n m } is a set of stateswhere there aren’t too few blue balls and B = { all assignments with Z < . n m } is a set of states where there aren’t too many red balls. It follows that allassignments in C ≜ A ∩ B satisfy ZY < . n m . n m = ⇒ ZY + Z < are non-failedstates where the ratio between red and blue balls is acceptable. Since A, B areindependent and must both occur in order to land C , we encounter C withprobability (1 − P r ( A )) · (1 − P r ( B )) which is at least (1 − e − n m )(1 − e − n m ) according to the bounds in (1) and (2). We’ve shown that all the states in C are acceptable, implying that all unacceptable (failed) states necessarily residewithin its complement, C which is bounded by − C z }| { (1 − e − n m ) | {z } A · (1 − e − n m ) | {z } B | {z } C
4. All bins follow the same distribution, allowing us to union bound (3) over all m bins.We conclude that P r ( failure ) < m · (2 e − n m ) = 2 me − n m . Corollary 4.2.
With m set appropriately, EagerMembership fails only with negligibleprobability, even when continuously running for over a million years ( ∗ ) . ( ∗ ) for any values of m ≤ ∧ n ≥ m we get astronomically small bounds.That is, we can safely support up to 10,000 shards when expecting at least 15,000 par-ticipants per shard on average. Similarly, any m ≤ and n/m ≥ adequatelybounds smaller networks.Proof. Running with
EagerMembership , every new round begins with a new uniformassignment adhering to the analyzed distribution. Plugging in the above numbersgives
P r ( failed round | m ≤ ∧ n ≥ m ) < − . We can also attain P r ( failed round | n ≥ m ) < me < − · m , which bounds the second casewith P r ( failed round | m ≤ ∧ n ≥ m ) < − .Following Algorand’s one round per minute rate gives a total of . · roundswhen running for one million years. By union bounding any of the above bounds overthe course of . · rounds, we are still at most − likely to fail.37 terated balls into bins LazyMembership can be viewed as an iterated balls intobins process. When a ball joins this process, it is imprinted with a uniformly drawnnumber between 0 and t lease − . During the first iteration, where r = 1 , all balls areuniformly thrown into m bins. Then, for any iteration r > , we take all balls whoseimprinted numbers equal r modulo t lease out of their bins and re-throw them at random. Lemma 4.3.8.
When dealing with a static adversary ( t takeover = ∞ , balls cannotchange color), the balls-into-bins distributing for any round r > equals the initialuniform assignment distribution that occurs for r = 1 .Proof. The initial distribution encountered at r = 1 consists of uniformly assigning ballsinto bins, in any order. The actual order in which balls are placed in their assignedbins makes no difference. Then, two equivalent ways to reach the state at r = 2 are:1. Uniformly assign all balls into bins, in any order. Remove all of the balls whoseimprinted numbers equals 2 from their bins. Reassign removed balls uniformly.2. Uniformly assign all balls whose imprinted numbers do not equal 2 into bins.Wait. Uniformly assign all remaining balls.Clearly, option 2 is a uniform assignment of all balls into all bins. Then, the first twoiterations are identically distributed, as succeeding iterations are, by induction. Corollary 4.3.
Faced with a static adversary that can introduce malicious users into thesystem but cannot attack honest users,
LazyMembership is as safe as
EagerMembership . Lemma 4.3.9.
For any selection of t lease where ≤ t lease ≤ t takeover , an adaptiveadversary (with a finite t takeover ) cannot do any better than a static adversary.Proof. By contradiction. Let us assume, for the sake of contradiction, that some adap-tive adversary can plan an attack on a target set B = { b , ..., b k } of k blue balls anddo better than simply sticking with the same set of red balls it already owns.We assume that the adversary is at capacity and owns n/ red balls (if this isn’tthe case, we paint red some randomly selected blue balls and hand those over). Let R = { r , ..., r k } be the subset of red balls that must released in order to attack B , asselected by the adversary’s strategy. When the attack completes, t takeover rounds afterit was set in motion, all balls in B are painted red and all balls in R are painted blue. Inthe meantime, since t lease ≤ t takeover all balls in the system, and in particular those in B and R , are uniformly reassigned. Recall that once an attack has been set in motionit cannot be aborted and must carry on until completion.Take any set B the adversary may possibly choose to target given any (poly-time)algorithm. By the time the attack completes, any b i ∈ B the adversary gained controlover (painted red) is exactly /m likely to be in any specific bin. At the same time,any r i ∈ R the adversary released (painted blue) is also /m likely to be in that samebin, as all assignments are unpredictable. Contradiction to the assumption that anysuch dynamic strategy beats a static strategy.38 orollary 4.4. Faced with either a static or an adaptive adversary,
LazyMembership is as safe as
EagerMembership . Its failure rate adheres the same upper bounds.
Conclusion:
This completes the proof of proposition 4.3.6. Correctness of both
Membership implementations has been asserted, when setting m ≤ n .Take note that the bounds we used in lemma 4.3.7 are rather loose.In practice, simulations with n/m ≈ passed hundreds of millions of rounds withoutfailing once. 390 hapter 5 Additional Related Work
Several recent efforts focus on sharding permissionless blockchain protocols. Common,rather intuitive, guiding lines seem to unify all of these efforts (this thesis included):nodes are scattered between distinct groups, members of each group run some internalprotocol among themselves and different groups can communicate with each other.Nevertheless, it appears that all existing work has taken an ad-hoc approach, byintroducing specific sharded protocols, rather than providing a generalized methodologyfor sharding. When introducing concrete protocols with no previous groundwork, eachprotocol’s correctness must be fully asserted from scratch, a tedious and error-pronetask. A modular framework design, like the one introduced in this thesis, seems like anoverall preferable approach. It provides additional advantages such as applicability toalready-existing protocols, reusable modules that could shard several different protocolswithout work duplication etc.
OmniLedger [10] is a sharded variant of ByzCoin [9] that follows the aforementionedguidelines. OmniLedger is a UTXO based blockchain protocol. It uses an Algorand-likeVRF to elect each epoch’s leader, who in turn leads an execution of RandHound [18].RandHound’s output is used to determine a global nodes-into-shards assignment per-mutation. Transitioning between epochs, swap-out policies are locally set by each shard.OmniLedger introduces a several-rounds-long cross shard transaction approval process.This process is driven by the client. After initializing a cross shard transaction, the userhas to gather proofs-of-acceptance for all UTXO inputs, from their respective shards,in order to admit the cross shard transaction. Aborting a cross shard transaction canbe done by collecting a proof-of-rejection for one of the UTXO inputs. Presenting aproof-of-rejection for any such input enables to reclaim the funds from another alreadyaccepted input. In cases where some input is neither accepted nor rejected, the usermust wait. An already-admitted input is marked as spent, it cannot be reclaimed41r reused until some proof-of-rejection is provided. Such lock-unlock mechanism is,in essence, very similar to using a multisig address as a payer, after gathering desiredfunds into that account. In fact, several constructions presented in Algoshard are rathersimilar to those appearing in OmniLedger. Still, it should be noted that OmniLedgeris far less formal and proofs for many of their claims are lacking or missing.
RapidChain [19] is a new blockchain protocol that comes sharded by design, ratherthan an adaptation of some pre-existing protocol. It executes in epochs, within astrongly synchronous environment and assumes a limited slowly adaptive adversary.RapidChain’s adversary is rather weak, it can commit only a small constant number ofjoin-leave attacks per epoch. Equivalently, the adversary can attack a constant numberof nodes and only during the epoch’s end.RapidChain maintains k + 1 committees. The first committee is called a referencecommittee. The reference committee is in charge of establishing participants identity,generating random strings and setting shard assignments. The other k committeesare simply known as shards. They run synchronous consensus protocols among shardmembers to approve pending transactions. Transaction approval as well the task ofstoring the entire UTXO set are partitioned between the different shards, according totransactions’ ids. RapidChain’s approach to cross shard transactions, in which someinputs reside within different partitions, is the following: Once a cross shard tx issubmitted, the committee in charge of approving tx creates a new transaction for anyUTXO input that resides within another shard. These intermediate transactions aremeant for transferring funds from their current shard into the same shard as tx . Onceall intermediate transactions have been approved, funds are locally available and tx too should be approved. RapidChain’s committee-to-committee routing mechanism ispatterned after distributed hash table (DHT) design, as in Kademlia [13]. In order toparticipate in the protocol, participants need to solve PoW puzzles, to “activate” theirpublic keys, this is done once per epoch. It is assumed that computing power held byall participants in the system is equal. 42 .2 A Few Notes Regarding Cross Shard Transactions Compared with the approach presented in this thesis, it seems as if both OmniLedgerand RapidChain overcomplicate the treatment of cross shard transactions. We manageto avoid the issue altogether, while also allowing all participants to send and receivemoney from each other. This was accomplished by providing general definitions, fine-tuning the partitions and ultimately maintaining a double-entry ledger . Double-entrybookkeeping [3, 4] is a well known bookkeeping system in which every transaction isrecorded (at least) twice, once as credit and once as debit . Then, our two-phasedround structure boils down to updating the debit column followed by updating thecredit column. Since our double-entry ledger is always kept up-to-date and partitionedappropriately, we never partially commit transactions that might need to be aborted, asin OmniLedger . Nor do we need to explicitly initiate complex multi-step transactionswith intermediate transfers that must first be completed to determine whether theenclosing transaction should be approved or denied, as in RapidChain. Note thatthe design presented in RapidChain does not reduce the storage overhead comparedwith our solution. That is because duplicate transfer transactions are created andlocally stored by the receiving shard. Moreover, since the enclosing transaction couldfail after some intermediate transfers were completed, new transactions that use thesealready transferred funds may originate from different shards. In this case, additionalintermediate transactions must be created, which means that the storage overhead isactually worse. Equivalently, once as an incoming transaction and once as an outgoing transaction Transferring money to a multisig address is external to this issue. However, this scenario is fairlyeasily solvable, for example by using a script requiring signature to send funds back if a multisigtransaction was not admitted within a certain timeframe. hapter 6
Conclusion
This thesis introduced a general methodology for sharding blockchain protocols.In chapter 2, a formal model expressing the behavior of blockchain protocols was laidout. In chapter 3, building on top of these definitions, a rigorous sharding schemewas introduced. The scheme is specified while black-boxing some core functionality,making it modular and generic. A proof binding the definitions described in chapter 2and requirements detailed in chapter 3 asserts the framework’s overall correctness.Chapter 4 demonstrates that the requirements details in chapter 3 are indeed feasible,by introducing concrete implementations and creating a sharded variant of Algorand.
In chapters 2 and 3 we made a few assumptions regarding an arbitrary blockchainprotocol, denoted P . These assumptions were made in order to attain stronger resultsand simplify the algorithms. Let us examine the consequences of removing them:• Intra-Block Transactions:
It must be assumed that P does not permit intra-block transactions to obtain P = SP .Removing this assumption, the SP ⊆ P direction of the proof in section 3.4 stillholds, for protocols that do allow intra-block transactions too. This makes suchprotocols’ sharded counterparts safe (albeit possibly slower than the original).• Forks:
We assume that P does not fork in order to simplify the algorithms.If P may fork with non-negligible probability, each sync phase could simply lag k rounds behind the corresponding protocol phase. Adjusting k such that theprobability of overturning blocks after k rounds is negligible.45 .1.2 Applicability to Existing Networks To transform an active, non-sharded network into a sharded one, these steps could befollowed:1. Create a snapshot of the current state of the ledger (preferably as an account-balance list), formatted as a genesis block B snapshot .2. Use B snapshot to bootstrap a sharded variant of the protocol. Our current model assumes that while nodes may join or leave arbitrarily, their orderof magnitude remains constant throughout the entire execution. Supporting furtherdynamicity requires restructuring partitions. This might be required in case of massivedepartures, in order to ensure that honest majority is maintained. Also, in case ofmassive arrivals, when shards overcrowd it might be wise to adjust their number. Thetechnical details of such scheme are left for future work.Another possible extension is forming a “snitching” committee that collects proofs ofByzantine behavior in a specific shard and propagates them to participants outside thatshard. Alternatively, per-shard validity proofs could be integrated into the protocol.These extensions can reduce the gap in tolerance to Byzantine nodes between / thatis required by Algorand and / that we currently require, it may also enable dealingwith a stronger adversary. 46 ibliography [1] Saveen A Abeyratne and Radmehr P Monfared. Blockchain ready manufacturingsupply chain using distributed ledger. 2016.[2] Elli Androulaki, Artem Barger, Vita Bortnikov, Christian Cachin, Konstanti-nos Christidis, Angelo De Caro, David Enyeart, Christopher Ferris, GennadyLaventman, Yacov Manevich, Srinivasan Muralidharan, Chet Murthy, BinhNguyen, Manish Sethi, Gari Singh, Keith Smith, Alessandro Sorniotti, ChrysoulaStathakopoulou, Marko Vukolic, Sharon Weed Cocco, and Jason Yellick. Hy-perledger Fabric: A Distributed Operating System for Permissioned Blockchains. CoRR , abs/1801.10228, 2018.[3] Rob A Bryer. Double-entry bookkeeping and the birth of capitalism: Accountingfor the commercial revolution in medieval northern italy.
Critical perspectives onAccounting , 4(2):113–140, 1993.[4] Bruce G Carruthers and Wendy Nelson Espeland. Accounting for rationality:Double-entry bookkeeping and the rhetoric of economic rationality.
Americanjournal of Sociology , 97(1):31–69, 1991.[5] Jing Chen, Sergey Gorbunov, Silvio Micali, and Georgios Vlachos. Algorand agree-ment: Super fast and partition resilient byzantine agreement. Cryptology ePrintArchive, Report 2018/377, 2018. https://eprint.iacr.org/2018/377 .[6] Ethereum. A Next-Generation Smart Contract and Decentralized ApplicationPlatform. 2018. URL https://github.com/ethereum/wiki/wiki .[7] Yossi Gilad, Rotem Hemo, Silvio Micali, Georgios Vlachos, and Nickolai Zeldovich.Algorand: Scaling byzantine agreements for cryptocurrencies. In
Proceedings of the26th Symposium on Operating Systems Principles , SOSP ’17, pages 51–68, NewYork, NY, USA, 2017. ACM. ISBN 978-1-4503-5085-3. doi: 10.1145/3132747.3132757. URL http://doi.acm.org/10.1145/3132747.3132757 .[8] Shafi Goldwasser and Rafail Ostrovsky. Invariant signatures and non-interactivezero-knowledge proofs are equivalent. In Ernest F. Brickell, editor,
Advances inCryptology — CRYPTO’ 92 , pages 228–245, Berlin, Heidelberg, 1993. SpringerBerlin Heidelberg. ISBN 978-3-540-48071-6.479] Eleftherios Kokoris Kogias, Philipp Jovanovic, Nicolas Gailly, Ismail Khoffi, Li-nus Gasser, and Bryan Ford. Enhancing bitcoin security and performance withstrong consistency via collective signing. In { USENIX } Security Symposium( { USENIX } Security 16) , pages 279–296, 2016.[10] E. Kokoris-Kogias, P. Jovanovic, L. Gasser, N. Gailly, E. Syta, and B. Ford.OmniLedger: A Secure, Scale-Out, Decentralized Ledger via Sharding. In
IEEESymposium on Security and Privacy (SP) , pages 583–598, May 2018.[11] Anna Lysyanskaya. Unique signatures and verifiable random functions from thedh-ddh separation. In Moti Yung, editor,
Advances in Cryptology — CRYPTO2002 , pages 597–612, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg. ISBN978-3-540-45708-4.[12] Bernard Marr. How Blockchain Will Transform The Supply Chain And LogisticsIndustry. mar. 2018.[13] Petar Maymounkov and David Mazieres. Kademlia: A peer-to-peer informationsystem based on the xor metric. In
International Workshop on Peer-to-Peer Sys-tems , pages 53–65. Springer, 2002.[14] Silvio Micali. ALGORAND: the efficient and democratic ledger.
CoRR ,abs/1607.01341, 2016. URL http://arxiv.org/abs/1607.01341 .[15] Silvio Micali, Salil Vadhan, and Michael Rabin. Verifiable random functions. In
Proceedings of the 40th Annual Symposium on Foundations of Computer Sci-ence , FOCS ’99, pages 120–, Washington, DC, USA, 1999. IEEE Computer Soci-ety. ISBN 0-7695-0409-4. URL http://dl.acm.org/citation.cfm?id=795665.796482 .[16] Satoshi Nakamoto. Bitcoin: A peer-to-peer electronic cash system, 2009. URL .[17] David Schwartz, Noah Youngs, and Arthur Britto. The Ripple Protocol ConsensusAlgorithm. Ripple Labs Inc, 2014.[18] Ewa Syta, Philipp Jovanovic, Eleftherios Kokoris Kogias, Nicolas Gailly, LinusGasser, Ismail Khoffi, Michael J Fischer, and Bryan Ford. Scalable bias-resistantdistributed randomness. In ,pages 444–460. Ieee, 2017.[19] Mahdi Zamani, Mahnush Movahedi, and Mariana Raykova. Rapidchain: Scalingblockchain via full sharding. In
Proceedings of the 2018 ACM SIGSAC Conferenceon Computer and Communications Security , CCS ’18, pages 931–948, New York,NY, USA, 2018. ACM. ISBN 978-1-4503-5693-0. doi: 10.1145/3243734.3243853.URL http://doi.acm.org/10.1145/3243734.3243853http://doi.acm.org/10.1145/3243734.3243853