Tame the Wild with Byzantine Linearizability: Reliable Broadcast, Snapshots, and Asset Transfer
TTame the Wild with Byzantine Linearizability: Reliable Broadcast,Snapshots, and Asset Transfer
SHIR COHEN,
Technion, Israel
IDIT KEIDAR,
Technion, Israel
We formalize Byzantine linearizability, a correctness condition that specifies whether a concurrent object with a sequential specificationis resilient against Byzantine failures. Using this definition, we systematically study Byzantine-tolerant emulations of various objectsfrom registers. We focus on three useful objectsβ reliable broadcast, atomic snapshot, and asset transfer. We prove that there is an π -resilient implementation of such objects from registers with π processes π < π . Over the last decade, cryptocurrencies have taken the world by a storm. The idea of a decentralized bank, independent ofpersonal motives has gained momentum, and companies like Bitcoin [16], Ethereum [18] and Diem [6] now play a bigpart in the worldβs economy. At the core of all of those companies lies the asset transfer problem. In this problem, thereare multiple accounts, operated by processes that wish to transfer assets between accounts. This environment raises theneed to tolerate the malicious behavior of processes that wish to sabotage the system.In this work, we consider the shared memory model that was somewhat neglected in the Byzantine discussion. Webelieve that the shared memory model offers an intuitive formulation of the abstract service offered by blockchains. Itis well-known that it is possible to implement reliable read-write shared memory registers via message-passing even ifa fraction of the servers are Byzantine [1, 3, 12, 14, 17]. As a result, as long as the processes using the service are notmalicious, any fault-tolerant object that can be constructed using registers can also be implemented in the presence ofByzantine servers. However, it is not clear what can be done with such objects when they are used by Byzantine processes.In this work, we answer this question. In Section 4 we define
Byzantine linearizability , a correctness condition applicableto any shared memory object with a sequential specification. We then systematically study the feasibility of implementingvarious Byzantine linearizable objects from registers.We observe that existing Byzantine fault-tolerant shared memory constructions [1, 13, 15] in fact implement Byzantinelinearizable registers. Such registers are the starting point of our study. When trying to implement more complex objects(e.g., snapshots and asset transfer) using registers, constructions that work in the crash-failure model no longer work whenByzantine processes are involved, and new algorithms β or impossibility results β are needed.As our first result, we prove in Section 5 that an asset transfer object used by Byzantine processes does not have await-free implementation, even when its specification is reduced to support only transfers operations (without readingprocessesβ balances). Furthermore, it cannot be implemented without a majority of correct processes constantly takingsteps. Since asset transfer has implementations from both reliable broadcast [5] and snapshots [11], these lower boundsalso apply to them. In Section 6, we present a Byzantine linearizable reliable broadcast algorithm with resilience π < π ,proving that the bound on the resilience is tight. We are not familiar with any previous shared memory constructionsof reliable broadcast from registers. Finally, in Section 7 we present a Byzantine linearizable snapshot with the sameresilience. In contrast, previous constructions of Byzantine lattice agreement, which can be directly constructed fromsnapshot [4], required π + processes to tolerate π failures.All in all, we establish a tight bound on the resilience of emulations of three useful shared memory objects fromregisters. On the one hand, we show that it is impossible to obtain wait-free solutions as in the non-Byzantine model, a r X i v : . [ c s . D C ] F e b hir Cohen and Idit Keidar and on the other hand, unlike previous works, our solutions do not require π > π . Taken jointly, our results yield thefollowing theorem:T HEOREM There exist π -resilient Byzantine linearizable implementations of reliable broadcast, snapshot, and assettransfer objects with π processes if and only if π < π . Although the construction of SWMR registers in message passing systems requires π > π servers, our improvedresilience applies to clients, which are normally less reliable than servers, particularly in the so-called permissioned model where servers are trusted and clients are ephemeral.In summary, we make the following contributions: β’ Formalizing Byzantine linearizability for any object with a sequential specification. β’ Proving that some of the most useful building blocks in distributed computing, such as atomic snapshot and reliablebroadcast, do not have π -resilient implementations from SWMR registers when π β₯ π processes are Byzantine. β’ Presenting a Byzantine linearizable implementations of a reliable broadcast object and a snapshot object with theoptimal resilience.
To the best of our knowledge, there is no known Byzantine linearizable implementation of a reliable broadcast in theliterature. Given such an object there are known implementations of lattice agreement [10, 19] which resembles asnapshot object. However, these constructions require π = π + processes. In our work, we present both Byzantinelinearizable reliable broadcast and Byzantine snapshot, (from which Byzantine lattice agreement can be constructed [4]),with resilience π = π + .The asset transfer object we discuss in this paper was introduced by Guerraoui et al. [11]. Their work provides aformalization of the cryptocurrency definition [16]. The highlight of their work is the observation that the asset transferproblem can be solved without consensus. It is enough to maintain a partial order of transactions in the systems, and inparticular, every process can record its own transactions. They present a wait-free linearizable implementation of assettransfer in crash-failure shared memory, taking advantage of an atomic snapshot object. Thus, we can use their solution,together with our Byzantine snapshot, to solve Byzantine linearizable asset transfer with π = π + .In addition, Guerraoui et al. present a Byzantine-tolerant solution in the message passing model. This algorithm utilizesreliable broadcast, where dependencies of transactions are explicitly broadcast along with the transactions. This solutiondoes not translate to a Byzantine linearizable one, but rather to a sequentially consistent asset transfer object. In particular,reads can return old (superseded) values, and transfers may fail due to outdated balance reads.Finally, recent work by Auvolat et al. [5] continues this line of work. They show that a FIFO order property betweeneach pair of processes is sufficient in order to solve the asset transfer problem. This is because transfer operations can beexecuted once a processβs balance becomes sufficient to perform a transaction and there is no need to wait for all causallypreceding transactions. However, as a result, their algorithm is not sequentially consistent, or even causally consistent forthat matter. For example, assume process π maintains an invariant that its balance is always at least 10, and performs atransfer with amount 5 after another process deposits 5 into its account, increasing its balance to 15. Using the protocolin [5], another process might observe π βs balance as 5 if it sees π βs outgoing transfer before the causally preceding deposit.Because our solution is Byzantine linearizable, such anomalies are prevented. ame the Wild with Byzantine Linearizability: Reliable Broadcast, Snapshots, and Asset Transfer We study a distributed system in the shared memory model. Our system consists of a well-known static set Ξ = { , . . . , π } of asynchronous processes. These processes have access to some shared memory objects. In the shared memory model,all communication between processes is done through the API exposed by the objects in the system: processes invokeoperations that in turn, return some response to the process. In this work, we assume a reliable shared memory. (Previousworks have presented constructions of such reliable shared memory in the message passing model [1, 3, 12, 14, 17]). Wefurther assume an adversary that may adaptively corrupt up to π processes in the course of a run. When the adversarycorrupts a process, it is defined as Byzantine and may deviate arbitrarily from the protocol. As long as a process is notcorrupted by the adversary, it is correct , follows the protocol, and takes infinitely many steps. In particular, it continues toinvoke the objectβs API infinitely often.We enrich the model with a public key infrastructure (PKI). That is, every process is equipped with a public-privatekey pair used to sign data and verify signatures of other processes. We denote a value π£ signed by process π as β¨ π£ β© π . Histories.
An execution of a concurrent protocol consists of operations called by processes. Operations start withan invocation event and end with a response event. Operations take time and invocations and responses mark discretepoints in time. These points form the history of an execution. A sub-history of a history π» is a sub-sequence of theevents of π» . A history π» is sequential if it begins with an invocation and each invocation, except possibly the last, isimmediately followed by a matching response. Operation ππ is pending in a history π» if ππ is invoked in π» but does nothave a matching response event.A history defines a partial order on operations: operation ππ precedes ππ in history π» , denoted ππ βΊ π» ππ , if theresponse event of ππ precedes the invocation event of ππ in π» . Two operations are concurrent if neither precedes theother. In addition, a history is well formed if for each process π the sub-sequence of π βs events in π» , denoted π» | π , issequential. Linearizability.
A popular correctness condition for concurrent objects is linearizability, which is defined with respectto an objectβs sequential specification. A linearization of a concurrent history π» of object π is a sequential history π» β² such that (1) after removing some pending operations from π» and completing others by adding matching responses,it contains the same invocations and responses as π» β² , (2) π» β² preserves the partial order βΊ π» , and (3) π» β² satisfies π βssequential specification. f-resilient. An algorithm is f-resilient if as long as at most π processes fail, every correct process eventually returnsfrom each operation it invokes. A wait-free algorithm is a special case where π = π β . Single Writer Multiple Readers Register.
The basic building block in shared memory is a single writer multiple readers(SWMR) register that exposes read and write operations. Such registers are used to construct more complicated objects.The sequential specification of a SWMR register states that every read operation from register π returns the value lastwritten to π . Note that this specification is meaningful only if the registerβs writer is not Byzantine. A Byzantine processcan write arbitrary values to its registers. Asset Transfer Object.
In [11], the asset transfer problem is formulated as a sequential object type, called
Asset TransferObject . The asset transfer object maintains a mapping from processes in the system to their balances . Initially, themapping contains the initial balances of all processes. The object exposes a transfer operation, transfer(src,dst,amount) , The definition in [11] allows processes to own multiple accounts. For simplicity, we assume a single account per-process in this work.3 hir Cohen and Idit Keidar which can be invoked by process src (only). It withdraws amount from process src βs account and deposits it at process dst βs account provided that src βs balance was at least amount . It returns a boolean that states whether the transfer wassuccessful (i.e., src had amount to spend). In addition, the object exposes a read(i) operation that returns the currentbalance of π . In this section we define Byzantine linearizability. Intuitively, we would like to tame the Byzantine behavior in a way thatprovides consistency to correct processes. We linearize the correct processesβ operations and offer a degree of freedom toembed additional operations by Byzantine processes.We denote by π» | πππππππ‘ the projection of a history π» to all correct processes. We say that a history π» is Byzantinelinearizable if π» | πππππππ‘ can be augmented with operations of Byzantine processes such that the completed historyis linearizable. In particular, if there are no Byzantine failures then Byzantine linearizability is simply linearizability.Formally:D EFINITION (Byzantine Linearizability) A history π» is Byzantine linearizable if there exists a history π» β² so that π» β² | πππππππ‘ = π» | πππππππ‘ and π» β² is linearizable. Similarly to linearizability, we say that an object is Byzantine linearizable if all of its executions are ByzantineLinearizable.Next, we characterize objects for which Byzantine linearizability is meaningful. The most fundamental component inshared memory is read-write registers. Not surprisingly, such registers, whether they are single-writer or multi-writersones are de facto Byzantine linearizable without any changes. This is because before every read from a Byzantine register,invoked by a correct process, one can add a corresponding Byzantine write.In practice, multiple writers multiple readers (MWMR) registers are useless in a Byzantine environment as an adversarythat controls the scheduler can prevent any communication between correct processes. SWMR registers, however, arestill useful for constructing more meaningful objects. Nevertheless, the constructions used in the crash-failure modelfor linearizable objects do not preserve this property. For instance, if we allow Byzantine processes to run a classicatomic snapshot algorithm [2] using Byzantine linearizable SWMR registers, it will not result in a Byzantine linearizablesnapshot object.
Relationship to Other Correctness Conditions.
Byzantine linearizability provides a simple and intuitive way to captureByzantine behavior in the shared memory model. We now examine the relationship of Byzantine linearizability withpreviously suggested correctness conditions involving Byzantine processes. Some works have defined linearizationconditions for specific objects. This includes conditions for SWMR registers [15], a distributed ledger [9], and assettransfer [5]. Our condition coincides with these definitions for the specific objects and thus generalizes all of them. Liskovand Rodrigues [13] presented a correctness condition that has additional restrictions. Their correctness notion relies onthe idea that Byzantine processes are eventually detected and removed from the system, and focuses on converging tocorrect system behavior after their departure. While this model is a good fit when the threat model is software bugs ormalicious intrusions, it is less appropriate for settings like cryptocurrencies, where Byzantine behavior cannot be expectedto eventually stop. ame the Wild with Byzantine Linearizability: Reliable Broadcast, Snapshots, and Asset Transfer In shared memory, one typically aims for wait-free objects, which tolerate any number of process failures. Indeed, manyuseful objects have wait-free implementations from SWMR registers in the non-Byzantine case. This includes reliablebroadcast, snapshots, and as recently shown, also asset transfer. We now show that in the Byzantine case, wait-freeimplementations of these objects are impossible. Moreover, a majority of correct processes is required.T
HEOREM For any π > , there does not exist a Byzantine linearizable implementation of asset transfer thatsupports only transfer operations in a system with π β€ π processes, π of which can be Byzantine, using only SWMRregisters. P ROOF . Assume by contradiction that there is such an algorithm. Let us look at a system with π = π correct processes.Partition Ξ as follows: Ξ = π΄ βͺ π΅ βͺ { π , π } , where | π΄ | = π β , | π΅ | = π β , π΄ β© π΅ = β , and π , π β π΄ βͺ π΅ . By assumption, | π΄ | > . Let π§ be a process in π΄ . Also, by assumption | π΅ | β₯ . Let π , π be processes in π΅ . The initial balance of allprocesses but π§ is , and the initial balance of π§ is . We construct four executions as shown in Figure 1.Let π be an execution where, only processes in π΄ βͺ { π } take steps. First, π§ performs transfer( π§, π ,1) . Since up to π processes may be faulty, the operation completes, and by the objectβs sequential specification, it is successful (returnstrue). Then, π performs transfer( π , π ,1) . By π -resilience and linearizability, this operation also completes successfully.Note that in π no process is actually faulty, but because of π -resilience, progress is achieved when π processes are silent.Similarly, let π be an execution where the processes in π΄ are correct, and π§ performs transfer( π§, π ,1) , followed by π performing transfer( π , π ,1) .We now construct π , where all processes in π΄ βͺ { π } are Byzantine. We first run π . Call the time when it ends π‘ . Atthis point, all processes in π΄ βͺ { π } restore their registers to their initial states. Note that no other processes took stepsduring π , hence the entire shared memory is now in its initial state. Then, we execute π . Because we have reset thememory to its initial state, the operations execute the same way. When π completes, processes in A \{ π§ } restore theirregisters to their state at time π‘ . At this point, the state of π§ and π is the same as it was at the end of π , the state ofprocesses in π΄ \ { π§ } is the same as it was at the end of π , and processes in π΅ are all in their initial states.We construct π where all processes in π΄ βͺ { π } are Byzantine by executing π , having all processes in π΄ βͺ { π } resettheir memory, executing π , and then having π§ and π restore their registers to their state at the end of π . At this point,the state of π§ and π is the same as it was at the end of π , the state of processes in π΄ \ { π§ } is the same as it was at the endof π , and processes in π΅ are all in their initial states.We observe that for processes in B , the configurations at the end of π and π are indistinguishable as they did not takeany steps and the global memory is the same. By π -resilience, in both cases π and π , together with processes in B andone of { π , π } should be able to make progress at the end of each of these runs. We extend the runs by having π and π invoke transfers of amount 1 to each other. In both runs processes in π΅ βͺ { π , π } help them make progress. In π π behaves as if it is a correct process and its local state is the same as it is at the end of π and in π π behaves as if it is acorrect process and its local state is the same as it is at the end of π . Thus, π and π are indistinguishable to all correctprocesses, and as a result π and π act the same in both runs. However, from safety exactly one of their transfers shouldsucceed. In π , π is correct and transfer( π , π ,1) succeeds, allowing π to transfer 1 and disallowing the transfer from π ,whereas π the opposite is true. This is a contradiction. β‘ Guerraoui et al. [11] use an atomic snapshot to implement an asset transfer object in the crash-fault shared memorymodel. In addition, they handle Byzantine processes in the message-passing model by taking advantage of reliable hir Cohen and Idit Keidar Fig. 1. An asset transfer object does not have an π -resilient implementation for π β€ π . broadcast. Their atomic snapshot-based asset transfer is wait-free and linearizable, and thus it is Byzantine linearizableassuming a Byzantine linearizable snapshot. Their reliable broadcast-based algorithm, on the other hand, is not linearizableand therefore not Byzantine linearizable even when using Byzantine linearizable reliable broadcast. Nonetheless, Auvolatet al. [5] have used reliable broadcast to construct an asset transfer object where transfer operations are linearizable(although reads are not).We note that our lower bound holds for an asset transfer object without read operations. This discussion leads us to thefollowing corollary:C OROLLARY For any π > , there does not exist an π -resilient Byzantine linearizable implementation of an atomicsnapshot or reliable broadcast in a system with π β₯ π Byzantine processes using only SWMR registers.
Furthermore, we prove in the following lemma that in order to provide π -resilience it is required that at least a majorityof correct processes take steps infinitely often.C OROLLARY For any π > , there does not exist an π -resilient Byzantine linearizable implementation of assettransfer in a system with π = π + processes, π of which can be Byzantine, using only SWMR registers if correctprocesses do not take steps infinitely often. P ROOF . Assume by way of contradiction that there exists an π -resilient Byzantine linearizable implementation of assettransfer in a system with π = π + processes where there is a correct process π that does not take steps infinitely often.Thus, there is a point π‘ in any execution such that from time π‘ π does not take any steps. Starting π‘ , the implementation isequivalent to one in a system with π = π , π of them may be Byzantine. This is a contradiction to Theorem 2. β‘ With the acknowledgment that not all is possible, we seek to find Byzantine linearizable objects that are useful evenwithout a wait-free implementation. One of the practical objects is a reliable broadcast object. We already proved in theprevious section that it does not have an π -resilient Byzantine linearizable implementation. In this section we provide animplementation if π > and π β€ π that tolerates π β€ π faults. ame the Wild with Byzantine Linearizability: Reliable Broadcast, Snapshots, and Asset Transfer The reliable broadcast object exposes two operations broadcast(ts,m) and deliver(j,ts) . Its classic definition, given formessage passing systems [8], requires the following properties: β’ Validity: if a correct process broadcasts π then all correct processes eventually deliver π . β’ Agreement: if a correct process delivers π then all correct processes eventually deliver π β’ Integrity: No process delivers two different messages for the same ( π‘π , π ) and if π is correct delivers only messages π previously broadcastIn the shared memory model, we define the sequential specification of reliable broadcast as follows:D EFINITION Reliable broadcast object exposes two operations broadcast(ts,v) and deliver(j,ts) . deliver(j,ts) returnsthe value v of the first broadcast(ts,v) invoked by process π before it is called. If π did not invoke broadcast before thedeliver, then it returns β₯ . Whereas in message passing systems reliable broadcast works in a push fashion, where the receipt of a message triggersaction at its destination, in the shared memory model processes need to actively pull information from the registers. But ifall messages are eventually pulled, the reliable broadcast properties are achieved, as proven in the following lemma.L
EMMA A Byzantine linearization of a reliable broadcast object satisfies the three properties of reliable broadcast. P ROOF . If a correct process broadcasts π , and all messages are subsequently pulled then according to Definition 2 allcorrect processes deliver π , providing validity. For agreement, if a correct process invokes deliver(j,ts) that returns π and all messages are later pulled by all correct processes, it follows that all correct process also invoke deliver(j,ts) andeventually return π β² β β₯ . Since deliver(j,ts) returns the value π£ of the first broadcast(ts,v) invoked by process π before it iscalled, and there is only one first broadcast, and we get that π = π β² . Lastly, if deliver(j,ts) returns π , by the specification, π previously invoked broadcast(ts,m) . β‘ In our implementation (given in Algorithm 1), each process has 4 SWMR registers: send, echo, ready, and deliver, towhich we refer as stages of the broadcast. We follow concepts from Brachaβs implementation in the message passingmodel [7] but leverage the shared memory to improve its resilience from π + to π + . The basic idea is that a processthat wishes to broadcast value π£ writes it in its send register and returns only when it reaches the deliver stage. Throughoutthe run, processes infinitely often call a refresh function whose role is to help the progress of the system. When refreshing,processes read all registers and help promote broadcast values through the 4 stages. For a value to be delivered, it has tohave been seen and signed by π + processes at the ready stage. Before promoting a value to the ready or deliver stage, acorrect process π performs a βdouble-collectβ of the echo registers. Namely, after collecting π + signatures on a valuein ready registers, meaning that it was previously written in the echo of at least one correct process, π re-reads all echoregisters to verify that there does not exist a conflicting value (with the same timestamp and sender). Using this method,concurrent deliver operations βseeβ each other, and delivery of conflicting values broadcast by a Byzantine process isprevented. Before delivering a value, a process writes it to its deliver register with π + signatures. Once one correctprocess delivers a value, the following deliver calls can witness the π + signatures and copy this value directly from itsdeliver register.We make two assumptions on the correct usage of our algorithm. The first is inherently required as shown in Corollary 2: hir Cohen and Idit Keidar Algorithm 1
Shared Memory Bracha: code for process π shared SWMR registers: π πππ π , ππβπ π , πππππ¦ π , πππππ£ππ π procedure CONFLICTING - ECHO ( β¨ π‘π , π£ β© π ) return β π€ β π£, π β Ξ such that β¨ π‘π , π€ β© π β ππβπ π procedure BROADCAST (ts,val) π πππ π β β¨ π‘π , π£ππ β© π repeat π β deliver(i,ts) until π β β₯ β² message is deliverable procedure DELIVER (j,ts) refresh() if β π β Ξ and π£ s.t. β¨β¨ π‘π , π£ β© π , π β© β πππππ£ππ π where π is a set of π + signatures on β¨ πππππ¦, β¨ π‘π , π£ β© π β© then πππππ£ππ π β πππππ£ππ π βͺ {β¨β¨ π‘π , π£ β© π , π β©} return π£ return β₯ procedure REFRESH for π β [ π ] do π β π πππ π if (cid:154) π‘π , π£ππ s.t. π = β¨ π‘π , π£ππ β© π then continue β² π is not a signed pair ππβπ π β ππβπ π βͺ {β¨ π β© π } if Β¬ conflicting-echo ( π ) then πππππ¦ π β πππππ¦ π βͺ {β¨ πππππ¦, π β© π } if β π β Ξ s.t. | π | β₯ π + , β π β π, β¨ πππππ¦, π β© π β πππππ¦ π and Β¬ conflicting-echo ( π ) then πππππ£ππ π β πππππ£ππ π βͺ {β¨ π, π = {β¨ πππππ¦, π β© π | π β π }β©} β² π is the set of π + signaturesA SSUMPTION All correct processes infinitely often invoke methods of the reliable broadcast API.
The second is a straight forward validity assumption:A
SSUMPTION Correct processes do not invoke broadcast(ts,val) twice with the same π‘π . We now prove our algorithmβs correctness. We first make two simple observations:O
BSERVATION If process π is correct and π£ appears in ππβπ π or πππππ¦ π it is never deleted. O BSERVATION If process π is correct and β¨β¨ π‘π , π£ β© π , π β© appears in πππππ£ππ π for any process π then π previously invoked ππππππππ π‘ ( π‘π , π£ ) . P ROOF . Since we assume unforgeable signatures, π has previously signed β¨ π‘π , π£ β© . By the code, this is only possible if π invoked ππππππππ π‘ ( π‘π , π£ ) . β‘ We next prove the following lemma, identifying invariants of Algorithm 1.L
EMMA Algorithm 1 satisfies the following invariants: ame the Wild with Byzantine Linearizability: Reliable Broadcast, Snapshots, and Asset Transfer Fig. 2. Concurrent deliver operations.
I1: If β¨β¨ π‘π , π£ β© π , π β© (where π is a set of π + ready signatures) appears in πππππ£ππ π for any processes π, π , then β¨ πππππ¦, β¨ π‘π , π£ β© π β© π β πππππ¦ π for a correct process π .I2: If β¨ πππππ¦, β¨ π‘π , π£ β© π β© π β πππππ¦ π for a correct process π , then β¨ π‘π , π£ β© π β ππβπ π .I3: If β¨ πππππ¦, β¨ π‘π , π£ β© π β© π appears in πππππ¦ π and β¨ πππππ¦, β¨ π‘π , π€ β© π β© π β² appears in πππππ¦ π β² for any two correct pro-cesses π, π β² then π£ = π€ .I4: If β¨β¨ π‘π , π£ β© π , π β© appears in πππππ£ππ π and β¨β¨ π‘π , π€ β© π , π β© appears in πππππ£ππ π β² for any two correct processes π, π β² then π£ = π€ . P ROOF . I1: Since β¨β¨ π‘π , π£ β© π , π β© appears in πππππ£ππ π and it contains a set of π + signatures on β¨ πππππ¦, β¨ π‘π , π£ β© π β© ,there is at least one correct process π that signed β¨ πππππ¦, β¨ π‘π , π£ β© π β© and added it to its ready register. By Ob-servation 1, it is not deleted from the register.I2: Immediate from the code and Observation 1.I3: Since β¨ πππππ¦, β¨ π‘π , π£ β© π π β© appears in πππππ¦ π and π is correct, by I2 at least one correct process signed β¨ π‘π , π£ β© π and added it to its echo register. Let π be the first correct process to do so, and let π‘ be the moment ofadding β¨ π‘π , π£ β© π to ππβπ π (see Figure 2 for illustration). By Observation 1, it is not deleted from the register.Similarly, let π be the first correct process to add β¨ π‘π , π€ β© π to ππβπ π at time π‘ . WLOG, π‘ β₯ π‘ . In addition,let π be the first correct process to add β¨ πππππ¦, β¨ π‘π , π£ β© π β© to πππππ¦ π , and let π‘ be the moment of the addition.By I2 it follows that π‘ > π‘ . By Observation 1, the content of ππβπ π and πππππ¦ π is not deleted duringthe run. By the protocol, at some point in time between π‘ and π‘ , π executes line 19 and reads all echoregisters. Let π‘ < π‘ β < π‘ be the time when π reads ππβπ π . Since π‘ β₯ π‘ we conclude that π‘ β > π‘ . Since, π does not see a conflicting value in ππβπ π , we get that π£ = π€ .I4: By I1 at least one correct process π signed β¨ πππππ¦, β¨ π‘π , π£ β© π β© and added it to πππππ¦ π and at least one correctprocess π β² signed β¨ πππππ¦, β¨ π‘π , π€ β© π β© and added it to πππππ¦ π β² . Thus, by I3 π£ = π€ . β‘ Let us examine an execution πΈ of the algorithm. Let π» be the history of πΈ . First, we define π» π to be the history π» afterremoving any pending deliver operations and any pending broadcast operations that did not complete line 11 (which iscalled from line 6). We define π» β² to be an augmentation of π» π | πππππππ‘ as follows. For every Byzantine process π and avalue π£ such that π£ is returned by deliver π (j,ts) for at least one correct process π , we add to π» β² a broadcast π (ts,v) operationthat begins and ends immediately before the first correct process adds β¨β¨ π‘π , π£ β© π , π β© to its delivery register. Since at least onecorrect process adds this value at line 11, this moment is well-defined. We construct a linearization πΈ β² of π» β² by definingthe following linearization points: hir Cohen and Idit Keidar β’ Let π be a broadcast π (ts,v) operation by a correct process π that completed line 11. Note that by the code everycompleted broadcast operation completes line 11 exactly once, and operations that do not complete this line areremoved from π» β² . The operation linearizes when β¨β¨ π‘π , π£ β© π , π β© is added for the first time to delivery register of acorrect process, which occurs either when π executes line 11 or when another correct process executes line 22beforehand. By the code, these lines are between the invocation and the return of the broadcast procedure. β’ Let π be a deliver π (j,ts) operation by a correct process π that completes line 11 and returns π£ β β₯ (note that by thecode every completed deliver operation that returns π£ β β₯ completes line 11 exactly once). If π finds β¨β¨ π‘π , π£ β© π , π β© for some value π£ in some correct processβ deliver register at line 10, then the operation linearizes when π first reads β¨β¨ π‘π , π£ β© π , π β© from a correct process. Otherwise, it linearizes at line 11 when π copies the data to πππππ£ππ π . β’ If π is a completed deliver π (j,ts) operation by a correct process π that returns β₯ it linearizes at the moment of itsinvocation. β’ Every Byzantine broadcast π (ts,v) operation by process π linearizes at the moment we added it.In π» β² there are no deliver operations by Byzantine processes. The following lemmas prove that πΈ β² , the linearization of π» β² , satisfies the sequential specification:L EMMA For a given deliver(j,ts) operation that returns π£ β β₯ , there is at least one preceding broadcast operationin πΈ β² of the form broadcast(ts,v) invoked by process π . P ROOF . Let π be a deliver π (j,ts) operation invoked by a correct process π that returns π£ β β₯ . Let π‘ be the time when β¨β¨ π‘π , π£ β© π , π β© is added for the first time to a delivery register of a correct process (where π contains π + ready signatures).If π is correct then by Observation 2 π previously invoked broadcast(ts,v) and that broadcast linearizes at time π‘ . If π isByzantine then broadcast(ts,v) by process π is added to π» β² immediately before π‘ . There are two options to the linearizationpoint of π . If π finds β¨β¨ π‘π , π£ β© π , π β© in some correct processβ deliver register at line 10, then π linearizes when π first reads β¨β¨ π‘π , π£ β© π , π β© from a correct process and thus it is after time π‘ . Otherwise, it linearizes at line 11 when π copies the data to πππππ£ππ π , which is also no earlier than time π‘ . β‘ L EMMA For a broadcast π (ts,v) in πΈ β² , there does not exist any broadcast π (ts,w) in πΈ β² for π£ β π€ . P ROOF . If π is a correct process, the proof follows from Assumption 2. If π is Byzantine, broadcast π (ts,v) is addedimmediately before the first correct process adds β¨β¨ π‘π , π£ β© π , π β© to its delivery register. By I4, no correct processes add β¨β¨ π‘π , π€ β© π , π β© to their delivery register for π£ β π€ and broadcast π (ts,w) does not appear in πΈ β² . β‘ L EMMA For a given deliver(j,ts) operation that returns β₯ , there is no preceding broadcast operation in π» β² of theform broadcast(ts,v) invoked by process π , for π£ β β₯ . P ROOF . Let π be a deliver(j,ts) operation invoked by a correct process π that returns β₯ . Assume by way of contradictionthat there is a preceding broadcast(ts,v) operation in π» β² invoked by process π , for π£ β β₯ . By definition, the broadcastlinearizes no later than the first adding of β¨β¨ π‘π , π£ β© π , π β© to a delivery register of a correct process. Thus, since π linearizes atthe moment of its invocation, it sees β¨β¨ π‘π , π£ β© π , π β© at some processβ delivery register and returns π£ β β₯ , in contradiction. β‘ Next, we prove π -resilience.L EMMA (Liveness) Every correct process that invokes some operation eventually returns. P ROOF . If a correct process π invokes a deliver operation then by the code it returns in a constant time. If it invokes broadcast(ts,v) , it copies β¨ π‘π , π£ β© π to π πππ π . By Assumption 1, all correct processes infinitely often call the reliable broadcast ame the Wild with Byzantine Linearizability: Reliable Broadcast, Snapshots, and Asset Transfer API and specifically the refresh procedure, see β¨ π‘π , π£ β© π and copy it to their echo registers. As signatures are unforgable and π is correct they do not find β¨ π‘π , π€ β© π for any other π€ β π£ in any other echo registers and copy a signed β¨ πππππ¦, β¨ π‘π , π€ β© π β© to their ready registers. By I1, eventually they all see β¨ πππππ¦, β¨ π‘π , π€ β© π β© in π + ready registers and copy β¨ π‘π , π€ β© π to theirdeliver registers. Eventually π + correct processes have β¨ π‘π , π€ β© π in their deliver registers, and since the signatures arevalid, the check at line 10 evaluates to true, and π returns π£ and finish the repeat loop. β‘ We conclude the following theorem:T
HEOREM Algorithm 1 implements an π -resilient Byzantine linearizable reliable broadcast object for any π < π . In this section, we utilize a reliable broadcast primitive to construct a Byzantine snapshot object with resilience π > π . A snapshot [2] is represented as an array of π shared single-writer variables that can be accessed with two operations: update(v) called by process π updates the π π‘β entry in the array and snapshot returns an array. The sequential specificationof an atomic snapshot is as follows: the π π‘β entry of the array returned by a snapshot invocation contains the value v lastupdated by an update(v) invoked by process π , or its variableβs initial value if no update was invoked.Following Corollary 2, we again must require that correct processes perform operations infinitely often. For simplicity,we require that they invoke infinitely many snapshot operations; if processes invoke either snapshots or updates, we canhave each update perform a snapshot and ignore its result.A SSUMPTION All correct processes invoke snapshot operations infinitely often.
Our pseudo-code is presented in Algorithms 2 and 3. During the algorithm, we compare snapshots using the (partial)coordinate-wise order. That is, let π and π be two π -arrays. We say that π > π if β π β [ π ] , π [ π ] .π‘π > π [ π ] .π‘π .Recall that all processes invoke snapshot operations infinitely often. In each snapshot instance, correct processes startby collecting values from all registers and broadcasting their collected arrays in βstartβ messages (message with timestamp0). Then, they repeatedly send the identities of processes from which they delivered start messages until there exists around such that the same set of senders is received from π + processes in that round. Once the set of senders stabilizes,the snapshot is formed as the supremum of the collects in their start messages.We achieve optimal resilience by waiting for only π + processes to send the same set. Although there is not necessarilya correct process in the intersection of two sets of size π + , we leverage the fact that reliable broadcast preventsequivocation to ensure that nevertheless, there is a common message in the intersection, so two snapshots obtained in thesame round are necessarily identical. Moreover, once one process obtains a snapshot π , any snapshot seen in a later roundexceeds π .Each process π collects values from all processesβ registers in a shared variable πππππππ‘ π . When starting a snapshotoperation, each process runs update-collect, where it updates its collect array (line 8) and saves it in a local variable π (line 9). When it does so, it updates the π π‘β entry to be the highest-timestamped value it observes in the π π‘β entriesof all processesβ collect arrays (lines 16 β 18). Then, it initiates the snapshot-aux procedure with a new auxnum tag.Snapshot-aux returns a snapshot, but not necessarily a βfreshβ one that reflects all updates that occurred before snapshot hir Cohen and Idit Keidar Algorithm 2
Byzantine Snapshot: code for process π shared SWMR registers: β π β [ π ] πππππππ‘ππ π [ π ] β {β₯} βͺ { N Γ π πππ } with selectors π‘π and val, initially β₯β π β N , π ππ£ππ πππ π [ π ] β {β₯} βͺ { array of π π πππ Γ set of messages } with selectors π πππ and proof, initially β₯ local variables: π‘π π β N , initially 0 β π β [ π ] , ππ‘π π [ π ] β N , initially 0 π, ππ’π₯ππ’π β N , initially 0 π β [ π ] , initially 1 β π β [ π ] , π β N , seen π [j][k] , π ππππππ π β P ( Ξ ) , initially β π β β set of messages procedure UPDATE ( π£ ) for π β [ π ] do β² collect current memory state update-collect( πππππππ‘ππ π ) π‘π π β π‘π π + πππππππ‘ππ π [ π ] β β¨ π‘π π , π£ β© π β² update local component of collected procedure SNAPSHOT for π β [ π ] do β² collect current memory state update-collect( πππππππ‘ππ π ) π β πππππππ‘ππ π repeat ππ’π₯ππ’π β ππ’π₯ππ’π + π πππ β snapshot-aux ( ππ’π₯ππ’π ) until π πππ β₯ π β² snapshot is newer than the collected state return π πππ procedure UPDATE - COLLECT (c) for π β [ π ] do if π [ π ] .π‘π > πππππππ‘ππ π [ π ] .π‘π and π [ π ] is signed by π then πππππππ‘ππ π [ π ] β π [ π ] was invoked. Therefore, snapshot-aux is repeatedly called until it collects a snapshot π such that π β₯ π , according to thesnapshots partial order (lines 10 β 13).By Assumption 3 and since the ππ’π₯ππ’π variable at each correct process is increased by 1 every time snapshot-auxis called, all correct processes participate in all instances of snapshot-aux. When a correct process invokes a snapshot-aux procedure with auxnum, it first initiates a new reliable broadcast instance at line 28, dedicated to this instance ofsnapshot-aux. Then, as another preliminary step it once again updates its collect array using the update-collect procedure(lines 30β 31) and broadcasts it to all processes at line 33. During the execution, a correct processes delivers messagesfrom all other processes in a round robin fashion. The local variable π represents the process from which it currentlydelivers. In addition, ππ‘π [ π ] maintains the next timestamp to be delivered from π (lines 37, 38 and 49). Note that if thedelivered message at some point is β₯ , ππ‘π [ π ] is not increased, so all of π βs messages are delivered in order (line 39).Snapshot-aux proceeds in rounds, which are reflected in the timestamps of the messages broadcast during its execution.Each correct process starts snapshot-aux at round , where it broadcasts its collected array; we refer to this as its startmessage. It then continues to round π + once it has delivered π + round π messages (line 51). Each process maintains ame the Wild with Byzantine Linearizability: Reliable Broadcast, Snapshots, and Asset Transfer Algorithm 3
Byzantine Snapshot auxiliary procedures: code for process π procedure MINIMUM - SAVED (auxnum) π β { π |β π β [ π ] , π = π ππ£ππ πππ π [ ππ’π₯ππ’π ] .π πππ and π ππ£ππ πππ π [ ππ’π₯ππ’π ] .ππππ π is a valid proof of π } if π = β then return β₯ πππ β infimum ( π ) β² returns the minimum value in each index π ππ£ππ πππ π [ ππ’π₯ππ’π ] β β¨ πππ , (cid:208) π β[ π ] π ππ£ππ πππ π [ ππ’π₯ππ’π ] .ππππ π β© update-collect( πππ ) return πππ procedure SNAPSHOT - AUX (auxnum) initiate new reliable broadcast instance π β β for π β [ π ] do β² collect current memory state update-collect( πππππππ‘ππ π ) π ππππππ π β { π } β² start message contains collect broadcast(0, β¨ πππππππ‘ π β© π ) while true do πππβππ β minimum-saved( ππ’π₯ππ’π ) β² check if there is a saved snapshot if πππβππ β β₯ then return πππβππ π β ( π + ) mod π + β² deliver messages in round robin π β deliver( π, ππ‘π π [ π ] ) β² deliver next message from π if π = β₯ then continue if ππ‘π π [ π ] = 0 and π contains a signed collect array π then β² start message (round 0) π β π βͺ { π } update-collect( π ) π ππππππ π β π ππππππ π βͺ { π } else if π contains a signed set of processes, ππ ππππππ then β² round π message for π > if ππ ππππππ β π ππππππ π then continue β² cannot process message, its dependencies are missing π β π βͺ { π } π πππ π [ π ] [ ππ‘π π [ π ]] β ππ ππππππ βͺ π πππ π [ π ] [ ππ‘π π [ π ] β ] ππ‘π π [ π ] β ππ‘π π [ π ] + if received π + round- π messages for the first time then π β π + broadcast( π, β¨ π ππππππ π β© π ) if β π s.t. |{ π | π πππ π [ π ] [ π ] = π ππππππ π }| = π + then β² stability condition π β senders π β β β π β [ π ] , π β N , seen π [j][k] β β πππβππ β minimum-saved( ππ’π₯ππ’π ) β² re-check for saved snapshot if πππβππ β β₯ then return πππβππ π ππ£ππ πππ π [ ππ’π₯ππ’π ] β β¨ πππππππ‘ π , π β© β² π contains all received messages in this snapshot-aux instance return πππππππ‘ π hir Cohen and Idit Keidar a local set π ππππππ that contains the processes from which it received start messages (line 43). In every round (from 1onward) processes send the set of processes from which they received start messages (line 52).Process π maintains a local map π πππ [ π ] [ π ] that maps a process π and a round π to the set of processes that π reported tohave received start messages from in rounds 1βr (line 48), but only if π has received start messages from all the reportedprocesses (line 45). By doing so, we ensure that if for some correct process π and a round r π πππ π [ π ] [ π ] contains a process π , π is also in π ππππππ π . If this condition is not satisfied, the delivered counter for π ( ππ‘π [ π ] ) is not increased and thismessage will be repeatedly delivered until the condition is satisfied.Once there is a process π such that there exists a round π and there is a set π of π + processes π for which π πππ π [ π ] [ π ] is equal to π ππππππ π , we say that the stability condition at line 53 is satisfied for π . At that time, π and π more processesagree on the collected arrays sent at round 0 by processes in π ππππππ π , and πππππππ‘ π holds the supremum of those collectedarrays. This is because whenever it received a start message, it updated its collect so that currently πππππππ‘ π reflects allcollects sent by processes in π ππππππ π . Thus, π can return its current collect as the snapshot-aux result. Since reliablebroadcast prevents Byzantine processes from equivocating, there are π more processes that broadcast the same π ππππππ set at that round, and any future round will βseeβ this set.To ensure liveness in case some correct processes complete a snapshot-aux instance before all do, we add a helpingmechanism. Whenever a correct process successfully completes snapshot-aux, it stores its result in a savesnap map, withthe auxnum as the key (either at line 24 or at line 59). This way, once one correct process returns from snapshot-aux,others can read its result at line 35 and return as well. To prevent Byzantine processes from storing invalid snapshots,each entry in the savesnap map is a tuple of the returned array and a proof of the arrayβs validity. The proof is the setof messages received by the process that stores its array in the current instance of snapshot-aux. Using these messages,correct processes can verify the legitimacy of the stored array. If a correct process reads from savesnap a tuple with aninvalid proof, it simply ignores it. We outline the key correctness arguments highlighting the main lemmas. Formal proofs of these lemmas appear in Appen-dix A. To prove our algorithm is Byzantine linearizable, we first show that all returned snapshots are totally ordered (bycoordinate-wise order):L
EMMA If two snapshot operations invoked by correct processes return π π and π π , then π π β₯ π π or π π < π π . Based on this order, we define a linearization. Then, we show that our linearization preserves real-time order, and itrespects the sequential specification. We construct the linearization πΈ as follows: First, we linearize all snapshot operationsof correct processes in the order of their return values. Then, we linearize every update operation by a correct processimmediately before the first snapshot operation that βseesβ it. We say that a snapshot returning π sees an update by process π that has timestamp π‘π if π [ π ] .π‘π β₯ π‘π . If multiple updates are linearized to the same point (before the same snapshot),we order them by their start times. Finally, we add updates by Byzantine processes as follows: We add update(v) by aByzantine process π if there is a linearized snapshot that returns π and π [ π ] .π£ππ = π£ . We add the update immediately beforeany snapshot that sees it.We next prove that the linearization respects the sequential specification.L EMMA
The π π‘β entry of the array returned by a snapshot invocation contains the value π£ last updated by an update(v) invoked by process π in πΈ , or its variableβs initial value if no update was invoked. ame the Wild with Byzantine Linearizability: Reliable Broadcast, Snapshots, and Asset Transfer Because an update is linearized immediately before some snapshot sees it and snapshots are monotonically increasing,all following snapshots see the update as well. Next, we prove in the two following lemmas that πΈ preserves the real-timeorder.L EMMA
If a snapshot operation invoked by a correct process π with return value π π precedes a snapshot operationinvoked by a correct process π with return value π π , then π π β€ π π . L EMMA
Let π be the return value of a snapshot operation π πππ π invoked by a correct process π . Let π’ππππ‘π π ( π£ ) bean update operation invoked by a correct process π that writes β¨ π‘π , π£ β© and completes before π πππ π starts. Then, π [ π ] .π‘π β₯ π‘π . It follows from Lemma 12 and the definition of πΈ , that if an update precedes a snapshot it is linearized before it, andfrom Lemma 11 that if a snapshot precedes a snapshot it is also linearized before it. The following lemma ensures that ifan update precedes another update it is linearized before it. That is, if a snapshot operation sees the second update, it seesthe first one.L EMMA
If update1 by process π precedes update2 by process π and a snapshot operation π πππ by a correct processsees update2, then π πππ sees update1 as well. Finally, the next lemma proves the liveness of our algorithm.L
EMMA (Liveness) Every correct process that invokes some operation eventually returns.
We conclude the following theorem:T
HEOREM Algorithm 2 implements an π -resilient Byzantine linearizable snapshot object for any π < π . P ROOF . Lemma 9 shows that there is a total order on snapshot operations. Using this order, we have defined alinearization πΈ that satisfies the sequential specification (Lemma 10). We then proved that πΈ also preserves real-timeorder (Lemmas 11 β 13). Thus, Algorithm 2 is Byzantine linearizable. In addition, Lemma 14 proves that Algorithm 2 is π -resilient. β‘ We have studied shared memory constructions in the presence of Byzantine processes. To this end, we have definedByzantine linearizability, a correctness condition suitable for shared memory algorithms that can tolerate Byzantinebehavior. We then used this notion to present both upper and lower bounds on some of the most fundamental componentsin distributed computing.We proved that atomic snapshot, reliable broadcast, and asset transfer are all problems that do not have π -resilientemulations from registers when π β€ π . On the other hand, we have presented an algorithm for Byzantine linearizablereliable broadcast with resilience π > π . We then used it to implement a Byzantine snapshot with the same resilience.Among other applications, this Byzantine snapshot can be utilized to provide a Byzantine linearizable asset transfer. Thus,we proved a tight bound on the resilience of emulations of asset transfer, snapshot, and reliable broadcast.Our paper deals with feasibility results and does not focus on complexity measures. In particular, we assume unboundedstorage in our constructions. We leave the subject of efficiency as an open question for future work. hir Cohen and Idit Keidar REFERENCES [1] Ittai Abraham, Gregory Chockler, Idit Keidar, and Dahlia Malkhi. 2006. Byzantine disk paxos: optimal resilience with byzantine shared memory.
Distributed Computing
18, 5 (2006), 387β408.[2] Yehuda Afek, Hagit Attiya, Danny Dolev, Eli Gafni, Michael Merritt, and Nir Shavit. 1993. Atomic snapshots of shared memory.
Journal of the ACM(JACM)
40, 4 (1993), 873β890.[3] Yehuda Afek, David S Greenberg, Michael Merritt, and Gadi Taubenfeld. 1995. Computing with faulty shared objects.
Journal of the ACM (JACM)
42, 6 (1995), 1231β1274.[4] Hagit Attiya, Maurice Herlihy, and Ophir Rachman. 1992. Efficient atomic snapshots using lattice agreement. In
International Workshop on DistributedAlgorithms . Springer, 35β53.[5] Alex Auvolat, Davide Frey, Michel Raynal, and FranΓ§ois TaΓ―ani. 2020. Money Transfer Made Simple: a Specification, a Generic Algorithm, and itsProof.
Bulletin of EATCS
3, 132 (2020).[6] Mathieu Baudet, Avery Ching, Andrey Chursin, George Danezis, François Garillot, Zekun Li, Dahlia Malkhi, Oded Naor, Dmitri Perelman, andAlberto Sonnino. 2019. State machine replication in the Libra blockchain.
The Libra Assn., Tech. Rep (2019).[7] Gabriel Bracha. 1987. Asynchronous Byzantine agreement protocols.
Information and Computation
75, 2 (1987), 130β143.[8] Christian Cachin, Rachid Guerraoui, and LuΓs Rodrigues. 2011.
Introduction to reliable and secure distributed programming . Springer Science &Business Media.[9] Vicent Cholvi, Antonio Fernandez Anta, Chryssis Georgiou, Nicolas Nicolaou, and Michel Raynal. 2020. Atomic Appends in Asynchronous ByzantineDistributed Ledgers. In . IEEE, 77β84.[10] Giuseppe Antonio Di Luna, Emmanuelle Anceaume, and Leonardo Querzoni. 2020. Byzantine generalized lattice agreement. In . IEEE, 674β683.[11] Rachid Guerraoui, Petr Kuznetsov, Matteo Monti, Matej PavloviΛc, and Dragos-Adrian Seredinschi. 2019. The consensus number of a cryptocurrency.In
Proceedings of the 2019 ACM Symposium on Principles of Distributed Computing . 307β316.[12] Prasad Jayanti, Tushar Deepak Chandra, and Sam Toueg. 1998. Fault-tolerant wait-free shared objects.
Journal of the ACM (JACM)
45, 3 (1998),451β500.[13] Barbara Liskov and Rodrigo Rodrigues. 2005. Byzantine clients rendered harmless. In
International Symposium on Distributed Computing . Springer,487β489.[14] Jean-Philippe Martin, Lorenzo Alvisi, and Michael Dahlin. 2002. Minimal byzantine storage. In
International Symposium on Distributed Computing .Springer, 311β325.[15] Achour MostΓ©faoui, Matoula Petrolia, Michel Raynal, and Claude Jard. 2017. Atomic read/write memory in signature-free byzantine asynchronousmessage-passing systems.
Theory of Computing Systems
60, 4 (2017), 677β694.[16] Satoshi Nakamoto. 2009.
Bitcoin: A peer-to-peer electronic cash system . Technical Report. Manubot.[17] Rodrigo Rodrigues and Barbara Liskov. 2003. Rosebud: A scalable byzantine-fault-tolerant storage architecture. (2003).[18] Gavin Wood et al. 2014. Ethereum: A secure decentralised generalised transaction ledger.
Ethereum project yellow paper , Quentin Bramas, RotemOshman, and Paolo Romano (Eds.). Schloss Dagstuhl - Leibniz-Zentrum fΓΌr Informatik, 4:1β4:16. https://doi.org/10.4230/LIPIcs.OPODIS.2020.416 ame the Wild with Byzantine Linearizability: Reliable Broadcast, Snapshots, and Asset Transfer
Appendix A BYZANTINE SNAPSHOT: CORRECTNESS L EMMA
For a correct process π , at each point during an execution πππππππ‘ π [ π ] contains the value signed by π withthe highest timestamp until that point. P ROOF . By induction on the execution; πππππππ‘ π [ π ] can change either at line 5 or at line 18. If it changes at line 5, π‘π π is increased and πππππππ‘ π [ π ] contains the value with the highest timestamp. By induction, no signed value encounteredat line 17 has s timestamp higher than the one in πππππππ‘ π [ π ] , so it is not updated at line 18. β‘ L EMMA
For a correct process π , πππππππ‘ π is monotonically increasing. P ROOF . Let π β [ π ] . We prove that every time the value in πππππππ‘ π [ π ] is updated from π to π β² , it holds that π β² .π‘π > π.π‘π . By the code πππππππ‘ π [ π ] changes either at line 5 or at line 18. In both cases, the value in πππππππ‘ π [ π ] is signedby π . If πππππππ‘ π [ π ] changes at line 18, then monotonicity is immediate from the condition at line 17. Otherwise, it changesat line 5, indicating that π = π and monotonicity follows from Lemma 15. β‘ Lemma 10.
The π π‘β entry of the array returned by a snapshot invocation in πΈ contains the value π£ last updated by an update(v) invoked by process π in πΈ , or its variableβs initial value if no update was invoked. P ROOF . Let π£ be the value in the π π‘β entry of the array returned by a snapshot , with a corresponding timestamp π‘π π£ . Bythe definition of πΈ , update(v) by process π with timestamp π‘π is linearized immediately before π‘π π£ β₯ π‘π . If π is correct andmultiple update operations by π are linearized at that point, then since π invokes updates sequentially and by Lemmas 15and 16 their start times are ordered according to the increasing timestamps. Thus, as updates are linearized by their starttimes, π£ matches the value of the last update. If π is Byzantine, since we add updates only for values at the moment theyare seen, π£ must match the value of the last update. Additionally, if π£ is an initial value, then no updates were linearizedbefore it in πΈ . β‘ O BSERVATION For a snapshot operation invoked by a correct process π , let π π be the collected array at line 8 andlet π be the return value. Then, π β₯ π π . P ROOF . Immediate from the condition at line 13. β‘ Lemma 11.
If a snapshot operation invoked by a correct process π with return value π π precedes a snapshot operationinvoked by a correct process π with return value π π , then π π β€ π π . P ROOF . Assume π invokes snapshot operation π πππ π , which returns π π before π invokes snapshot π πππ π , returning π π .Let π be the value of πππππππ‘ π that π reads at line 8 of π πππ π and let π be the value it writes in πππππππ‘ π at line 9. At theend of the last snapshot-aux in π πππ π , πππππππ‘ππ π β₯ π π either because the return value is πππππππ‘ππ π (if snapshot-aux returnsat line 60), or because π π is reflected in collect by the end of line 25 if it is a savesnap returned at line 36 or at line 58.Due to the monotonicity of collects (Lemma 16), π π β€ π . Because π reads π when calculating π , π β€ π . Finally,by Observation 3, π β€ π π and by transitivity we get that π π β€ π π . β‘ Lemma 12.
Let π be the return value of a snapshot operation π πππ π invoked by a correct process π . Let π’ππππ‘π π ( π£ ) be anupdate operation invoked by a correct process π that writes β¨ π‘π , π£ β© and completes before π πππ π starts. Then, π [ π ] .π‘π β₯ π‘π . P ROOF . Let π‘ be the time when π completes line 5 in π’ππππ‘π π ( π£ ) and writes β¨ π‘π , π£ β© . Let π‘ be the time when π reads πππππππ‘ π [ π ] at line 8 in π πππ π . By Lemmas 15 and 16, since π is correct, it follows that πππππππ‘ π [ π ] .π‘π β₯ π‘π at time π‘ β₯ π‘ .Thus, after line 9 in π πππ π πππππππ‘ π [ π ] .π‘π β₯ π‘π and by Observation 3, π [ π ] .π‘π β₯ π‘π . hir Cohen and Idit Keidar β‘ I NVARIANT For any correct process π that invokes snapshot-aux( π ), it holds that πππππππ‘ π is the supremum ofthe arrays in start message sent by processes in π ππππππ π from line 33 and until the return value of snapshot-aux( π ) isdetermined at line line 23 or at line 60. P ROOF . First, at line 33 π ππππππ π contains π itself, and π sends exactly its πππππππ‘ π array. The argument continues byinduction on steps of snapshot-aux( π ). Other than line 25, πππππππ‘ π and π ππππππ π change together: Whenever π receives astart message with an array π from process π , it updates πππππππ‘ π with the higher-timestamped values found in π and adds π to π ππππππ π (lines 42β 43). β‘ D EFINITION We say that the stability condition holds for a return value π of snapshot-aux( π ) return value π witha round π and a set of processes π if (1) | π | β₯ π + , (2) there is a set π β² β π so that for each π β π the union of alljsenders sets sent in π βs messages in rounds 1 to π is π β² , and (3) π is the supremum of the collects sent in start messagesof members of π β² . O BSERVATION If π is returned from snapshot-aux( π ) by a correct process π , then π satisfies the stability conditionfor some set π in some round π . P ROOF . Consider two cases. First, if π returns π at line 60, then the condition is satisfied for π with the round π thatsatisfies the condition at line 53 and the set of π + processes for which the condition at line 53 holds. π β² is the set in π ππππππ π at the time the condition is satisfied. Since messages are delivered in order, we get that π β² β π . Because thereturn value is πππππππ‘ π , (3) follows from Invariant 1.Second, if π adopts a saved snapshot π with a proof and returns at line 36 or at line 58, then the proof contains π + messages from some round π and corresponding start messages satisfying the stability condition. β‘ L EMMA
For a given π , Let π, π be two correct processes that return π π , π π from snapshot-aux( π ). Then π π β€ π π or π π > π π . P ROOF . By Observation 4, π π satisfies the stability condition for some set π in some round π . Let π β² be the setguaranteed from the definition. Also by Observation 4, π π satisfies the stability condition and some set π in some round π . Let π β² be the set guaranteed from the definition.Since | π | β₯ π + and | π | β₯ π + , there is at least one process π β π β© π . Due to reliable broadcast, π cannotequivocate with the set of processes ππ ππππππ sent in each round of snapshot-aux( π ). β’ If π = π : By property (2) of Definition 3 π β² = π β² , and by (3) π π = π π . β’ If π β π : Assume WLOG π < π . Since the union of all jsenders sets sent in π βs messages in rounds 1 to π is asuperset of those sent in rounds 1 to π , π β π and then by (3) π π β₯ π π . β‘ L EMMA
Let π, π be two correct processes returning π π , π π resp. from snapshot-aux with ππ’π₯ππ’π = π , such that π π > π π . Then when π begins any snapshot-aux π ( π β² ) for π β² > π , πππππππ‘ π > π π . P ROOF . Since π is correct, by Observation 4, π π satisfies the stability condition. Let π‘ be a time when the condition issatisfied. At time π‘ , there is at least one correct process π such that πππππππ‘ π β₯ π π . We show that either (1) π does not return ame the Wild with Byzantine Linearizability: Reliable Broadcast, Snapshots, and Asset Transfer π π or (2) π begins snapshot-aux π ( π β² ) with πππππππ‘ π > π π . If π begins snapshot-aux π ( π β² ) after π‘ , then when it updates itscollect at lines 30β31, it reads the values in πππππππ‘ π . By Lemma 16, πππππππ‘ π is greater than or equal to its value at time π‘ .Thus, we get that πππππππ‘ π β₯ πππππππ‘ π β₯ π π and (2) holds. Otherwise, π saves π π at line 59 before starting snapshot-aux( π β² ),which is before time π‘ . Between time π‘ and the time it returns π π , j checks stored snapshots (at line 21). When it does so, π reads π π , and since π π > π π and π returns the minimal array it sees, (1) holds. β‘ L EMMA
If snapshot-aux π ( π ) of a correct process π returns π π , there is a correct process π s.t. π invoked snapshot-aux π ( π ) and π π β₯ π π , where π π is the value of πππππππ‘ππ π after the collection at line 31 in snapshot-aux( π ) at π . P ROOF . If snapshot-aux π ( π ) returns at line 60, then π returns πππππππ‘ π and by Lemma 16, π π = πππππππ‘ π is greater than orequal to its value after the collection at line 31 so the lemma holds with π = π . Otherwise, snapshot-aux π ( π ) returns π π at line 36 or at line 58 and π π is an array saved in savesnap with a proof π signed by process π . Since π validates π π , therewas a round π such that |{ π | π πππ π [ π ] [ π ] = π ππππππ π }| β₯ π + . Thus, there was at least one correct process π in this set.Since π adds itself to π ππππππ π (line 32), π ππππππ π is broadcast by π at every round (line 52), and it is the set added to π πππ , the array π π sent in π βs start message is reflected in π π . This set is exactly the value of πππππππ‘ππ π after the collectionat line 31 in snapshot-aux( π ) at π , and hence π π β₯ π π . β‘ Lemma 9.
If two snapshot operations invoked by correct processes return π π and π π , then π π β₯ π π or π π < π π . P ROOF . By the code, π π is the return value of some snapshot-aux π ( π π ) and π π is the return value of some snapshot-aux π ( π π ) . WLOG, π π β₯ π π . β’ If π π = π π , the proof follows from Lemma 17. β’ If π π > π π : By Lemma 19, there is a correct process π that invoked snapshot-aux π ( π π ) , collected π π at line 31 ofsnapshot-aux π ( π π ) (where π π is the value of πππππππ‘ππ π at that time), and π π β₯ π π . Let π π be the return value ofsnapshot-aux π ( π π ) (note that π invokes snapshot-aux with increasing auxnums, so such a value exists). Considertwo cases. First, if π π > π π , then by Lemma 18, π π β€ π π . Thus, π π β€ π π β€ π π and the lemma follows. Otherwise, π π β€ π π . At the end of snapshot-aux π ( π π ) πππππππ‘ππ π β₯ π π because either the return value is πππππππ‘ππ π , or π π isreflected in collect by the end of line 25. Due to the monotonicity of collects (Lemma 16), π π β€ π π . We concludethat π π β€ π π β€ π π β€ π π , as required. β‘ Lemma 13.
If update1 by process π precedes update2 by process π and a snapshot operation π πππ by a correct processsees update2, then π πππ sees update1 as well. P ROOF . Let π be the return value of a snapshot that sees update2. By Observation 4, π is the supremum of πππππππ‘ arrays sent at line 33. If π sees update2, by Lemma 15, it means that π reflects πππππππ‘ π after line 5 of update2. After, π performed line 3 and update1 was reflected in πππππππ‘ π . Hence, π sees update1 as well. β‘ We now prove the liveness of our snapshot algorithm.L
EMMA
Every correct process that invokes snapshot-aux(auxnum) eventually returns. P ROOF . Assum by induction on auxnum that all snapshot-aux instances with π β² < π (if any) have returned at allcorrect processes. Then, for auxnum= π , all correct processes initiate reliable broadcast instances and broadcast β¨ , π β© . hir Cohen and Idit Keidar This is because all correct process invoke snapshot infinitely often. Since all messages by correct processes are eventuallydelivered, they all eventually complete line 50 in each round. Because | π ππππππ | is bounded, eventually the π ππππππ sets ofall correct process stabilize, and due to reliable broadcast, they contain the same set of processes for all correct processes.Thus, there is be a round π for which the condition at line 53 is satisfied. Therefore, at least one correct process returnsfrom snapshot-aux at line 60 (if it did not return sooner). Before returning, it updates its savesnap register at line 59. If itreturns at line 36 or at line 58 it also updates its savesnap register at line 24. Every other correct process that has not yetreturned from snapshot-aux will read the updated savesnap in the next while iteration and will return at line 36. β‘ Lemma 14.
Every correct process that invokes some operation eventually returns. P ROOF . If a correct process π invokes an update operation then by the code it returns in constant time. If π invokes asnapshot operation at time π‘ , let π be the collected array at line 8. Additionally, let π be the maximum ππ’π₯ππ’π of anysnapshot-aux operation that was initiated by some process before time π‘ . By Lemma 20, all snapshot-aux invocationseventually return. At snapshot-aux ( π + ) , all correct processes see π at lines 30β31 when they update their collect. Sincethe return value is the supremum of π + collect arrays, it is guaranteed that when π executes snapshot-aux ( π + ) , thereturned value πππ will satisfy πππ β₯ π . β‘β‘