Saguaro: Efficient Processing of Transactions in Wide Area Networks using a Hierarchical Permissioned Blockchain
Mohammad Javad Amiri, Ziliang Lai, Liana Patel, Boon Thau Loo, Eric Lo, Wenchao Zhou
SSaguaro: Efficient Processing of Transactions inWide Area Networks using a HierarchicalPermissioned Blockchain
Mohammad Javad Amiri Ziliang Lai Liana Patel Boon Thau Loo Eric Lo Wenchao Zhou Department of Computer and Information Science, University of Pennsylvania Department of Computer Science and Engineering, Chinese University of Hong Kong Department of Computer Science, Georgetown University {mjamiri, lianap, boonloo}@seas.upenn.edu, {zllai, ericlo}@cse.cuhk.edu.hk, [email protected] Abstract
The next frontier for the Internet leading by innovations in mobile computing, in particular,5G, together with blockchains’ transparency, immutability, provenance, and authenticity,indicates the potentials of running a new generation of applications on the mobile inter-net. A 5G-enabled blockchain system is structured as a hierarchy and needs to deal withdifferent challenges such as maintaining blockchain ledger at different spatial domains andvarious levels of the networks, efficient processing of cross-domain transactions, establishingconsensus among heterogeneous nodes, and supporting delay-tolerant mobile transactions.In this paper, we present
Saguaro , a hierarchical permissioned blockchain designed specific-ally for Internet-scale mobile networks. Saguaro benefits from the hierarchical structureof mobile network infrastructure to address the aforementioned challenges. Our extensiveexperimental results demonstrate the high potential of Saguaro being the first 5G-enabledpermissioned blockchain system in the community.
Blockchain, originally devised for Bitcoin [47], is a distributed data structure for recording transactionsmaintained by nodes without a central authority [15].
Permissioned blockchains, which target for commercialuses among a set of known but untrusted enterprises, are getting increasing traction across a wide spectrumof applications, e.g., cross-border payments [29], patient data exchange [51], and supply chain assurance [66],in recent years.The unique features of permissioned blockchains such as transparency, provenance, fault tolerance, andauthenticity, are appealing to various distributed applications, however, their scalability remains a majorchallenge. Partitioning the data into multiple shards that are maintained by different subsets (i.e., clusters)of nodes is a proven approach to improve the scalability of distributed databases, e.g., Spanner [18], and hasbeen used in permissioned blockchain systems. Nevertheless, processing cross-shard transactions especiallyin wide area networks is challenging. Sharding techniques follows either a decentralized flattened approach,e.g., SharPer [5] [3], or a centralized coordinator-based approach, e.g., AHL [19] to process cross-shard trans-actions. In the flattened approach, nodes of all involved shards participate in the consensus protocol, thus,if the involved shards are geographically far apart, the network distance results in high latency. Similarly,in the coordinator-based approach, since a single coordinator (a node or a cluster of nodes) processes allcross-shard transactions, the coordinator likely has a far network distance to either the shards or the clients,depending on where the shards are placed (close to the clients vs close to the coordinator). In either scenario,the performance of the system will be severely impacted. To reduce the latency of global communication,i.e., multiple rounds of cross-cluster message exchange, GeoBFT [30] [31] [32], replicates the entire ledgeron every node and establishes consensus on each transaction within only a single cluster. Clusters, however,communicate with each other to multicast their locally-replicated transactions and enable other clusters to a r X i v : . [ c s . D B ] J a n EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking Systemupdate their ledgers and execute transactions. Such cross-cluster communications for every single transactionresults in high latency.In the past few years, in the mobile computing world, there is an increasing trend towards infrastructuresthat are geared towards the edge and peer-to-peer computing. This is best epitomized by , the fifthgeneration technology standard for cellular networks, that has attracted extensive research interest andindustry adoption. 5G’s unique characteristics include very low latency, ultra-high reliability, device todevice communication, and increased network capacity and connectivity. 5G networks are cellular networks,in which the service area is divided into small geographical areas ( spatial domains ). A typical 5G networkinfrastructure is hierarchical — from edge devices, edge servers, fog servers, to cloud servers.The hierarchical infrastructure of wide area networks enables the efficient processing of transactions overwide area networks. First, since transactions are initiated and initially processed within lower-level domains,i.e., edge devices and edge servers, the load on the internal domains, e.g., cloud servers, is reduced, whichis suitable for 5G and edge networks. Second, for each cross-domain transaction, the internal domainwith the minimum total distance from the involved edge domains, i.e., the lowest common ancestor ofall involved domains, can be chosen as the coordinator, using the coordinator-based approach. Finally,the hierarchical structure of the network enables optimistic ordering of transactions where each involveddomain independently orders cross-domain transactions and higher-level domains validate the commitmentof transactions.While 5G suffers from trust and secure interoperability among sub-networks, immutable and decentral-ized blockchains enable distributed communication with high security and trustworthiness [59]. Buildingan Internet-scale hierarchical permissioned blockchain that can support mobile applications, however, ischallenging. We highlight some of the challenges below: Maintaining Hierarchical Ledgers.
A blockchain ledger used to be a flat append-only data structureshared among all entities. A hierarchical blockchain in contrast requires maintaining ledgers from differentdomains at different levels of the hierarchy. However, maintaining the full ledger of different child nodesand preserving ordering dependency between transactions of different domains (resulting from cross-domaintransactions) are challenging.
Heterogeneous Nodes.
Nodes in different domains might follow different failure models, i.e., crash orByzantine. Moreover, the system needs to process transactions even when the edge devices do not trust theedge servers (e.g., traveling abroad) or when the connection to the edge servers is unavailable (e.g., in ruralareas).
Edge Node Mobility.
Edge devices may transiently or persistently move across the spatial domains.Furthermore, a transaction may be started by a mobile node in one domain, buffered, and then committedby the same node once it reaches another domain.In this paper, to address the above challenges, we present
Saguaro , a hierarchical permissioned blockchainsystem designed specifically for wide area networks. In Saguaro, nodes are organized in a hierarchicalstructure following the wide area network infrastructure from edge devices to edge, fog, and cloud serverswhere nodes at each level are further clustered into fault-tolerant domains . Saguaro processes cross-domaintransactions efficiently by either enabling an optimistic ordering of transactions or relying on the lowestcommon ancestor of the involved domains (i.e., a higher level domain with minimum total distance from theinvolved domains), resulting in lower latency. Saguaro is also able to process delay-tolerant transactions thatare initiated by mobile devices. Specifically, Saguaro makes the following three key technical contributions:•
DAG-Structured Summarized Views.
The hierarchical structure of Saguaro enables internaldomains to not maintain the full information but just a summarized view (e.g., selected columns oraggregated values) resulting in higher performance and enhanced privacy. In addition, while ledgersat the lower-levels are formed as linear chains for consistency, maintaining cross-domain transactionsresult in ledgers structured as directed acyclic graph at the internal domains to capture dependencies.•
Heterogeneous Consensus.
Saguaro provides a suite of consensus protocols to process transac-tions within and across domains with crash-only or Byzantine nodes. Saguaro benefits from thehierarchical structure of wide area networks to establish consensus on cross-domain transactions.The consensus protocols are hierarchy-aware and geographically optimized.•
Delay-tolerant Mobile Consensus.
Saguaro supports mobility of nodes by providing delay-tolerant mobile consensus where edge devices can initiate transactions in different (spacial) domains2EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking Systemfar from their initial local area. To establish delay-tolerant mobile consensus, Saguaro relies oncommunication between local and remote domains to share the node’s history.We validated these technical innovations by developing a prototype of Saguaro, where our evaluation resultsacross a wide range of workloads demonstrate the effectiveness of Saguaro in scalably support a range ofcross-domain and delay-tolerant transactions.The rest of this paper is organized as follows. Section 2 discussed the background and motivates Saguaroby several possible applications. The Saguaro model is introduced in Sections 3. Section 4 and Section 5present transaction processing in Saguaro. Section 6 evaluates the performance of Saguaro. Section 7discusses related work, and Section 8 concludes the paper.
We motivate Saguaro, first through technology trends and opportunities enabled by 5G, followed by newapplications that arise from these innovations.
5G is the fifth generation of cellular network technologies developed in response to meet the growing demandfor a variety of new services such as autonomous vehicles and massive IoT. Using different technologies suchas the millimeter-wave spectrum, massive multiple-input and multiple-output, and network slicing, 5G aimsto provide high data rates, coverage, connectivity, and bandwidth, with a massive reduction in latency andenergy consumption. 5G, in addition, provides device-to-device (D2D) communication that enables edgedevices to communicate directly with each other. The D2D communication accelerates the developmentof edge-centric applications [43] and also enables edge devices to establish consensus among themselves toprocess transactions. Processing transactions by edge devices and independent of edge servers is needed dueto the lack of trust or connectivity between edge devices and edge servers or to offload traffic from edgeservers.5G networks are cellular networks, in which the service area is divided into small domains . In a 5G structure,machines (i.e., devices, servers) are organized in a hierarchical structure where at the leaf level, edge devices,e.g., smart cars, smartphones, within a local area (i.e., a spatial domain) are connected to each other and toan edge server (as the parent vertex). Nearby edge servers (e.g., campus area) are then connected to a fogserver (e.g., metropolitan area) and finally, at the root level, cloud servers are placed. The network mightinclude different layers of edge, fog, or cloud servers, i.e., the hierarchy might have more than four layers.Furthermore, to provide fault tolerance, each vertex of the hierarchy, e.g., edge, fog, or cloud servers, consistof a set of nodes, called a domain . We envision permissioned blockchain as a promising technology to realize the full potential of 5G. Permis-sioned blockchains over the 5G network can enable many new applications. We show how different distributedapplications benefit from a hierarchical blockchain system and discuss the aforementioned challenges in eachapplication.
Accountable ride-sharing and 5G-enabled gig economy.
In the ridesharing applications, drivers giverides to travelers through platforms, e.g., Uber, Lyft, and Curb. Participants in ridesharing environments,however, require to satisfy global regulations, e.g., the total work hours of a driver, who might work formultiple platforms, may not exceed 40 hours per week to follow the
Fair Labor Standards Act [6]. Asanother example, California Proposition 22 challenges Assembly Bill 5 ( AB5 ) by imposing its own set ofregulations, e.g., if a driver works at least 25 hours per week, platforms are required to provide healthcaresubsidies. The transparency of blockchains helps in verifiability of global regulations [6]. Ride-sharingapplications also deal with the mobility of edge devices (i.e., cars) across spatial domains (e.g., a ridefrom JFK to Manhattan involves multiple edge servers). A 5G-enabled hierarchical permissioned blockchainsystem can exactly fill the bill by supporting cross-domain transactions. Moreover, while lower-level domains https://ballotpedia.org/California_Proposition_22,_App-Based_Drivers_as_Contractors_and_Labor_Policies_Initiative_(2020) https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=201920200AB5 Mobile delay-tolerant micropayments.
Most popular micropayment infrastructures do not allow usersto do cross-application payments, e.g., an Apple Pay sender cannot send money to a PayPal receiver. How-ever, a 5G-enabled permissioned blockchain system can enable so. For micropayments under the same spatialdomain and the same application domain (e.g., Alice pays Bob at a coffee shop, both using Apple Pay),transactions can be committed efficiently and securely by offloading most part of the transactions (e.g., con-sensus) to 5G’s D2D communications framework and propagating the consensus results (possibly an abstractversion) to the ledger at a higher level. For micropayments under the same spatial but different applicationdomains (e.g., Alice pays Bob in the same coffee shop, but Alice is using Apple Pay while Bob is usingPayPal), transactions can be executed efficiently if each edge server hosts ledgers from different paymentcompanies and executes the cross-domain transactions at the edge. For micropayments that cross spatialand application domains (e.g., Alice in the West pays Bob in the East), transactions can also be executedefficiently when ledgers are deployed in the entire wide-network hierarchy but (cross-domain) consensus isestablished only among the involved domains. The system needs to support the mobility of nodes as well.
Resource management and provisioning.
As mobile devices and Internet-of-Things (IoT) devicesscale to permeate the Internet, security concerns will become increasingly important as IoT devices can beeasily compromised if software updates are not done. An example of a security attack is denial-of-service(DoS) attacks. One promising application for blockchain technology is doing resource provisioning over theInternet. This can come in the form of provisioning network resources for quality-of-service (QoS) trafficor provisioning against over usage of networks that lead to DoS attacks. Given that network resources areshared across multiple entities, one can use blockchain as a tamper-evident logging mechanism to track andenforce network resource utilization of edge devices and service providers in a hierarchical manner.
Secure network slicing.
Within the core of the 5G network, network services can be broken into severalprivate slices, one for each tenant. Network services are deployed as virtualized network functions that arereplicated across nodes for fault tolerance. These network functions can leverage blockchains to providetamper-evident communication and cross-domain transactions in the cloud [74].
Saguaro is a permissioned blockchain system designed specifically to achieve high performance in wide areanetworks. In this section, we present the infrastructure and the blockchain ledger of Saguaro.
Saguaro is a distributed system consisting of edge devices, edge servers, fog servers, and cloud servers,organized in a hierarchical tree structure. Each vertex of the tree, called a domain , further consists of a4EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking SystemFigure 2: An Example of Saguaro Blockchain Ledgersufficient number of nodes to guarantee fault tolerance. Nodes within each domain follow either the crash orByzantine failure model. In the crash failure model, nodes may fail by stopping, and may restart, whereas,in the Byzantine failure model, faulty nodes may exhibit arbitrary, potentially malicious, behavior. Crashfault-tolerant protocols, e.g., Paxos [40], guarantee safety in an asynchronous network using 2 f +1 crash-onlynodes to overcome f simultaneous crash failures while in Byzantine fault-tolerant protocols, e.g., PBFT [16],3 f +1 nodes are usually needed to guarantee safety in the presence of f malicious nodes [11]. Figure 1 presentsa sample 4-layer Saguaro infrastructure consisting of 11 domains where domains have heterogeneous failuremodels. For example, D includes 4 Byzantine nodes (3 f +1 where f = 1) while D consists of 5 crash-onlynodes (2 f + 1 where f = 2).The hierarchy is globally known by all nodes and the network as well domains are reconfigurable, e.g.,nodes can be added, as long as the domains are still fault-tolerant. Edge devices are also authenticated bycorresponding edge servers (height − message digest is a fixed-length numeric representation of the contents of a messageproduced by collision-resistant hash functions. Message digests are used to protect the integrity of a messageand detect changes and alterations to any part of the message. We denote a message m signed by node r as h m i σ r and the digest of a message m by ∆( m ). Saguaro also uses threshold signatures [61] [14] in domainsconsisting of Byzantine nodes to reduce the quadratic communication cost of consensus among Byzantinenodes. In a ( k, n )-threshold signature scheme, a single public key is maintained by all nodes within a domainand each of the n nodes holds a distinct private key. In this paper, we use a threshold of k = 2 f + 1 (sinceeach Byzantine domain includes n = 3 f + 1 nodes). For signature verification, we assume that all nodes haveaccess to the public keys of all other nodes. We also assume that a strong adversary can coordinate maliciousnodes and delay communication to compromise the service. However, the adversary cannot subvert standardcryptographic assumptions.In most situations, edge severs ( height − domains) process transactions that are initiated by edge devices.Depending on the infrastructure and due to the lack of trust or connectivity between edge devices andedge servers or to offload traffic from edge servers, consensus among edge devices within a leaf domainand independent of edge servers might also be needed. This is also consistent with the D2D feature of 5Gnetworks. We call the lowest domain, i.e., either the height − edge domain . Transactions in a blockchain system are recorded in an append-only data structure, called the blockchainledger . In Saguaro, each domain processes a different set of transactions, hence, each domain maintains itsown blockchain ledger where the ledger is replicated on every node within the domain.Leaf domains (i.e., edge devices) maintain blockchain ledger in rare cases either due to lack of trust orconnectivity between the leaf domain and the corresponding height − − − − totally ordered to ensure data consistency. Thetotal order of transactions is captured by chaining transaction blocks together, i.e., each block has a sequencenumber or includes the cryptographic hash of the previous transaction block.A domain in height − directed acyclic graph to capture the ordering dependencies.If a leaf domain maintains the ledger (i.e, is an edge domain), the ledger of the leaf domain and the corres-ponding height − D is an edge domain (i.e.,maintains the ledger). As shown, the ledgers of leaf and height − − −
31 where domains D , D , and D are involved in. However, in height − −
42, the ledger is formed as a DAG.In Saguaro, nodes proceed through a succession of rounds where at the end of each round (i.e., after somepredefined time interval), each domain sends a block message including all transaction records that it hasreceived from its child domain(s) in that round (i.e., a chunk of its blockchain ledger) to its parent domain.The predefined time interval for the domains at the same level is the same, however, domains at higher levelshave larger time intervals. If a domain has not received any transaction in that round, it sends an empty block message. Depending on the application, the domain might send an abstract version of the recordswhere some attributes of a transaction record might not be sent. For example, in the ridesharing use case,it might be sufficient to send only the working hour attribute of each record to the higher-level nodes. Theabstraction strategy is deterministic, predefined, and known by all nodes.Figure 3 presents a snapshot of Saguaro for the same network as Figure 1 assuming all leaf domains areedge domains. Each height − D , has received blocks B
110 and B
110 from leaf domain D . Each height − D , has created a block of transactionsincluding transaction blocks received from its child domain, and finally, the root domain D has appendedblock B − B − B − B −
1, and B − D and D . In this example, the time interval of height − − Processing transactions requires establishing consensus on a unique ordering of incoming requests. Consensusprotocols use the State Machine Replication (SMR) algorithm [39]. The algorithm has to satisfy four mainproperties [13]: (1) agreement: all non-faulty nodes must agree on the same value, (2)
Validity (integrity): a committed value must have been proposed by some (non-faulty) node, (3)
Consistency (total order): allnon-faulty nodes commit the same values in the same order, and (4) termination: eventually every nodecommits some value. The first three properties are known as safety and termination is known as liveness .6EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking SystemAs shown by Fischer et al. [23], in an asynchronous system, where nodes can fail, consensus has no solutionthat is both safe and live. Based on that impossibility (FLP) result, in Saguaro, safety is guaranteed in anasynchronous network that can drop, delay, corrupt, duplicate, or reorder messages. However, a synchronyassumption is needed to ensure liveness. Transactions of the system are either internal, i.e., access recordswithin a single domain, or cross-domain, i.e., access records across different domains. We first present theinternal consensus protocol to process internal transactions followed by the cross-domain consensus protocolto process cross-domain transactions.
The internal consensus is needed among the nodes within a single domain. While nodes within edge (i.e., leafor height −
1) domains establish consensus on the order of every single request that they receive from edgedevices, nodes in higher-level domains agree on block messages that they receive from their child domains.Edge devices (i.e., clients) initiate transactions by sending request messages to the primary node (a pre-electednode that initiates consensus) of the corresponding edge domain. The block messages, however, are sent bythe primary node of a domain to all nodes of its parent domain upon completion of a round (i.e., after apredefined time interval). Depending on the failure model of the child domain, the block message is signed byeither the primary (if nodes are crash-only) or at least 2 f + 1 nodes (if nodes are Byzantine). If the primarynode of the parent domain has not received the block message from a child domain after a predefined time(e.g., the primary of the child domain might be faulty), it sends a block-query message to all nodes of thechild domain. To ensure that the completion of each round is deterministic on all nodes of a domain, theprimary puts a "cut" symbol (parameter) into the propose message of the last request informing other nodesthe completion of a round. Since nodes establish consensus on the received messages, if a malicious primarysends the "cut" symbol maliciously it will be easily detected.The internal consensus protocol of Saguaro is pluggable and depending on the failure model of nodes, acrash fault-tolerant protocol, e.g., Paxos [40] or a Byzantine fault-tolerant protocol, e.g., HotStuff [71],can be used. Upon receiving a request message, the primary node of the edge domain assigns a sequencenumber to the transaction and initiates the consensus protocol by multicasting a propose message includingthe transaction and its digest (cryptographic hash) to the nodes of its domain. The primary node of eachhigher-level domain, however, waits for the block messages of every child domain (completion of a round),provides a unique ordering among them (as mentioned in Section 3), and multicasts a propose messageincluding a sequence number and a transaction block (i.e., including all transactions of received ledgers) tothe nodes of its domain. Since nodes have already received the block messages, they can validate the orderof transactions within the transaction block. Once consensus is achieved, nodes append the transactionsblock and the corresponding commit messages to their ledgers. The commit messages are appended to theledger to guarantee immutability. Indeed, commit messages include the digest of the transaction block andby appending them to the ledger, any attempt to alter the block data can easily be detected. Cross-domain transaction access records across different edge domains, e.g., a micropayment transactionwhere the sender and receiver belong to two different domains. To ensure data consistency, such transac-tions are required to be appended to the blockchain ledgers of all involved domains in the same order. Toestablish consensus on cross-domain transactions, the decentralized flattened, e.g., SharPer [5], and cent-ralized coordinator-based, e.g., AHL [19], approaches have been proposed. However, as discussed earlier,both approaches suffer from high latency especially in wide area networks. In the flattened approach, e.g.,SharPer [5], a consensus protocol is run among the nodes of all involved domains. The flattened approachdoes not require a centralized coordinator and can order cross-domain transactions with non-overlappingdomains in parallel. However, in a wide area network where the involved domains might be far apart, theoverall performance will severely be impacted due to the network latency. The coordinator-based approach,e.g., AHL [19], on the other hand, is more centralized by relying on a single coordinator domain (calledreference committee in AHL) to order cross-domain transactions. However, the coordinator-based approach,similar to the flattened approach, suffers from network latency especially in a wide area network becausedepending on where the domains are placed (close to the clients or close to the coordinator), the coordin-ator has a far network distance from either the involved domains or the clients. In this section, we showhow the hierarchical structure of Saguaro enables the efficient processing of cross-domain transactions usingcoordinator-based and optimistic approaches. 7EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking System
Algorithm 1
Coordinator-based Cross-Domain Consensus init(): r := node_id d c := coordinator (lowest common ancestor) domain π ( d ) := the primary node of domain d π ( D ) := set of primary nodes of involved domains (participants) D upon receiving valid request m and r ∈ π ( D ) forward request m to d c upon receiving valid request m and r is π ( d c ) if r is not processing request m where m and m intersect in at least 2 domains establish consensus on m among nodes in d c send h PREPARE , n c , δ, m i σ to D (signed by π ( d c ) or 2 f + 1 nodes) upon receiving valid h PREPARE , n c , δ, m i σ message(s) and ( r = π ( d i ) ∈ π ( D )) if r is not processing request m where m and m intersect in at least 2 domains establish consensus on the message among nodes in d i send h PREPARED , n c , n i , δ, r i σ to d c (signed by π ( d i ) or 2 f + 1 nodes) upon receiving valid h PREPARED , n c , n i , δ, r i σ from every d ∈ D and r == π ( d c ) establish consensus on the order of m within d c multicast h COMMIT , n i − n j − ... − n k , δ, r i σ to D (signed by π ( d c ) or 2 f + 1 nodes) upon receiving a valid h COMMIT , n i − n j − ... − n k , δ, r i σ message and r ∈ D append the transaction and the commit message to the ledger send h ACK , n c , n i − n j − ... − n k , δ, r i σr to π ( d c ) The coordinator-based approach in Saguaro is inspired by the traditional coordinator-based sharding tech-niques in distributed databases as well as the sharded permissioned blockchain AHL [19], where the two-phasecommit protocol is used for communication between the coordinator and all involved participant domains.Saguaro relies on the Lowest Common Ancestor (LCA) domain of all involved edge domains (participants)to play the coordinator role and process cross-domain transactions initiated by edge devices. Since thehierarchy is structured based on the geographical distance of nodes, the lowest common ancestor domain hasthe optimal location to minimize the total distance (i.e., latency). In comparison to existing coordinator-based approaches, Saguaro, however, deals with several new challenges. First, in Saguaro, in contrastto sharded databases where all nodes follow crash failure model, the coordinator and the involved domains(participants) might follow different possibly Byzantine failure model. As a result, messages from a Byzantinecoordinator or Byzantine participant domain must be certified by at least 2 f + 1 (out of 3 f + 1) nodes ofthe domain (since the primary node of the domain might be malicious). Second, in Saguaro, in contrastto the coordinator-based approaches where a single coordinator (node or domain) sequentially orders allcross-domain transactions, there are multiple independent coordinator domains in the network, i.e., anydomains in height − m (between d i , d j , and d k ) and m (between d i , d j , and d l ) are initiated, Saguaro must guarantee that m and m are appended to the ledger ofboth d i and d j domains in the same order, i.e., either m → m or m → m . Since the LCA of d i , d j , and d k might be different from the LCA of d i , d j , and d l , Saguaro can not rely on the LCA domain to guaranteeconsistency.The normal case operation of the coordinator-based approach is presented in Algorithm 1. Although notexplicitly mentioned, every sent and received message is logged by nodes. As indicated in lines 1-5 of thealgorithm, d c is the coordinator domain, π ( d ) represents the primary node of domain d , and D and π ( D )are the set of involved domains in the transaction and their primary nodes.Once the primary node of an involved domain receives a valid cross-domain transaction m , as shown inlines 6 −
7, the primary forwards it to the LCA of the involved domains. Upon receiving a cross-domaintransaction, as presented in lines 8 −
11, the primary node of the LCA domain validates the message. If theprimary node π ( d c ) is currently processing another cross-domain transaction m (i.e., has not sent commit message for m ) where the involved domains of two requests m and m intersect in at least two domains,the node does not process the new request m before the earlier request m gets committed. This is needed8EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking Systemto ensure cross-domain requests are committed in the same order on overlapping domains (i.e., consistencyproperty). Otherwise, node π ( d c ) assigns a sequence number n c to m and initiates consensus on request m in the coordinator domain. Once consensus is established, the primary node sends a signed prepare messageincluding the sequence number n c , request m and its digest δ = ∆( m ) to the nodes of all involved domains.Note that if the nodes of the LCA domain are Byzantine, as explained earlier, a threshold signature is usedwhere 2 f +1 (out of 3 f +1) nodes of the LCA domain sign the prepare message.Upon receiving a valid prepare message, as shown in lines 12 −
15, if the primary node of an involved domain d i is not processing another cross-domain transaction m where the involved domains of two requests m and m intersect in at least two domains, the primary assigns a sequence number n i to request m and initiatesconsensus in d i on the order of the message. Once consensus is achieved, the primary node of each involveddomain d i sends a signed prepared message to d c (as before, for Byzantine nodes, a threshold signatureis used). As shown in lines 16 −
18, when primary node π ( d c ) receives valid prepared messages from everyinvolved domain, the coordinator domain establishes consensus and sends a signed commit message to everynode of all involved domains. Otherwise (if some involved domain does not agree with the transaction), thedomain sends a signed abort message. The sequence number of the commit messages is indeed a concatenationof the received sequence numbers from all involved domains. The nodes of the LCA domain do not appendthe transaction to their ledgers in this step and update their ledgers only after receiving block messages.Upon receiving a valid commit message, as shown in lines 19 −
21, each node considers the transaction ascommitted and sends an ack message to the coordinator domain. If all transactions with lower sequencenumbers have been committed, the node appends the transaction and the corresponding commit message tothe ledger and executes it. This ensures that all nodes execute requests in the same order as required toensure safety. Depending on the application, a reply message including the execution results might also besent to the edge device (requester) by either the primary (if nodes are crash only) or all nodes (if nodes areByzantine) of the domain that has received the request.It should be noted that in an unlikely situation where cross-domain transactions (1) are concurrent, (2)overlap on at least two domains, and (3) are received by overlapping domains in a different order, ensuringconsistency (total order) might result in a deadlock situation. In such a situation, each domain waits forthe commit message of one transaction before processing another one. This happens when the LCA of twotransactions are different, i.e., if the LCA is the same, the LCA never initiates the second transaction,as shown in Algorithm 1, lines 8 −
11. Therefore, once the timer of an LCA domain for its cross-domaintransaction is expired, the LCA aborts the transaction (i.e., resolves the deadlock) by sending a new prepare message to the involved domains. Saguaro assigns different timers to different domains to prevent paralleland also consecutive aborting of transactions in case of deadlock situations.
Primary Failure Handling.
The primary failure handling routine improves liveness by allowing thesystem to make progress when a primary node fails. If the primary of either the LCA or a participantdomain is faulty, the primary failure handling routine of the internal consensus protocol, e.g., view changein PBFT [16], is triggered by timeouts to elect a new primary. In particular, for cross-domain transactions,if node r of an involved domain does not receive a commit message from the LCA domain for a preparedrequest and its timer expires, the node sends a h COMMIT-QUERY , n c , n i , δ, r i σ r message to all nodes of the LCAdomain where n c and n i are the sequence numbers assigned by the primary nodes of LCA and d i and δ isthe digest of the request. Similarly, if node r in the LCA domain has not received prepared message from aninvolved domain soon enough, it sends a h PREPARED-QUERY , n c , δ, r i σ r to all nodes of the involved domain. Ineither case, if the message has already been processed, the nodes simply re-send the corresponding response.Nodes also log the query messages to detect denial-of-service attacks initiated by malicious nodes. If thequery message is received from a majority of a domain (i.e., f + 1 crash-only or 2 f + 1 Byzantine nodes),the primary will be suspected to be faulty resulting in running the failure handling routine. Note that sincein all communications between a participant and an LCA domain, the primary node of the sender domainmulticasts messages, e.g., request , prepare , or prepared , to all nodes of the receiver domain, if the primary of thereceiver domain does not initiate consensus on the message among the nodes of its domain, it will eventuallybe suspected to be faulty by the nodes. Finally, if an edge device does not receive reply soon enough, itmulticasts the request to all nodes of the domain that it has already sent its request. If the request hasalready been processed, the nodes simply send the execution result back to the edge device. Otherwise, ifthe node is not the primary, it relays the request to the primary. If the nodes do not receive prepare messages,the primary will be suspected to be faulty, i.e., it has not multicast request to the LCA domain. Correctness.
Consensus protocols have to satisfy safety and liveness . We briefly analyze the safety (agree-ment, validity, and consistency) and liveness (termination) properties of Saguaro.9EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking System
Lemma 4.1 ( Agreement ) If node r commits request m with sequence number h , no other non-faulty nodecommits request m ( m = m ) with the same sequence number h . Proof:
We assume that the internal consensus protocol of both coordinator and participants domains ensuresagreement. Let m and m ( m = m ) be two committed cross-domain requests with sequence numbers h = [ h i , h j , h k , ... ] and h = [ h k , h l , h m , .. ] respectively. Committing a request requires matching prepared messages from n − f different nodes of every involved domain. Therefore, given an involved domain d k inthe intersection of m and m , at least a quorum of n − f nodes of d k have sent matching prepared messagesfor m and at least a quorum of n − f nodes of d k have sent matching prepared messages for m . Since any 2quorums intersect on at least one non-faulty node, h k = h k , hence, h = h . Lemma 4.2 ( Validity ) If a non-faulty node r commits m , then m must have been proposed by some node π . Proof:
If nodes follow the crash failure model, since crash-only nodes do not send fictitious messages, validityis ensured. With byzantine nodes, validity is guaranteed based on standard cryptographic assumptions aboutcollision-resistant hashes, encryption, and signatures which the adversary cannot subvert them (as explainedin Section 3). Since all messages are signed (by 2 f + 1 nodes) and either the request or its digest is includedin each message (to prevent changes and alterations to any part of the message), if request m is committedby non-faulty node r , the same request must have been proposed earlier by some node π . Lemma 4.3 ( Consistency ) Let D µ denote the set of involved domains (participants) for a request µ . Forany two committed requests m and m and any two nodes r and r such that r ∈ d i , r ∈ d j , and { d i , d j } ∈ D m ∩ D m , if m is committed before m in r , then m is committed before m in r . Proof:
As shown in lines 12 −
15 of Algorithm 1, when node r of a participant domain d i receives a prepare message for some cross-domain transaction m , if the node is involved in another uncommitted cross-domaintransaction m where | D m ∩ D m | >
1, i.e., some other domain d j is also involved in both transactions, node r does not send a prepared message for transaction m before m gets committed. Since committing request m requires a quorum of prepared messages from every involved domains, m cannot be committed until m iscommitted. As a result, the order of committing messages is the same in all involved domains. As shownin Algorithm 1, lines 8 −
11 the coordinator domain d c also checks the same condition before sending prepare messages. Property 4.4 ( Termination ) A request m issued by a correct client eventually completes.Due to the FLP result [23], Saguaro guarantees liveness only during periods of synchrony. Saguaro addressesliveness in primary failure and deadlock situations. First, if the primary of either the LCA or a participantdomain is faulty, e.g., does not multicast valid request , prepare , prepared , or commit messages, as explainedearlier, its failure will be detected and using the primary failure handling routine of the internal consensusprotocol, a new primary will be elected. Second, Saguaro, as explained earlier, addresses deadlock situationsresulting from concurrent cross-domain transactions that overlap on at least two domains and are receivedby overlapping domains in a different order to ensure liveness. The coordinator-based consensus protocol of Saguaro is more efficient than the existing coordinator-basedprotocols because Saguaro first, relies on the lowest common ancestor domain to minimize the distancebetween the coordinator and involved (participant) domains, and second, processes cross-domain transactionsin parallel. However, it still requires multiple rounds of intra- and cross-domain communication. To reducethe latency of coordinator-based protocol, Saguaro can leverage the hierarchical structure of the networkto enable the optimistic processing of cross-domain transactions. In the optimistic protocol, each involveddomain optimistically processes and commits a cross-domain transaction independent of other involveddomains assuming that all other involved domains commit the transaction as well. Since the transactionswill propagate up, nodes in higher levels and eventually the LCA of all involved domains can check thecommitment of the transaction.In the optimistic approach, upon receiving a valid cross-domain request from an authorized edge device, therequest is multicast to the nodes of all other involved domains. Since the primary might be malicious andnot send the request to some involved domains, all nodes multicast the request to ensure that other domainsreceive the request. Each involved domain (including the initiator domain) then, uses its internal consensus10EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking SystemFigure 4: Example of Optimistic Cross-Domain Consensusprotocol to optimistically establish agreement on the order of the transaction, append the transaction to itsledger (assuming all other involved domains also append the transaction), and execute it. For each executedcross-domain transaction t , nodes of a domain maintain a list of transactions (both internal and cross-domain)that are executed after t and have direct or indirect data (i.e., read-write) dependency to transaction t . Iftransaction t gets aborted, e.g., some other involved domain does not commit the transaction, all data-dependent committed transactions need to be aborted as well. When the domain is informed (by theLCA domain) about the commitment or abortion of transaction t , the domain no longer maintains the list.Figure 4 presents ledgers of different domains using the optimistic cross-domain consensus protocol for thesame network as Figure 1 assuming all edge domains are in height −
1. In this figure, m b is a cross-domaintransaction between D , D , and D and m i and m j are between D , and D . Each domain maintains alist of data-dependent transactions for each cross-domain transaction, e.g., in D , m g has data dependencyto m b .Each edge domain processes all internal and cross-domain requests and upon completion of a round (i.e.,the predefined time interval) sends a block message including the committed transactions, non-committed(aborted) cross-domain transactions (to inform other domains), and the dependency lists for cross-domaintransactions (within this block and the previous blocks) that have not yet been decided by all their involveddomains to its parent domain.Each parent domain and eventually the LCA of all involved domains in a cross-domain transaction first,ensures that concurrent cross-domain transactions (if any) have been appended to the ledger of the intersec-tion domains in the same order (i.e., consistency). Otherwise, one of the transactions will be aborted. Forexample, in Figure 4, m i and m j are appended to the ledger of D and D in an inconsistent order, hence,domain D aborts m j . Saguaro guarantees that aborting transactions is deterministic, i.e., all higher-leveldomains reach the same decision on choosing transactions to be aborted, e.g., abort the transaction with thelowest id.Note that intermediate domains between involved domains and the LCA domain might receive the transactionfrom a subset (more than one) of involved domains and be able to partially check the consistency and earlyabort in case of inconsistency. For example, in Figure 4, domain D receives m b from D and D . Uponfinding an inconsistency, the primary node of the domain marks one of the transactions and all its data-dependent transactions as aborted, e.g., in Figure 4, m j is marked as aborted. The primary also sendsan abort message (singed by a crash-only primary or 2 f + 1 Byzantine nodes) including the digest of therequest to the nodes of the involved domains. Involved domains need to roll back the aborted cross-domaintransaction and all its data-dependent ones. Since transactions have been already appended to the ledger,the domains append compensating transactions to the ledgers to cancel the effect of those transactions.Each intermediate and eventually the LCA domain then checks whether the cross-domain transaction iscommitted by involved domains. The intermediate domains can check the commitment of the transaction bya subset of the involved domains. For example, in Figure 4, domain D can check whether m b is committedby D and D while the LCA domain D can also check the commitment of m b by D as well. If thetransaction is committed by all involved domains, the transaction will be appended to the ledger and uponthe completion of the round sent to the parent domain. Once the primary of the LCA domain receives thetransaction from all involved domains, it sends a signed commit message to all domains informing them thatthe transaction is committed.If the transaction has not been appended to the blockchain ledger ( block message) of an involved domain(due to the asynchronous nature of the network), the intermediate or the LCA domain does not append the11EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking Systemtransaction and waits for the next block messages. The domain does not append the following transactionswithin the block message to its ledger as well. This is needed because there might be an inconsistency issueand the domain might need to mark the transactions as aborted (as explained earlier).If a node of an involved domain does not receive the commit message for cross-domain transaction m aftera predefined time, it sends a signed commit-query including the digest of m to its parent domain. If theparent domain has already been informed about the decision, it sends the commit (or abort ) message tothe node. Otherwise, it forwards the commit-query to its parent domain. If the LCA domain receives the commit-query and the transaction has been processed, the domain sends the commit (or abort ) message tothe node, otherwise, it sends a commit-query to the involved domain(s) that has not sent their transactions.Upon receiving a valid commit-query message, if the domain has not processed the transaction, it immediatelystarts processing the transaction. If the request is not committed in some predefined number of rounds byall involved domains it is considered to be aborted. It should be noted that in the optimistic approach,the predefined time interval for completion of rounds, i.e., sending block messages to the parent domains, isconsidered to be very small to avoid aborting a large number of transactions in case of aborting cross-domaintransactions. Correctness.
We now briefly show the safety and liveness of the optimistic approach.
Lemma 4.5 ( Agreement ) If node r commits request m with sequence number h , no other non-faulty nodecommits request m ( m = m ) with the same sequence number h . Proof:
Assuming the internal consensus protocols guarantee agreement, the agreement property of thecross-domain protocol is ensured. In addition, if the transaction is not committed in an involved domain,the LCA domain detects it resulting in aborting the transaction.
Lemma 4.6 ( Validity ) If a non-faulty node r commits m , then m must have been proposed by some node π . Proof:
Validity is guaranteed in the same way as cross-domain consensus (lemma 4.2).
Lemma 4.7 ( Consistency ) For any two committed requests m and m and any two nodes r and r suchthat r ∈ d i , r ∈ d j , and { d i , d j } ∈ D m ∩ D m , if m is committed before m in r , then m is committedbefore m in r . Proof:
Upon receiving a cross-domain transaction, each intermediate and eventually the LCA domain(s)of both m and m checks consistency and resolves any inconsistencies by aborting either m or m . Theaborting strategy is deterministic resulting in aborting the same transaction independent of the domain thatchecks. While transactions might be initially optimistically committed in an inconsistent order, eventuallyinconsistency will be resolved, i.e., the protocol guarantees eventual consistency. Property 4.8 ( Termination ) A request m issued by a correct client eventually completes.The liveness of the algorithm is guaranteed in periods of synchrony based on the assumption that both LCAand involved (participants) domains ensure liveness for all transactions. Furthermore, if a node does notreceive commit (or abort ) message from the LCA domain for some cross-domain request and its timer expires,as discussed earlier, it sends a commit-query message to the parent domain, resulting in sending a commit-query message to the nodes of the involved domain(s) that have not committed the message. If the request is notcommitted in some predefined number of rounds by all involved domains it is considered to be aborted. Edge devices (i.e., leaf-level nodes) over wide-area networks might be mobile, e.g., smart cars or smartphones.The mobility of edge devices is either permanent or transient. If the mobility is permanent, the node will beauthenticated by the edge servers of the new domain and is able to initiate transactions. If consensus at theleaf level is needed, the node also obtains the blockchain ledger of the new domain and participates in theconsensus routine. However, if an edge device temporary moves from its leaf domain ( local ) to another leafdomain ( remote ), reaching consensus on transactions that are initiated by the mobile device is challenging.In fact, nodes of the remote domain do not access to the history of the mobile node, hence, are not able toprocess its requests. Moreover, any communication across domains is costly.12EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking System
Algorithm 2
Delay-Tolerant Mobile Consensus init(): i := node_id d l := local domain d r := remote domain upon receiving valid request m from a remote node n and i is π ( d r ) send h HISTORY-QUERY , m, δ m i σπ ( dr ) to d l upon receiving valid h HISTORY-QUERY , m, δ m i σπ ( dr ) and i is π ( d l ) if lock ( n ) = false then π ( d l ). GenerateHistory ( n, d l , d r ) else . lock ( n ) is true and remote ( n ) = d r π ( d l ). GetHistory ( n, d l , d r ) π ( d l ). GenerateHistory ( n, d l , d r ) end if function GenerateHistory (node n , domain d , domain d ) generate (abstract) history H ( n ) establish consensus on history H ( n ) among nodes in d lock ( n ) = true, remote ( n ) = d send h HISTORY , H ( n ) , δ h , δ m i σ to d (signed by π ( d ) or 2 f + 1 nodes of d ) end function function GetHistory (node n , domain d , domain d ) send h HISTORY-QUERY , m, δ m i σπ ( d ) to π ( d ) π ( d ). GeneratHistory ( n, d , d ) upon receiving valid h HISTORY , H ( n ) , δ h , δ m i σ message from π ( d ) lock ( n ) = false establish consensus on transactions of history message among nodes in d append the transactions and commit message(s) to the ledger end function Saguaro can deal with transactions initiated by mobile devices in the same way as cross-domain transactions.Once a mobile device sends a request to a remote edge (either leaf or height −
1) domain, the remote domain,similar to cross-domain transactions, sends the request to the local edge domain and then using either thecoordinator-based or the optimistic approach, consensus among both local and remote domains is established.This, however, requires several rounds of communications across domains for every single request. A moreefficient way to address consensus with mobile devices is to provide access to the history of the mobile nodein one round of communication across domains.The normal case operation of delay-tolerant mobile consensus is presented in Algorithm 2. As indicated inlines 1 − d l and d r are the local and remote edge domains.Upon receiving a valid request m from an unauthorized edge device, as shown in lines 5 −
6, the primary ofthe remote domain, sends a signed history-query message including the request m and its digest δ m to thelocal domain to obtain the history of the node. Each domain maintains a lock bit for each edge device tokeep track of its mobility. When an edge device initiates a transaction in a remote domain, the lock is setto true representing that the history of the edge device in the local domain is out-of-date. The domainalso defines a variable remote for each edge device to maintain the id of the remote domain that has themost recent transaction records of the node. Once the primary node of the local domain receives a valid history-query message for its edge node n , as presented in lines 7 −
13, if lock ( n ) is false (i.e., the history of node n in the local domain is complete and up-to-date), the GenerateHistory function is called. As shown inlines 14 −
19, the primary node of the local domain, π ( d l ), first generates an abstract history of mobile node n . The abstract history is application-dependent and includes the information that is needed to processtransactions. For example, in the micropayment application, the abstract history might include only thebalance of node or in the ridesharing application, the abstract history might include only the work hours ofthe driver. The primary then initiates consensus among nods of the local domain on the generated historyby sending a message including both history-query message received from the remote domain as well as thegenerated abstract history H ( n ). Once consensus is achieved, the primary sends a signed history messageincluding the generated history H ( n ), the digest δ h of the corresponding history-query message, and the digest δ m of request m to the remote domain. Nodes in the local domain also set lock ( n ) to be true and remote ( n )to d r . 13EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking System L a t e n c y [ m s ] (a) 0% Cross-Domain L a t e n c y [ m s ] Opt-10% F AHLSharPer CoordinatorOpt-90% C Opt-50% COpt-10% C (b) 20% Cross-Domain L a t e n c y [ m s ] Opt-10% F AHLSharPer CoordinatorOpt-90% C Opt-50% COpt-10% C (c) 80% Cross-Domain L a t e n c y [ m s ] Opt-10% F AHLSharPer CoordinatorOpt-90% C Opt-50% COpt-10% C (d) 100% Cross-Domain
Figure 5: Performance with Different Percentages of Cross-Domain Transactions in Networks with Crash-Only NodesIf lock ( n ) is true and remote ( n ) = d r , i.e., some other remote domain d r has the most recent transactionrecords, the local domain gets the recent transactions from d r by calling the GetHistory function, andthen calls the
GenerateHistory function to update its history and send an abstract history of n to theremote domain d r . This situation happens when an edge device moves to a remote domain d r , initiatessome transactions and then moves to another remote domain d r . The GetHistory function, as shown inlines 20 −
27, sends a history-query to remote ( n ) to obtain the most recent transaction records. Upon receivingthe history-query message, the remote domain generates a complete history of the transactions that node n has been involved in them. Note that, in contrast to the local domain, the history of the remote domainis complete because these transactions need to be appended to the blockchain ledger of the local domain.Once the history message is received, the local domain appends transactions to its ledger and set lock ( n ) tobe false. If the mobile node returns to its local domain and initiates a transaction, the local domain calls GetHistory , updates the ledger and then processes the transaction locally.
Correctness.
The correctness of delay-tolerant mobile consensus protocol is mainly ensured based on thecorrectness of internal consensus protocols in both local and remote domains. Since edge devices are mobileand might move from a leaf domain to another domain, we need to ensure safety in unlikely situations whereconsensus at the leaf-level is required, i.e., the edge domain is the leaf domain. First, mobile devices (in caseof transient mobility) do not participate in the consensus protocol of remote domains, hence, they can notviolate the safety condition of remote domains. Furthermore, nodes in a leaf domain can establish consensusonly if the safety condition holds, e.g., excluding mobile nodes, more than two-thirds of the existing nodesare non-malicious. Assuming the internal consensus protocols are safe and live, we just need to show thatcommunications across domains do not violate safety or liveness. Safety is guaranteed because to send a history message consensus among nodes of a domain is needed and history messages are signed by a crash-onlyprimary or 2 f + 1 Byzantine nodes.To guarantee liveness, if node r of a domain does not receive a history message after sending a history-query message and its timer expires, the node sends the history-query message to all nodes of the other domain. Ifthe message has already been processed, the nodes simply re-send the corresponding response. Nodes alsolog the query messages to detect denial-of-service attacks initiated by malicious nodes. If the query messageis received from a majority of a domain (i.e., f + 1 crash-only or 2 f + 1 Byzantine nodes), the primary willbe suspected to be faulty resulting in running the failure handling routine. If nodes of domain d receive history-query from domain d , however, the primary of d does not initiate consensus on history message, nodesof d suspect that their primary is faulty. Similarly, upon receiving history messages, nodes of domain d waitfor the primary node of d to initiate consensus. Otherwise, the primary will be suspected to be faulty. In this section, we conduct several experiments to evaluate Saguaro. We have implemented a micropaymentapplication (as a representative and the most demanding application) using Saguaro and emulated a four-level5G network (following the structure in Figure 1) where the leaf domains do not maintain the blockchainledger (i.e., height − L a t e n c y [ m s ] (a) 0% Cross-Domain L a t e n c y [ m s ] Opt-10% F AHLSharPer CoordinatorOpt-90% C Opt-50% COpt-10% C (b) 20% Cross-Domain L a t e n c y [ m s ] Opt-10% F AHLSharPer CoordinatorOpt-90% C Opt-50% COpt-10% C (c) 80% Cross-Domain L a t e n c y [ m s ] Opt-10% F AHLSharPer CoordinatorOpt-90% C Opt-50% COpt-10% C (d) 100% cross-Domain
Figure 6: Performance with Different Percentages of Cross-Domain Transactions in Networks with ByzantineNodesThe experiments were conducted on the Amazon EC2 platform on multiple VM instances. Each VM isc4.2xlarge instance with 8 vCPUs and 15GB RAM, Intel Xeon E5-2666 v3 processor clocked at 3.50 GHz.When reporting throughput measurements, we use an increasing number of requests until the end-to-endthroughput is saturated, and state the throughput ( x axis) and latency ( y axis) just below saturation. In the first set of experiments, we measure the performance of Saguaro in workloads with different percent-ages of cross-domain transactions. We implemented the coordinator-based and optimistic approaches andcompared them with SharPer [5] and AHL [19]. Due to the emphasis of the experiments, we only implemen-ted the cross-shard consensus protocol of AHL where a reference committee uses 2PC to order transactions(without using trusted hardware). The internal transactions of all approaches are processed in the same way.We consider different workloads with different percentages of cross-domain transactions (i.e., 0%, 20%, 80%,and 100%) where two randomly chosen domains (sender and receiver) are involved in each cross-domaintransaction. The optimistic approach, as expected, works best when all cross-domain transactions are com-mitted by all involved domains, hence, for a fair comparison, we also consider the situation where 10% ofcross-domain transactions are not committed (aborted) by some involved domain. Furthermore, as discussedearlier, a committed cross-domain transaction might be aborted due to inconsistency, i.e., two concurrentcross-domain transactions have been appended to the ledger of two domains in a different order. Abortingsuch transactions in the optimistic approach results in aborting all their data-dependent transactions. Weconsider three workloads with different degrees of contention between transactions of each domain, i.e., 10%(the default value for all workloads), 50%, and 90% read-write conflicts, to measure the effects of contentionon the performance of the protocol. Figures 5 and 6 demonstrate the results with crash and Byzantine nodes.When all nodes are crash-only and all transactions are internal, as shown in Figure 5(a), Saguaro is ableto process more than 31000 transaction with less than 100 ms latency before the end-to-end throughput issaturated. Since, each domain process its transactions independently, the throughput of the entire systemwill increase linearly by increasing the number of domains. Adding 20% cross-domain transactions, theoptimistic approach with 10% contention shows the best performance by processing 22800 transactions with80 ms latency. This is expected because the optimistic approach, does not require any communicationacross domains, and transactions are optimistically committed. In this scenario, only 0 .
03% of transactionswere appended to the ledgers in an inconsistent order, hence, increasing the percentage of contention inthe workload to 50% and 90% (Opt-50% C and Opt-90% C graphs) does not significantly impact theperformance of the optimistic protocol. Similarly, since only 20% of transactions are cross-domain, failureof 10% of cross-domain transactions (2% of all transactions) with 10% contention in the workload does notsignificantly reduce throughput (Opt-10% F graph). The coordinator-based approach also processes 21150transactions with 95 ms latency which is 23% more than AHL (17300 transactions with the same latency).Increasing the percentage of cross-shard transactions to 80% and 100% results in a larger performance gapbetween the coordinator-based approach and existing approaches (SharPer and AHL). This is expectedbecause in AHL, the single coordinator becomes overloaded by cross-domain transactions and in SharPerconsensus across domains becomes a bottleneck. However, Saguaro, by relying on multiple coordinator15EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking System L a t e n c y [ m s ] (a) Crash-Only Nodes L a t e n c y [ m s ] (b) Byzantine Nodes Figure 7: Performance with Mobile Devicesdomain, is still able to process transactions efficiently, e.g., in the workload with 100% cross-domain trans-action, the coordinator-based approach processes 10400 transactions, which is 67% more than the AHL withthe same latency.In the optimistic approach, and with 80% cross-domain transactions, 1 .
35% of transactions are appendedto the ledgers in an inconsistent order, Hence, increasing the percentage of contention in the workload to50% and 90% results in 5% and 11% reduction in the throughput (respectively). With 100% cross-domaintransaction and with 50% and 90% contention in the workload, we measure 6% and 14% reduction inthe throughput. Similarly, the failure of 10% of cross-domain transactions decreases 24% and 29% of thethroughput with 80% and 100% cross-domain transactions respectively. Note that since both inconsistenciesand failures are detected by higher-level domains, the latency of processing transactions by edge domains isnot affected.In the presence of Byzantine nodes, as shown in Fig 6(b), Saguaro shows similar behavior, however, withlower throughput and higher latency. This is expected because Byzantine fault-tolerant protocols are ingeneral more expensive than crash fault-tolerant protocols.
In the second set of experiments, we measure the performance of the delay-tolerant mobile consensus protocolto process remotely initiated transactions. We consider four workloads with different percentages (i.e., 0,20, 80, and 100) of remotely initiated (called mobile ) transactions where a local and a remote domain areinvolved in each mobile transaction. Figure 7(a) and Figure 7(b) show the results with crash and Byzantinenodes.When nodes are crash-only and all transactions are local, as shown in Figure 7(a), Saguaro processes morethan 31000 transaction with less than 100 ms latency (same scenario as Figure 5(a)). Adding 20% mobiletransactions, Saguaro still processes 30000 transactions (only ∼
3% reduction) with 107 ms latency before theend-to-end throughput is saturated. Similarly, with 80% and 100% mobile transactions, Saguaro processes26000 (with 129 ms latency) and 23800 (with 144 ms latency) transactions respectively. This indeed showsthe effectiveness of Saguaro in handling mobile devices: when the percentage of mobile transactions increasesfrom 0% to 100%, Saguaro incurs only a 23% reduction in its throughput.With Byzantine nodes, as shown in Figure 7(b), Saguaro demonstrates similar behavior and is able to process15400 transactions even when all transactions are mobile (overall throughput of Saguaro reduces only 25%compared to when there is no mobile transaction).
In the third set of experiments, the impact of network distance on the performance of Saguaro is measured.We compare the setting where all nodes are placed in a single AWS region with the setting where domainsare distributed over seven different AWS regions, i.e., California ( CA ), Oregon ( OR ), Virginia ( VA ), Ohio( OH ), Tokyo ( TY ), Seoul ( SU ), and Hong Kong ( HK ) . In this scenario, each leaf and its correspondingheight − TY , HK , VA , and OH data-centers, the height − SU and OR and the root domain is in the CA region. We consider workloads with 90%-internal 10%-cross- L a t e n c y [ m s ] AHLSharPercoordinatoroptimistic (a) Close by Domains L a t e n c y [ m s ] AHLSharPercoordinatoroptimistic (b) Far Apart Domains
Figure 8: Scalability Over Wide Area (Crash-Only) L a t e n c y [ m s ] AHLSharPerCoordinatorOptimistic (a) Close by Domains L a t e n c y [ m s ] AHLSharPerCoordinatorOptimistic (b) Far Apart Domains
Figure 9: Scalability Over Wide Area (Byzantine)domain transactions (typical settings in partitioned databases [64]), where two randomly chosen domains areinvolved in each cross-domain transaction. Since the workload includes only 10% cross-domain transactions,we do not perform experiments on optimistic protocol with different percentages of contention as well asinjected failures (when the percentage of cross-domain transactions is low, as shown in Section 6.1, theseparameters do not impact the performance). The results are demonstrated in Figures 8 and 9 for crash-onlyand Byzantine nodes.As shown in Figure 8(a), when domains are close to each other, the optimistic approach processes 26000 trans-actions with 100 ms latency. Similarly, the coordinator-based approach, SharPer, and AHL process 23500,20200, and 18400 transactions with the same latency (respectively). Placing nodes in far apart domains, asshown in Figure 8(b), however, results in lower throughput and higher latency, e.g., the throughput of the op-timistic approach is decreased by 40% and its latency is increased by 61%. Interestingly, AHL demonstratesbetter performance than SharPer which is expected due to the flattened consensus protocol of SharPer thatneeds rounds of communication among domains over a wide area. Furthermore, increasing the gap betweenthe performance of the coordinator-based approach and AHL (single coordinator) from 23% to 47% higherperformance demonstrates the effectiveness of the coordinator-based approach over wide area networks. Inthe presence of Byzantine nodes, as shown in Figure 9, all protocols demonstrate similar behavior to theprevious case. L a t e n c y [ m s ] (a) Crash-Only Nodes L a t e n c y [ m s ] (b) Byzantine Nodes Figure 10: Performance with Mobile Devices Over Wide Area17EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking SystemWe then use the same settings to measure the impact of network distance on mobile transactions where edgenodes move one, two, and three hubs away from their local domain and initiate transactions. The workloadsinclude 90% internal and 10% mobile transactions. As shown in Figure 10(a), in the presence of crash-onlynode and when mobile nodes are one hub away, e.g., the local domain in Tokyo and the remote domain inHong Kong, the throughput is decreased only by 4% (20% higher latency). Similarly, for two hubs away andthree hubs away scenarios, the throughput reduces only 10% and 14% (in comparison to no mobile devices)which demonstrates the efficiency of the mobile consensus protocol. In the presence of Byzantine nodes, asshown in Figure 10(b), processing transactions of mobile nodes incurs even less overhead due to the cost ofByzantine consensus protocols which makes the overhead less notable. L a t e n c y [ m s ] AHLSharPercoordinatoroptimistic (a) | p | = 5 L a t e n c y [ m s ] AHLSharPercoordinatoroptimistic (b) | p | = 9 Figure 11: Increasing the Number of Crash-Only NodesFinally, we evaluate the performance of Saguaro with a different number of nodes (i.e., maximum possiblefailures ( f )). We consider two scenarios with f = 2 and f = 4, i.e., each crash-only domain includes 5 and9 nodes, and each Byzantine domain includes 7 and 13 nodes respectively. All nodes are placed within anAWS region and the workload includes 90%-internal 10%-cross-domain transactions. As shown in Figure 11and Figure 12, increasing the number of nodes, reduces the performance of all protocols (insignificantly),e.g., the end-to-end throughput of the coordinator-based protocol is reduced by 6% and 11% (with the samelatency) when the number of nodes within each domain increases from 3 to 5 and 7. This is expected becauseonce the number of nodes increased, achieving consensus requires larger quorums. L a t e n c y [ m s ] AHLSharPercoordinatoroptimistic (a) | p | = 7 L a t e n c y [ m s ] AHLSharPercoordinatoroptimistic (b) | p | = 13 Figure 12: Increasing the Number of Byzantine Nodes
Despite several years of intensive research, existing blockchain solutions do not adequately address theperformance and scalability requirement of wide area networks, which is characterized by possibly mobilenodes communicating over the wide-area transacting across domains. In general, the ordering and executionof transactions are the two main phases of processing transactions. Early blockchains, e.g., Tendermint [37]and Quorum [17] follow the sequential order-execute paradigm resulting in high latency. Hyperledger Fabric[7], as a permissioned blockchain, introduces the execute-order-validate (XOV) architecture and leveragesparallelism by executing the transactions of different applications simultaneously. Several recent permissionedblockchains, e.g., blockchain relational Database [48], ParBlockchain [4], Fast Fabric [27], XOX Fabric [26],18EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking SystemFabric++ [60], and FabricSharp [57] also, execute transactions in parallel and try to address the shortcomingsof Hyperledger Fabric, e.g., dealing with contentious workloads.Parallel ordering of transactions is addressed in blockchains using both off-chain, e.g., Lightning Networks[45] [53] and sidechains [8] [21] [38] [41] [68], and on-chain, sharding techniques. Off-chain techniques increasethe system throughput by moving a portion of the transactions off the chain. However, such techniques donot scale more than two levels, thus, they are not compatible with the hierarchical structure of wide areanetworks.Sharding is a proven technique to improve databases scalability [18] [55] [72]. Data sharding is commonlyused in globally distributed databases such as H-store [34], Calvin [65], Spanner [18], Scatter [25], Google’sMegastore [9], Amazon’s Dynamo [20], Facebook’s Tao [12], and E-store [64] where the nodes are assumedto be crash-only and a coordinator processes transactions across shards. To achieve better performance insharding protocols, consensus protocol and transaction management have been combined [46] [70] [73] (incontrast to to layering one on top of another). The sharding technique has been used in both permissionless(e.g., Elastico [44], OmniLedger [36] and Ethereum 2 [1]) and permissioned blockchains (e.g., AHL [19],Blockplane [49], and SharPer [5]). However, sharding has several shortcomings that make it inappropriateto be used in wide area networks. First, sharding approaches maintain data shards mainly on cloud servers(far from edge devices) and do not benefit from the hierarchical structure of wide area networks, resultingin increased latency. Second, sharding approaches either use a flattened approach, e.g., SharPer [5], or acoordinator-based approach, e.g., AHL [19] to process cross-shard transactions where in both approaches,as explained earlier, the far network distance either between the involved shards (in the flattened approach)or between the coordinator and involved shards (in the coordinator-based approach) results in high latency.Third, while many edge devices in wide area networks are mobile, mobility has not been addressed in shardingsolutions.GeoBFT [31], as another scalability solution, maintains the entire ledger on every node and orders eachtransaction within only a single cluster to reduce the latency of cross-cluster transactions. After orderingevery transaction, however, all clusters need to communicate with each other to enable other clusters toupdate their ledgers and execute transactions, resulting in high latency.The hierarchical permissioned blockchain model presented in [58] focuses only on the data abstraction acrossdifferent levels of the hierarchy and does not address cross-domain transactions, consensus, and the mobilityof nodes. Plasma [52] also uses hierarchical chains to improve transaction throughput of Ethereum blockchainusing a series of smart contracts to create hierarchical trees of sidechains, however, processing cross-domaintransactions and mobility of nodes have not been addressed in Plasma.Our work is also related to blockchain systems with DAG-structured ledgers, e.g., Iota [54], Vegvisir [35],Phantom [63], Spectre [62], and Caper [2]. In particular, in Caper, the DAG structure is the result ofthe simultaneous appending of local and global transactions to the ledger whereas in Saguaro, the DAGstructure is used to support cross-domain transactions in internal levels. Moreover, Caper neither followsthe hierarchical structure of wide area networks nor supports the mobility of nodes.Blockchain brings the capability of managing wide area networks data through its secure distributed ledgerand provide immutability, decentralization, and transparency, all of which promise to tackle security issues ofcurrent 5G networks [50]. A permissionless blockchain-based architecture for IoT has been introduced in [22]where each cluster of nodes, e.g., a smart home, maintains a blockchain ledger. Blockchain is also utilizedto build an authentication system for reliable authentication and information sharing among different edge-based IoT platforms [28]. In [24] [67], a trusted access control scheme for energy sharing and distributionis designed using blockchains. Vehicular network [42], resource management [56], and resource allocation[69] are some of the other applications that use blockchain to achieve performance and trustworthiness.Nonetheless, the challenges of maintaining hierarchical ledgers, processing cross-domain transactions, andestablishing consensus in presence of mobile devices are not addressed in these studies.Finally, in comparison to mobile databases [33] [10], while Saguaro has a similar architecture, the trustmodel (e.g., Byzantine nodes), the level of mobility (only edge vs every node), and how Saguaro establishesconsensus and tracks mobility are different.
In this paper, we proposed Saguaro, a hierarchical permissioned blockchain system designed specificallyfor wide area networks. Saguaro leverages the hierarchical structure of wide area networks to enable effi-19EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking Systemcient processing of cross-domain transactions using coordinator-based and optimistic consensus protocols.Moreover, Saguaro addresses the mobility of edge devices in wide area networks by introducing a delay-tolerant mobile consensus protocol. Eminent applications of Saguaro include cross-companies micropaymentand global regulation over the gig economy. The experimental results reveal the efficiency of Saguaro inprocessing cross-domain transactions and transactions initiated by mobile devices especially in wide areanetworks where the involved nodes are far apart.
References [1] The beacon chain ethereum 2.0 explainer you need to read first. https://ethos.dev/beacon-chain/.[2] M. J. Amiri, D. Agrawal, and A. E. Abbadi. Caper: a cross-application permissioned blockchain.
Proc.of the VLDB Endowment , 12(11):1385–1398, 2019.[3] M. J. Amiri, D. Agrawal, and A. E. Abbadi. On sharding permissioned blockchains. In
Int. Conf. onBlockchain , pages 282–285. IEEE, 2019.[4] M. J. Amiri, D. Agrawal, and A. E. Abbadi. Parblockchain: Leveraging transaction parallelism inpermissioned blockchain systems. In
Int. Conf. on Distributed Computing Systems (ICDCS) , pages1337–1347. IEEE, 2019.[5] M. J. Amiri, D. Agrawal, and A. E. Abbadi. Sharper: Sharding permissioned blockchains over networkclusters. arXiv preprint arXiv:1910.00765 , 2019.[6] M. J. Amiri, J. Duguépéroux, T. Allard, D. Agrawal, and A. El Abbadi. Separ: A privacy-preservingblockchain-based system for regulating multi-platform crowdworking environments. In
Proceedings ofthe Web Conference (WWW) , 2021.[7] E. Androulaki, A. Barger, V. Bortnikov, C. Cachin, et al. Hyperledger fabric: a distributed operatingsystem for permissioned blockchains. In
European Conf. on Computer Systems (EuroSys) , page 30.ACM, 2018.[8] A. Back, M. Corallo, L. Dashjr, M. Friedenbach, G. Maxwell, A. Miller, A. Poelstra, J. Timón, andP. Wuille. Enabling blockchain innovations with pegged sidechains. , 2014.[9] J. Baker, C. Bond, J. C. Corbett, J. Furman, A. Khorlin, J. Larson, J.-M. Leon, Y. Li, A. Lloyd, andV. Yushprakh. Megastore: Providing scalable, highly available storage for interactive services. In
Conf.on Innovative Data Systems Research (CIDR) , 2011.[10] D. Barbará. Mobile computing and databases-a survey.
IEEE transactions on Knowledge and DataEngineering , 11(1):108–117, 1999.[11] G. Bracha and S. Toueg. Asynchronous consensus and broadcast protocols.
Journal of the ACM(JACM) , 32(4):824–840, 1985.[12] N. Bronson, Z. Amsden, G. Cabrera, P. Chakka, P. Dimov, H. Ding, J. Ferris, A. Giardullo, S. Kulkarni,H. Li, et al. Tao: Facebook’s distributed data store for the social graph. In
Annual Technical Conf.(ATC) , pages 49–60. USENIX Association, 2013.[13] C. Cachin, R. Guerraoui, and L. Rodrigues.
Introduction to reliable and secure distributed programming .Springer Science & Business Media, 2011.[14] C. Cachin, K. Kursawe, and V. Shoup. Random oracles in constantinople: Practical asynchronousbyzantine agreement using cryptography.
Journal of Cryptology , 18(3):219–246, 2005.[15] C. Cachin and M. Vukolić. Blockchain consensus protocols in the wild. In
Int. Symposium on DistributedComputing (DISC) , pages 1–16, 2017.[16] M. Castro, B. Liskov, et al. Practical byzantine fault tolerance. In
Symposium on Operating systemsdesign and implementation (OSDI) , volume 99, pages 173–186. USENIX Association, 1999.[17] J. M. Chase. Quorum white paper, 2016.[18] J. C. Corbett, J. Dean, M. Epstein, A. Fikes, et al. Spanner: Google’s globally distributed database.
Transactions on Computer Systems (TOCS) , 31(3):8, 2013.[19] H. Dang, T. T. A. Dinh, D. Loghin, E.-C. Chang, Q. Lin, and B. C. Ooi. Towards scaling blockchainsystems via sharding. In
SIGMOD Int. Conf. on Management of Data . ACM, 2019.20EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking System[20] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian,P. Vosshall, and W. Vogels. Dynamo: amazon’s highly available key-value store. In
Operating SystemsReview (OSR) , volume 41, pages 205–220. ACM SIGOPS, 2007.[21] J. Dilley, A. Poelstra, J. Wilkins, M. Piekarska, B. Gorlick, and M. Friedenbach. Strong federations:An interoperable blockchain solution to centralized third-party risks. arXiv preprint arXiv:1612.05491 ,2016.[22] A. Dorri, S. S. Kanhere, and R. Jurdak. Blockchain in internet of things: challenges and solutions. arXiv preprint arXiv:1608.05187 , 2016.[23] M. J. Fischer, N. A. Lynch, and M. S. Paterson. Impossibility of distributed consensus with one faultyprocess.
Journal of the ACM (JACM) , 32(2):374–382, 1985.[24] K. Gai, Y. Wu, L. Zhu, L. Xu, and Y. Zhang. Permissioned blockchain and edge computing empoweredprivacy-preserving smart grid networks.
IEEE Internet of Things Journal , 6(5):7992–8004, 2019.[25] L. Glendenning, I. Beschastnikh, A. Krishnamurthy, and T. Anderson. Scalable consistency in scatter.In
Symposium on Operating Systems Principles (SOSP) , pages 15–28. ACM, 2011.[26] C. Gorenflo, L. Golab, and S. Keshav. Xox fabric: A hybrid approach to transaction execution. In
Int.Conf. on Blockchain and Cryptocurrency (ICBC) , pages 1–9. IEEE, 2020.[27] C. Gorenflo, S. Lee, L. Golab, and S. Keshav. Fastfabric: Scaling hyperledger fabric to 20,000 trans-actions per second. In
Int. Conf. on Blockchain and Cryptocurrency (ICBC) , pages 455–463. IEEE,2019.[28] S. Guo, X. Hu, S. Guo, X. Qiu, and F. Qi. Blockchain meets edge computing: A distributed and trustedauthentication system.
Transactions on Industrial Informatics , 16(3):1972–1983, 2019.[29] Y. Guo and C. Liang. Blockchain application and outlook in the banking industry.
Financial Innovation ,2(1):24, 2016.[30] S. Gupta, J. Hellings, and M. Sadoghi. Rcc: Resilient concurrent consensus for high-throughput securetransaction processing. In
Int. Conf. on Data Engineering (ICDE) . IEEE, 2021.[31] S. Gupta, S. Rahnama, J. Hellings, and M. Sadoghi. Resilientdb: Global scale resilient blockchainfabric.
Proceedings of the VLDB Endowment , 13(6):868–883, 2020.[32] S. Gupta, S. Rahnama, and M. Sadoghi. Permissioned blockchain through the looking glass: Architec-tural and implementation lessons learned. In
Int. Conf. on Distributed Computing Systems (ICDCS) .IEEE, 2020.[33] T. Imielinski and B. Badrinath. Querying in highly mobile distributed environments. In
VLDB ,volume 92, pages 41–52, 1992.[34] R. Kallman, H. Kimura, J. Natkins, A. Pavlo, A. Rasin, S. Zdonik, E. P. Jones, S. Madden, M. Stone-braker, Y. Zhang, et al. H-store: a high-performance, distributed main memory transaction processingsystem.
Proc. of the VLDB Endowment , 1(2):1496–1499, 2008.[35] K. Karlsson, W. Jiang, S. Wicker, D. Adams, E. Ma, R. van Renesse, and H. Weatherspoon. Vegvisir: Apartition-tolerant blockchain for the internet-of-things. In
Int. Conf. on Distributed Computing Systems(ICDCS) , pages 1150–1158. IEEE, 2018.[36] E. Kokoris-Kogias, P. Jovanovic, L. Gasser, N. Gailly, E. Syta, and B. Ford. Omniledger: A secure,scale-out, decentralized ledger via sharding. In
Symposium on Security and Privacy (SP) , pages 583–598.IEEE, 2018.[37] J. Kwon. Tendermint: Consensus without mining.
Draft v. 0.6, fall , 2014.[38] J. Kwon and E. Buchman. Cosmos: A network of distributed ledgers.
URL https://cosmos. net-work/whitepaper , 2016.[39] L. Lamport. Time, clocks, and the ordering of events in a distributed system.
Communications of theACM , 21(7):558–565, 1978.[40] L. Lamport. Paxos made simple.
ACM Sigact News , 32(4):18–25, 2001.[41] S. D. Lerner. Rootstock: Bitcoin powered smart contracts, 2015.[42] M. Li, L. Zhu, and X. Lin. Efficient and privacy-preserving carpooling using blockchain-assisted vehicularfog computing.
Internet of Things Journal , 6(3):4573–4584, 2018.21EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking System[43] D. Loghin, S. Cai, G. Chen, T. T. A. Dinh, F. Fan, Q. Lin, J. Ng, B. C. Ooi, X. Sun, Q.-T. Ta, et al.The disruptions of 5g on data-driven technologies and applications.
IEEE Transactions on Knowledgeand Data Engineering , 32(6):1179–1198, 2020.[44] L. Luu, V. Narayanan, C. Zheng, K. Baweja, S. Gilbert, and P. Saxena. A secure sharding protocol foropen blockchains. In
SIGSAC Conf. on Computer and Communications Security (CCS) , pages 17–30.ACM, 2016.[45] A. Miller, I. Bentov, S. Bakshi, R. Kumaresan, and P. McCorry. Sprites and state channels: Paymentnetworks that go faster than lightning. In
Int. Conf. on Financial Cryptography and Data Security(FC) , pages 508–526. Springer, 2019.[46] S. Mu, L. Nelson, W. Lloyd, and J. Li. Consolidating concurrency control and consensus for commitsunder conflicts. In
USENIX Symposium on Operating Systems Design and Implementation (OSDI) ,pages 517–532, 2016.[47] S. Nakamoto. Bitcoin: A peer-to-peer electronic cash system. 2008.[48] S. Nathan, C. Govindarajan, A. Saraf, M. Sethi, and P. Jayachandran. Blockchain meets database:design and implementation of a blockchain relational database.
Proceedings of the VLDB Endowment ,12(11):1539–1552, 2019.[49] F. Nawab and M. Sadoghi. Blockplane: A global-scale byzantizing middleware. In , pages 124–135. IEEE, 2019.[50] D. C. Nguyen, P. N. Pathirana, M. Ding, and A. Seneviratne. Blockchain for 5g and beyond networks:A state of the art survey.
Journal of Network and Computer Applications , page 102693, 2020.[51] K. Peterson, R. Deeduvanu, P. Kanjamala, and K. Boles. A blockchain-based approach to healthinformation exchange networks. In
Proc. NIST Workshop Blockchain Healthcare , volume 1, pages 1–10,2016.[52] J. Poon and V. Buterin. Plasma: Scalable autonomous smart contracts.
White paper , 2017.[53] J. Poon and T. Dryja. The bitcoin lightning network: Scalable off-chain instant payments. https://lightning.network/lightning-network-paper.pdf , 2016.[54] S. Popov. The tangle.
URL https://iota.org/IOTA Whitepaper.pdf , 2018.[55] G. Prasaad, A. Cheung, and D. Suciu. Handling highly contended oltp workloads using fast dynamicpartitioning. In
SIGMOD Int. Conf. on Management of Data , pages 527–542. ACM, 2020.[56] G. Qiao, S. Leng, H. Chai, A. Asadi, and Y. Zhang. Blockchain empowered resource trading in mobileedge computing and networks. In
International Conference on Communications (ICC) , pages 1–6.IEEE, 2019.[57] P. Ruan, D. Loghin, Q.-T. Ta, M. Zhang, G. Chen, and B. C. Ooi. A transactional perspective onexecute-order-validate blockchains. In
SIGMOD Int. Conf. on Management of Data , pages 543–557.ACM, 2020.[58] S. Sahoo, A. M. Fajge, R. Halder, and A. Cortesi. A hierarchical and abstraction-based blockchainmodel.
Applied Sciences , 9(11):2343, 2019.[59] T. Salman, M. Zolanvari, A. Erbad, R. Jain, and M. Samaka. Security services using blockchains: Astate of the art survey.
IEEE Communications Surveys & Tutorials , 21(1):858–880, 2018.[60] A. Sharma, F. M. Schuhknecht, D. Agrawal, and J. Dittrich. Blurring the lines between blockchainsand database systems: the case of hyperledger fabric. In
SIGMOD Int. Conf. on Management of Data ,pages 105–122. ACM, 2019.[61] V. Shoup. Practical threshold signatures. In
International Conference on the Theory and Applicationsof Cryptographic Techniques , pages 207–220. Springer, 2000.[62] Y. Sompolinsky, Y. Lewenberg, and A. Zohar. Spectre: A fast and scalable cryptocurrency protocol.
IACR Cryptology ePrint Archive , 2016:1159, 2016.[63] Y. Sompolinsky and A. Zohar. Phantom: A scalable blockdag protocol., 2018.[64] R. Taft, E. Mansour, M. Serafini, J. Duggan, A. J. Elmore, A. Aboulnaga, A. Pavlo, and M. Stonebraker.E-store: Fine-grained elastic partitioning for distributed transaction processing systems.
Proc. of theVLDB Endowment , 8(3):245–256, 2014. 22EPAR: A Privacy-Preserving Blockchain-based Multi-Platform Crowdworking System[65] A. Thomson, T. Diamond, S.-C. Weng, K. Ren, P. Shao, and D. J. Abadi. Calvin: fast distributedtransactions for partitioned database systems. In
SIGMOD Int. Conf. on Management of Data , pages1–12. ACM, 2012.[66] F. Tian. A supply chain traceability system for food safety based on haccp, blockchain & internet ofthings. In
International conference on service systems and service management (ICSSSM) , pages 1–6.IEEE, 2017.[67] J. Wang, L. Wu, K.-K. R. Choo, and D. He. Blockchain-based anonymous authentication with key man-agement for smart grid edge computing infrastructure.
IEEE Transactions on Industrial Informatics ,16(3):1984–1992, 2019.[68] G. Wood. Polkadot: Vision for a heterogeneous multi-chain framework.
White Paper , 2016.[69] C. Xia, H. Chen, X. Liu, J. Wu, and L. Chen. Etra: Efficient three-stage resource allocation auctionfor mobile blockchain in edge computing. In , pages 701–705. IEEE, 2018.[70] X. Yan, L. Yang, H. Zhang, X. C. Lin, B. Wong, K. Salem, and T. Brecht. Carousel: Low-latencytransaction processing for globally-distributed data. In
Proceedings of the 2018 International Conferenceon Management of Data , pages 231–243, 2018.[71] M. Yin, D. Malkhi, M. K. Reiter, G. G. Gueta, and I. Abraham. Hotstuff: Bft consensus with linearityand responsiveness. In
Symposium on Principles of Distributed Computing (PODC) , pages 347–356.ACM, 2019.[72] E. Zamanian, J. Shun, C. Binnig, and T. Kraska. Chiller: Contention-centric transaction execution anddata partitioning for modern networks. In
SIGMOD Int. Conf. on Management of Data , pages 511–526.ACM, 2020.[73] I. Zhang, N. K. Sharma, A. Szekeres, A. Krishnamurthy, and D. R. Ports. Building consistent trans-actions with inconsistent replication.
ACM Transactions on Computer Systems (TOCS) , 35(4):1–37,2018.[74] W. Zhou, Q. Fei, A. Narayan, A. Haeberlen, B. T. Loo, and M. Sherr. Secure network provenance. In23rd symp. on operating systems principles (SOSP)