CCompact Merkle Multiproofs
Lum [email protected] Arber [email protected]
Abstract — The compact Merkle multiproof is a new andsignificantly more memory-efficient way to generate and verifysparse Merkle multiproofs. A standard sparse Merkle multi-proof requires to store an index for every non-leaf hash in themultiproof. The compact Merkle multiproof on the other handrequires only k leaf indices, where k is the number of elementsused for creating a multiproof. This significantly reduces thesize of multiproofs, especially for larger Merkle trees. I. R
ELATED W ORK
We received a lot of useful feedback from the HackerNews community, after we put the first version of this paperwas put online. Even though we were not able to find similarwork to the compact Merkle multiproof in the literature, itturns out two people already worked on almost identicalproposals before. Luke Champine contributed to a repositorymore than a year ago with almost the same concept as theone we will propose in this paper [1]. The only differencebetween our approach and Champine’s approach, is that hisimplementation uses a right-to-left technique for building amultiproof, whereas our approach uses a bottom-up tech-nique to construct the multiproof. Besides Champine’s work,it is also worth noting Pieter Wuille’s contributions to thebitcoin protocol [2]. Wuille’s multiproof approach is differentfrom both our work and Champine’s work, as it does not relyon leaf indices for multiproof construction. The end resulthowever is just as compact, and because of that deservesmentioning. II. I
NTRODUCTION
In this paper we will introduce the compact Merkle mul-tiproof, a more efficient way to compute and transmit sparseMerkle multiproofs [3]. To understand how the compactMerkle multiproof works, we have to understand first howMerkle trees function, and what sparse Merkle multiproofsare. This is why in this brief introduction, we are goingto briefly explain both concepts, before continuing with thecompact Merkle multiproof algorithm.
A. Merkle trees
A Merkle tree is a binary tree in which all leaf nodes(i.e. the Merkle tree’s elements) are associated with a cryp-tographic hash, and all none-leaf nodes are associated witha cryptographic hash, that is formed from the hashes of itschild nodes (as shown in figure 1).The Merkle tree is a data structure that allows forbandwidth-efficient and secure verification of elements ina list. It is used to verify the presence of elements inand between computers, without having to send the whole
Root H , , , H , H T H T H , H T H T H , , , H , H T H T H , H T H T Fig. 1. Depiction of a Merkle tree. The leaf nodes, i.e. the elements of aMerkle tree, are written as ’ T i . The non-leaf nodes are written as H j . list of elements to another computer. Merkle trees havefound a variety of uses cases: They are used in peer-to-peersystems to verify if the integrity of data blocks [4], for batchsigning of time synchronisation requests [5], for transactionverification in blockchain systems [6], and more.To verify that an element is present in the Merkle tree,a series of hashes are provided. the series of hashes is alsoknown as a Merkle proof. By sequentially hashing an elementhash with the provided Merkle proof, one can recreate thethe Merkle root of the Merkle tree (as shown in figure 2). Ifan element is present in a list and the Merkle proof is correct,then the end result of the sequential hashing will be theMerkle root. The recipient of the Merkle proof thus alreadyhas to have a copy of the Merkle root, before verifying theintegrity of a Merkle proof. As an example, by periodicallystoring Merkle roots, a verifier will be able to prove thatsome data is still unaltered. Root H , , , H , H T H T H , H T H T H , , , H , H T H T H , H T H T Fig. 2. Depiction of a Merkle tree, with the Merkle proof (shown in orange)for a given element ( T ). B. Sparse Merkle Multiproofs
A sparse Merkle multiproof (not to be confused withsparse Merkle trees) is a more efficient Merkle proof, forwhen it is necessary to prove the presence of a multipleelements that are in the same Merkle tree [3]. Let’s takefigure 3 as an example to better understand what this means. a r X i v : . [ c s . CR ] F e b o prove that three different elements are present in aMerkle tree, we could compute three separate Merkle proofsand verify the presence of each element separately. In theprovided example, a node would need nine hashes in totalto verify the presence of three elements. Root H , , , H , H T H T H , H T H T H , , , H , H T H T H , H T H T Root H , , , H , H T H T H , H T H T H , , , H , H T H T H , H T H T Root H , , , H , H T H T H , H T H T H , , , H , H T H T H , H T H T Fig. 3. Three Merkle proofs for three different elements.
By using a sparse Merkle multiproof however, we can dropthe number of hashes significantly. When overlapping thethree the Merkle proofs from figure 3 (as shown in figure 4),we can see that many of the hashes can in fact be recreatedby previous hashes. Instead of using three separate Merkleproofs that consist of nine hashes in total, one can provethe presence of the three elements with only four hashes (asshown in figure 5. This simple trick is also known as a sparseMerkle multiproof.
Root H , , , H , H T H T H , H T H T H , , , H , H T H T H , H T H T Fig. 4. Three overlapped Merkle proofs.
Using sparse Merkle multiproofs over standard Merkleproofs can have enormous space savings in certain scenarios,as shown in figure 6. There is however one important prob-
Root H , , , H , H T H T H , H T H T H , , , H , H T H T H , H T H T Fig. 5. An illustration of a Merkle multiproof.Fig. 6. Table taken from Jim McDonald’s wonderful article "Understandingsparse Merkle multiproofs" [3]. Space saving for Merkle pollards and spareMerkle multiproofs over simple Merkle proofs. lem with current sparse Merkle multiproof implementationsthat we thought needs addressing, today’s implementationsrequire additional data besides the multiproof [3]. Today’ssparse Merkle multiproofs require to store the hash indicesfor every non-leaf node. In other words, for every hash ina multiproof, we need an index to figure out the order ofcomputations in order to reconstruct a given Merkle root.One could argue that the necessity for additional data defeatsthe purpose of using a sparse Merkle multiproof, or at leastsignificantly limits its potential. This precise issue is whatthe compact Merkle multiproof solves.III. T HE C OMPACT M ERKLE M ULTIPROOF
The compact Merkle multiproof is a special techniqueto generate and verify sparse Merkle multiproofs, withoutthe need for non-leaf index information. A standard sparseMerkle multiproof requires to store an index for every non-leaf hash in the multiproof, the compact Merkle multiproofon the other hand requires only k leaf indices (or in the caseof the Bloom tree [7], k Bloom filter chunks), where k isthe number of elements used for creating a multiproof. Thissignificantly reduces the size of multiproofs, especially forlarger Merkle trees.In the next subsections we will explain how to generateand verify a compact Merkle multiproof for a Merkle tree.It is important to note that the compact Merkle multiprooftechnique works with other kinds of Merkle trees as well,such as Bloom trees, sparse Merkle trees, sorted Merkletrees, etc. We are going to take figure 7 and 8 as referencesto better understand how compact Merkle multiproofs aregenerated and verified. A. Compact Merkle Multiproof for Merkle Trees1) Compact Merkle Multiproof Generation:
Every leafnode has an index from 0 to N , where N is the total numberof leaves in a Merkle tree. We first determine the index forevery element that takes part in the multiproof. In the case oot H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H A : B : B pruned : A new : 0 2 3[0 1] [2 3] [2 3][0 1] [2 3]0 1 A : B : B pruned : A new : 1 4 6[0 1] [4 5] [6 7][0 1] [4 5] [6 7]0 2 3 A : B : B pruned : A new : 0 1[0 1] [0 1][0 1]0 A : B : B pruned : A new : H H H H H H H H H H H H H H H H H H H M : M : M : M : Fig. 7. An illustration of the compact Merkle multiproof generation procedure. Orange boxes represent the hashes of the Merkle proof. Blue boxesrepresent the indices of A in every Merkle layer. In each iteration, we append hashes to M , until the tree root is reached. For a more detailed descriptionof the procedure, refer to subsection III-A.1. Root H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H A : B : B pruned : A new : 0 2 3[0 1] [2 3] [2 3][0 1] [2 3]0 1 A : B : B pruned : A new : 1 4 6[0 1] [4 5] [6 7][0 1] [4 5] [6 7]0 2 3 A : B : B pruned : A new : 0 1[0 1] [0 1][0 1]0 A : B : B pruned : A new : H H H H H H H H H H H M : M : M : Fig. 8. An illustration of the compact Merkle multiproof verification procedure. Orange boxes represent the hashes of the Merkle proof. Blue boxesrepresent the indices of A in every Merkle layer. In each iteration, we hash elements between the element hashes in E and M until no element is left in M . For a more detailed description of the procedure, refer to subsection III-A.2. f our example in figure 7, that would be indices [2 , , , .Let’s name this array A and let’s name the "Merkle layer" onwhich we operate as L (The Merkle Layer at the beginningare simply the leaf nodes of the tree). After determiningthese indices, we run the following steps recursively untiltermination:– For each of the indices in A , take the index of its imme-diate neighbor in layer L , and store the given elementindex and the neighboring index as a pair of indices(an "immediate neighbor" is the leaf index right nextto a target leaf index that shares the same parent). Inthe first iteration of our example in figure 7, we end upwith an array of the form [[2 , , [2 , , [8 , , [12 , .Let’s name this array B – Remove any duplicate from B . In the first iteration ofour example in figure 7, B would end up to be of theform [[2 , , [8 , , [12 , . In figure III-A.1 we referto this as B pruned .– Take the difference between the set of indices in B pruned and A and append the hash values for thegiven indices, for the given Merkle layer to the multi-rpoof M . In the first iteration of our example in figure7, we would end up with the indices [9 , , which arethe indices for H and H . Append H and H tothe multiproof M .– We take all the even numbers from B pruned , and dividethem by two. We assign the newly computed numbersto A . In the first iteration of our example in figure 7, A would end up to be of the form [1 , , .– Go one layer up the tree. Assign that layer to L . Eachlayer in the tree is indexed from 0 to N , where N isthe size of that layer.– Repeat the above steps with the newly assigned vari-ables A and L until you reach the root of the tree.The proof at the end must contain the indices of the elementsused for the multiproof, as well as the gathered hashes inside M .
2) Compact Merkle Multiproof Verification:
For a com-pact Merkle multiproof verification, we require the indicesof the elements used for a multiproof (Let’s name this array A ), the corresponding hashes for the elements used for amultiproof, as well as the hashes of the multiproof M .Array A in case of our example 8 would be [2 , , , . Wefirst need to sort the k element hashes in increasing orderaccording to A . Let’s name this sorted array E . To verify agenerated multiproof, we run the following steps recursivelyuntil termination:– For each of the indices in A , take the index of itsimmediate neighbor, and store the given element indexand the neighboring index as a pair of indices (an"immediate neighbor" is the leaf right next to a targetleaf that shares the same parent). In the first iterationof our example in figure 8, we would end up withan array of the form [[2 , , [2 , , [8 , , [12 , . Let’sname this array B .– B will always have the same size as E . After com- puting B , we check for duplicate index pairs inside it.If two pairs are identical, we hash the correspondingvalues (that have the same indices) inside E with oneanother. If an index pair has no duplicates, we hashthe corresponding value inside E with the first hashinside M . If a value inside M was used, we remove itfrom M . All the newly generated hashes are assignedto a new E that will be used for the next iteration.– We take all the even numbers from B pruned , and dividethem by two. We assign the newly computed numbersto A . In the first iteration of our example in figure 8, A would end up to be of the form [1 , , – Repeat the above steps until M has no elementsanymore.At the end of this procedure, E will have a single value,the Merkle root of the tree. If the final value is not equal tothe stored Merkle root, the verifier knows that the proof isinvalid. IV. C ONCLUSION
We showed a new way how to compute more memory-efficient Merkle multiproofs. The compact Merkle multiproofcan generate and verify sparse Merkle multiproofs, withoutthe need for non-leaf index information. A standard sparseMerkle multiproof requires to store an index for every non-leaf hash in the multiproof, the compact Merkle multiproofon the other hand requires only k leaf indices, where k is the number of elements used for creating a multiproof.This significantly reduces the size of multiproofs, especiallyfor larger Merkle trees. The compact Merkle multiprooftechnique can be applied to a various Merkle tree variants,such as the Bloom tree, sparse Merkle tree, etc.V. F UTURE W ORK
We have an implementation of the compact Merkle mul-tiproof for our Bloom tree package (which can be foundon the Bloom Lab’s github page). In future work, we aregoing to show how one can combine Bloom trees that usecompact Merkle multiproofs, with distributed Bloom filters[8] to create an ”interactive Boom proof”. We will show howthe interactive Bloom proof can be used to build a new kindof blockchain architecture, that requires one magnitude lessstorage, while still allowing nodes to independently verifytransaction validity. The efficiency of the compact Merklemultiproof procedure will play an integral part in this setup.R
EFERENCES[1] “add support for multi-range proofs (bae5b547) · Com-mits · NebulousLabs / merkletree · GitLab.” [On-line]. Available: https://gitlab.com/NebulousLabs/merkletree/-/commit/bae5b547495f9dddbd4ddeecfdcb00cb89d99d76[2] “Add CPartialMerkleTree · bitcoin/bitcoin@4bedfa9 ··