Density Evolution on a Class of Smeared Random Graphs: A Theoretical Framework for Fast MRI
DDensity Evolution on a Class of Smeared Random Graphs:A Theoretical Framework for Fast MRI
Kabir Chandrasekher, Orhan Ocal, and Kannan RamchandranDepartment of Electrical Engineering and Computer Sciences, University of California, Berkeley { kabirc, ocal, kannanr } @berkeley.edu Abstract
We introduce a new ensemble of random bipartite graphs, which we term the ‘smearing ensemble’, where each left node isconnected to some number of consecutive right nodes. Such graphs arise naturally in the recovery of sparse wavelet coefficientswhen signal acquisition is in the Fourier domain, such as in magnetic resonance imaging (MRI). Graphs from this ensembleexhibit small, structured cycles with high probability, rendering current techniques for determining iterative decoding thresholdsinapplicable. In this paper, we develop a theoretical platform to analyze and evaluate the effects of smearing-based structure.Despite the existence of these small cycles, we derive exact density evolution recurrences for iterative decoding on graphs withsmear-length two. Further, we give lower bounds on the performance of a much larger class from the smearing ensemble, andprovide numerical experiments showing tight agreement between empirical thresholds and those determined by our bounds.Finally, we describe a system architecture to recover sparse wavelet representations in the MRI setting, giving explicit thresholdson the minimum number of Fourier samples needing to be acquired for the -stage Haar wavelet setting. In particular, we showthat K -sparse -stage Haar wavelet coefficients of an n -dimensional signal can be recovered using . K Fourier domain samplesasymptotically using O ( K log K ) operations. We explain our problem through an intriguing balls-and-bins game. There are n distinct colors, d balls of each color and M bins.You know beforehand that only K (cid:28) n of the colors, which are selected uniformly at random from the n possible colors, will be‘active’, but you do not know which ones they are. You have to throw all the ( dn ) balls into the ( M ) bins. The rules of the gameare as follows:R1) For each color c , you choose a subset (possibly using a randomized strategy) B c ⊂ { , · · · , M − } of size d . Then, thesystem throws the balls of that color c into bins { b + b c : b ∈ B c } modulo M where b c is sampled uniformly at randomfrom { , · · · , M − } .R2) If a bin contains a single active ball, then all d balls having the same color as that ball can be removed.R3) The process continues iteratively until either (a) all active balls have been removed or (b) there is no bin having a singleactive ball.The goal of the game is to remove all active balls using the minimum number of bins. We focus on the regime in which ( n, K, M ) → ∞ , d = O (1) and ask the following questions:1. What is the optimal method of dispatching the d balls? (That is, what is the optimal strategy for designing the subsets inR1?)2. Given ( n, K, d ) , what is the minimum number of bins ( M ) necessary?While this is an intriguing game in its own right, more importantly, it has connections to the design of sparse-graph codes andpeeling decoding. Surprisingly, and more relevant here, it is also intimately related to the recovery of sparse wavelet representa-tions from Fourier domain samples (see Section 3).To illustrate, suppose that d = 3 , then the best known strategy is to throw each ball at a bin selected uniformly at random.It has been demonstrated that we need asymptotically M (cid:39) . K bins as K grows. This can be shown through densityevolution methods, introduced by Richardson and Urbanke in [1], which have proven powerful in analyzing the performance ofsparse-graph codes. Now suppose that d = 6 . The natural strategy is to again throw each ball at a bin selected uniformly atrandom. Surprisingly, this strategy is not optimal. To see this, we give a brief introduction to the smearing ensemble. Consider a g dimensional vector s = [ s , s , . . . , s g ] where (cid:80) gi =1 s i = d . The ensemble is such that g bins are selected at random, and for Note that the only effect of this is to randomly offset the bins for each color. a r X i v : . [ c s . I T ] M a y he i th bin, the immediately following s i − bins are deterministically selected ; this is what we term the smearing ensemble.Figure 1 shows an example illustration for s = [2 , , and we formally define the smearing ensemble in Section 2. ExaminingTable 1, one can see that many simple smearing strategies outperform the fully random ensemble. In this paper, we do not claimto design an optimal strategy for this game; rather, we provide a theoretical platform to analyze these structure-exploiting policies. bin: K active colorsd balls0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Figure 1: An example graph showing active balls and bins that can result from rule R1. Here K = 4 , M = 15 and d = 4 . Each color ispartitioned into g = 2 groups and the number of balls per group is . The game proceeds by following the rules R2-R3. We want to find theminimum number of bins necessary to recover all the active colors. Our balls-and-bins game is motivated by an extension of the recently proposed FFAST (Fast Fourier Aliasing-based SparseTransform) algorithm [2] to the case where sparsity is with respect to some wavelet basis. In this setting, the balls correspondto the wavelet coefficients and the bins correspond to samples. Wavelets are universally recognized to be an efficient sparserepresentation for the class of piecewise smooth signals having a relatively small number of discontinuities, a very good modelfor many real-world signals such as natural images [3]. In particular, we note that in MRI, images are observed to be sparsewith respect to appropriately chosen wavelet bases, and acquisition is in the Fourier domain [4]. The computational bottleneckin recovering these images has been observed to be the computation of multiple large Fourier transforms. Our extension of theFFAST algorithm targets this problem. The details of this extension can be found in Section 3.
Table 1: Thresholds (
M/K ) for d = 6 . Note that this table contains a strict subset of all possible strategies. Regime , , , , , , , , , , , , , , , , , , , , , M/K .
570 1 .
533 1 .
489 1 .
518 1 .
533 1 .
542 1 . Density evolution methods have proven powerful in analyzing the performance of sparse-graph codes and their extensions [1, 5].Unfortunately, these methods apply only for sparse random graphs that are locally tree-like. This is not the case for all ball-throwing strategies in the game we outlined above, e.g. the [2 , , scheme. Recently Donoho et. al. have introduced approximatemessage passing (AMP) techniques to extend the message passing paradigm to the case when the underlying factor graph isdense [6]. These techniques were rigorously analyzed by Bayati and Montanari in [7]. Although AMP has been successfullyapplied to many problem domains, e.g., [8, 9], it imposes a dense structure on the factor graph. Additionally, Kudekar et. al havebeen able to show the benefit of structure in convolutional LDPC codes through the spatial coupling effect [10, 11]. However, ifthe bipartite graph is sparse, but contains small, structured cycles, it may not be necessary to invoke such methods. The main results of this paper are the derivation of exact thresholds for random graphs with smear-length , and bounds forhigher smear-length which are empirically shown to be very tight. We additionally detail an application to fast recovery of sparsewavelet representations of signals when acquisition is in the Fourier domain. In particular, given an n -dimensional signal whichis K -sparse with respect to the -stage Haar wavelet, our analysis shows that . K Fourier domain samples are needed torecover the signal in O ( K log K ) time. For example, if s i = 2 (we henceforth refer to this as the smear-length), then bins b i and b i + 1 (modulo M ) are selected, where b i is uniformly selected on { , , . . . , M − } . See Fig. 1 for an illustration. That is, an n -dimensional signal with exactly K non-zero entries -smearing. InSection 2.2, we derive lower bounds for the probability of recovery in the case of arbitrary smearing. We outline the connectionbetween the ball coloring game and the recovery of sparse wavelet representations in Section 3. We conclude with Section 4 bysummarizing some interesting open problems and conjectures that have resulted from this work. In this section, we introduce a new random graph ensemble, termed the ‘smearing ensemble’, and show how to derive densityevolution recursions for graphs from this ensemble. We detail the derivation of density evolution for smearing with smear-length , and we give lower bounds for smear-length L . The density evolution for smear-length is relegated to Appendix B. We nowformally define the smearing ensemble: Definition 1.
Let G ( K, M, s ) denote the ‘smearing graph ensemble’ with K left nodes and M right nodes with connectivitycharacterized by s . Each left node selects g right nodes, where g is the length of the vector s . At the i th iteration ( ≤ i < g ),edges are put between the left node and right nodes { b i , b i +1 , . . . , b s i − } modulo M where b i is selected uniformly at randomfrom { , , . . . , M − } .Henceforth we are going to use the terms ‘left node’ and ‘ball’ interchangeably as well as ‘right node’ and ‘bin’ interchange-ably. We additionally refer to balls thrown to the same set of bins as a stream. Now, let λ := gK/M . If K balls are thrown intothe M bins randomly, the degree (that is the number of balls in) of each stream will be distributed like Poisson ( λ ) by the Poissonapproximation to the binomial distribution. For the sake of brevity, we omit a general introduction to density evolution methods,pointing the interested reader to [5]. We now carefully derive the density evolution recurrence for smearing ensembles with themaximum smearing length of 2. -smear Graphs ( s = [2 , , In our ball-coloring game, we noted that threshold for the setting [2 , , outperforms that for the setting [1 , , , , , . In thissection, we give exact analysis for these thresholds . First, we define our notation in Table 2. Table 2: Notation for density evolution with smear-length x t : Probability that a random ball is not removed at iteration tq t : Probability that none of the bins in a smeared pair is removed at iteration td t : Probability that all balls in the same stream as the reference ball are removed at time ts t : Probability that all balls in a stream which intersects, but does not fully overlap, with the reference streamare removed at time t Figure 2: Depth neighborhood of a ball in a graph with [2 , , smearing (see Definition 1). The bins and balls are grouped by their colorswith respect to distance from the root. Unlike as in conventional density evolution methods in the LDPC literature which are based on an edge perspective, wetake a node perspective here because dependencies between edges in the smearing setting complicate the analysis. We refer the Although thresholds can be derived for the other elements in the table, we give the [2 , , derivation for clarity. x t and q t are clear: x t = q t , (1) q t = 1 − d t (1 − (1 − s t ) ) . (2)To see these, we recall the dynamics of the peeling decoder: a ball is removed as soon as any of its neighbors are removed,whereas a bin is removed only if all balls contained in it are removed. Thus, x t only occurs when none of the pairs of smearedbins to which it is connected are removed. On the other hand, for a bin to be removed, all of its connected nodes must be removed.Thus, all of the balls in the same stream as the reference ball must be removed, which happens with probability d t . Additionally,at least one of the two streams that do not fully overlap with the reference ball must be removed, which happens with probability (1 − (1 − s t ) ) . Now, note that in Fig. 2, the recurring structures are those with probabilities s t and d t , so we are done uponcalculating these quantities.We first calculate d t . Let D denote the number of balls in this stream other than the reference ball we are looking at. Itfollows that D is distributed as Binomial (3( K − , /M ) , which can be approximated well for large K and M by Poisson( λ ),where λ = 3 K/M . It follows that: d t = ∞ (cid:88) i =0 P ( D = i )(1 − q t − ) i = ∞ (cid:88) i =0 e − λ λ i i ! (1 − q t − ) i = e − λq t − . (3)Now, we tackle s t . Note that intuitively, s t ≥ d t . This is because each ball in a stream tracked by s t gets the same independenthelp from bins as d t . However, there is additionally a shared bin between all these balls. This shared bin is able to aid in theremoval of the stream tracked by s t when exactly one ball is left in the stream, and the bin has no contributions from elsewhere.The incorporation of this help is the key ingredient in using the structure to help characterize the decoding thresholds. Letting D be the number of balls in this stream, we precisely characterize this as follows: s t = ∞ (cid:88) i =0 P ( D = i ) (cid:2) (1 − q t − ) i + is t − q t − (1 − q t − ) i − (cid:3) = e − λq t − + λs t − q t − e − λq t − . (4)Note that the first term in this recursion is exactly d t . The second term describes the help received from the shared bin. Inorder to properly characterize this term, it is necessary to introduce the notion of memory : the shared bin can help if it has allcontributions removed except for one by time t . Unlike when the neighborhood is tree-like and contains no cycles and branchesof the tree become independent [13], when cycles are introduced, dependence between branches is introduced. The introducedmemory captures exactly this dependence.We summarize the results of this section with the following lemma: Lemma 1.
Consider a random graph from the ensemble G ( K, M, [2 , , . Then, for M ≥ . K , recovery using the peelingdecoder will succeed with high probability.Proof. The threshold follows from the density evolution derived above. It is important to note that these recurrences werederived using only high probability cycles and to be precise, it is necessary to show that the actual fraction of unidentified ballsconcentrates around the average, x t . We need to show convergence of the neighborhood of a random node in the bipartite graphcreated by the independent streams without smearing to a tree. This follows directly from the arguments in [14] and we omit thedetails here.The analogous recurrences are derived exactly for the case of smear-length in the appendix and highlight the difficultyin extending the exact analysis to larger smear-lengths. In the next section, we derive simple, but effective, lower bounds for L -smearing. We do this by using the following principles of generalization, inspired by the derivation above: Generalizing to L -smearing
1) In a stage with L smearing, there will be L − steps of memory necessary in order to capture the smearing structure2) In a stage with L smearing, the number of recurring structures will be L
3) Shared bins can be used through the introduction of memory in the recursion4 .2 Lower Bound on L -smearing For clarity, we will consider the ensembles with s = [ L, L, L ] . Along with x t , q t , d t as described in Table 2, we define quantity s ( i ) t in Table 3. Table 3: Notation for density evolution with smear-length L s ( i ) t : Probability that all nodes in the streams which does not intersect with the reference stream in j bins where ≤ j ≤ i bins are removed at time t Figure 3: Depth neighborhood of a ball under L -smearing. The quantities d t and { s ( i ) t } L − i =1 are the recurrent structures, where s ( i ) t tracks thejoint probability that all the streams which intersect with the reference stream in { j } L − j = L − i bins are removed at time t . As described in the principles of generalization, there will be L recursions, and up to L − memory. The probability d t isunaffected as it depends only on the other stages. All of the s ( i ) t , however, can have up to i memory (corresponding to the numberof bins that do not overlap with the reference ball). Thus, we take an approach much like a first-order approximation of a Taylorseries, and allow each s ( i ) t to use only one step of memory. The following lemma characterizes the critical quantity q t in terms ofits component streams. Lemma 2.
In a stage with L -smearing, − q t = d t (cid:20) s ( L − t + L − (cid:88) i =2 s ( i − t s ( L − i ) t − L − (cid:88) i =1 s ( i ) t s ( L − t (cid:21) . Proof.
See Appendix A.1The following lemma then establishes monotonicity of − q t with respect to d t , s ( i ) t , which we may use to complete thebound. Lemma 3.
Let f ( d t , s (1) t , . . . s ( L − t ) = 1 − q t , then f ( d t , s (1) t , . . . s ( L − t ) is non-decreasing in ( d t , s (1) t , . . . , s ( L − t ) .Proof. See Appendix A.2.Given the above characterization of q t , it now remains to give lower bounds for d t , s (1) t , . . . , s ( L − t . We can bound these asfollows: d t = e − λq t − , (5) s ( i ) t ≥ e iλq t − + λq t − e − λq t − r ( i ) t − , (6)for i ∈ { , . . . , L − } , where r ( i ) t def = i (cid:88) j =1 j (cid:88) k =1 s ( k ) t − e − ( L − k − λq t − e − ( k − λq t − . (7)There is a simple way to think about the problem so that these bounds appear. Consider the bound on s ( i ) t . This tracks thejoint probability that all the streams which intersect the reference stream in L − j bins where ≤ j ≤ i are removed at time t . Inorder for all of these streams to be removed, there are two cases:5 . . . M/K − x ∞ Figure 4: We plot our lower bounds against empirical simulations with settings s = [1 , , L ] for various L . The y -axis here denotes theprobability that a random ball is removed when peeling stops (i.e., − x t as t → ∞ ). We note that this ensemble is used as it is of the mostrelevant practical interest to recovery of sparse wavelet representations (see Section 3).
1. Each stream was removed from another stage. This probability is tracked by the first term: e iλq t − .2. Exactly one ball remains among all the streams. This probability is tracked by the second term.We focus on the second case. Suppose that the remaining ball is in the stream which intersects the reference stream in j bins.This implies that it is also contained in L − j shared bins that do not intersect the reference stream. It can be removed by any ofthese bins, as long as it is the only contribution. This help is tracked by the summation in the second term. Corollary 1.
The lower bounds given in equations (5) and (6) imply a lower bound on x t , the probability that a random node isremoved at time t .Proof. This follows directly from Lemma 2 and Lemma 3.We now give numerical experiments corroborating that the bounds in Corollary 1 capture the actual thresholds well. We experi-mentally find the thresholds for full recovery by sweeping λ , and compare them to the thresholds implied by the bounds. Fig. 4shows these for filters with different lengths. In MRI, one acquires samples of the Fourier transform of an input signal of interest. MRI speed is directly related to the numberof samples acquired. An inverse transform is then used to recover the original signal. Mathematically, let x be an n -length signal,and X def = F n x be its Fourier transform, where F n is the Fourier matrix of size n × n . In MRI, the problem is to recover x from { X i : i ∈ I} where the set I denotes the sampling locations, and this set is a design parameter.We now present how the game of balls-and-bins and its analysis as described in Sections 1 and 2 relates to MRI. For ease ofillustration we confine ourselves to the noiseless setting and exact sparsity, but these assumptions can be relaxed. If the signal x is K -sparse, one can use the FFAST algorithm to recover x from O ( K ) samples with O ( K log K ) computations [2, 15]. However,the images of interest in MRI are generally not sparse, but they do have sparse wavelet representations [4]. That is, we canexpress x = W − n α , where W − n is an appropriate wavelet, and α is sparse. Under this signal model, the problem of recovering α can be transformed into the problem of decoding on an erasure channel using a sparse-graph code. In particular, the graph forthe code is drawn from a smearing ensemble with smearing length L a function of the length of the underlying wavelet filter.Furthermore, assume α is K -sparse and the length of x is of the form n = f f f where f , f and f are co-prime. For m ∈ Z that divides n , let D m,n be the regular downsampling matrix from length n to m , that is, D m,n = [ I m · · · I m ] with I m isrepeated n/m times. Let y f (cid:96) for (cid:96) ∈ { , , } be the inverse Fourier transform of the downsampled X , that is, y f (cid:96) def = F − f (cid:96) D f (cid:96) ,n X .Using the properties of Fourier Transform, it follows that y f (cid:96) = D f (cid:96) ,n x = D f (cid:96) ,n W − n α .Now, for simplicity, assume that W is a block transform with block size L (eg., for stage Haar wavelets L = 2 ), andthe support of α is chosen uniformly random over the subsets of size K . Using the relations between y f , y f and y f and α ,recovering α is equivalent to decoding on a random graph from G ( K, M = f + f + f , s = [ L, L, L ]) .We can actually ‘improve’ the induced graph if a factor of the signal length has L as a factor. Say that L divides f , it followsthat A f ,n W − n = [ W − f · · · W − f ] , where W − f is repeated n/f times. It can be verified that y (cid:48) f def = W f y f = A f ,n a , henceit aliases the wavelet coefficients regularly without smearing. The relation between y (cid:48) f , y f and y f and α then induces a graphfrom G ( K, M = f + f + f , s = [1 , L, L ]) , which gives raise to a better threshold.To complete the equivalence to decoding a sparse-graph code on an erasure channel, we need a mechanism to check if thereis a single component in a bin (a single color in a bin). This can be implemented by processing a shifted version of x (incurringan additional factor of of oversampling). We end this section with the following lemma.6 emma 4. Consider a signal α with ambient dimension n and sparsity K , and access to samples from F n W − n α . Then, thesubsampling scheme described above along with the peeling decoder is able to exactly recover the sparse signal α using . K samples and time complexity O ( K log K ) .Proof. The threshold . K follows from the density evolution derived in Section 2. The proof of complexity is given inAppendix C. We have introduced a new random graph ensemble, termed the ‘smearing ensemble’ and devoloped a framework for derivingdensity evolution recurrences for random graphs from this ensemble. Recalling our balls-and-bins game, our results show thatsome amount of smearing can lead to a better strategy than the full random case. A fascinating open question arises here: what isthe optimal ball-throwing strategy and what are the density evolution recurrences for such a strategy? In this paper, we have giventhe first steps in analyzing this problem rigorously. To do this, we have leveraged the existence of small, structured cycles andintroduced the notion of memory into our density evolution. We believe there to be a deep connection between the introductionof memory in our recurrences and the introduction of the ‘Onsager’ term in the update equations of AMP [16]. We additionallybelieve the gains seen in spatially coupled ensembles [11] are intimately related to the structural gains of the smearing ensemble.An extremely interesting open problem is to determine the nature of these connections. We have additionally shown the practicalconnection between the smearing ensemble and the recovery of a sparse wavelet representation of a signal whose samples aretaken in the Fourier domain.
References [1] T. J. Richardson and R. L. Urbanke, “The capacity of low-density parity-check codes under message-passing decoding,”
IEEE Trans. Inf. Theory , vol. 47, no. 2, pp. 599–618, Feb. 2001.[2] S. Pawar and K. Ramchandran, “Computing a k-sparse n-length discrete fourier transform using at most 4k samples andO(klogk) complexity,” in
IEEE Int. Symp. on Inf. Theory . IEEE, Jul. 2013, pp. 464–468.[3] M. Vetterli and J. Kovacevic, “Wavelets and subband coding,” 1995.[4] M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly, “Compressed sensing MRI,”
IEEE Signal Process. Mag. , vol. 25,no. 2, pp. 72–82, Mar. 2008.[5] T. Richardson and R. Urbanke,
Modern coding theory . Cambridge University Press, 2008.[6] D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algorithms for compressed sensing,”
Proc. Natl. Acad. Sci.USA , vol. 106, no. 45, pp. 18 914–18 919, Sep. 2009.[7] M. Bayati and A. Montanari, “The dynamics of message passing on dense graphs, with applications to compressed sensing,”
IEEE Trans. Inf. Theory , vol. 57, no. 2, pp. 764–785, Jan. 2011.[8] J. Tan, Y. Ma, and D. Baron, “Compressive imaging via approximate message passing with image denoising,”
IEEE Trans.Sig. Process. , vol. 63, no. 8, pp. 2085–2092, Mar. 2015.[9] A. Maleki, L. Anitori, Z. Yang, and R. G. Baraniuk, “Asymptotic analysis of complex LASSO via complex approximatemessage passing (CAMP),”
IEEE Trans. Inf. Theory , vol. 59, no. 7, pp. 4290–4308, Mar. 2013.[10] S. Kudekar, T. J. Richardson, and R. L. Urbanke, “Threshold saturation via spatial coupling: Why convolutional LDPCensembles perform so well over the BEC,”
IEEE Transactions on Information Theory , vol. 57, no. 2, pp. 803–834, Jan.2011.[11] S. Kudekar, T. Richardson, and R. L. Urbanke, “Spatially coupled ensembles universally achieve capacity under beliefpropagation,”
IEEE Transactions on Information Theory , vol. 59, no. 12, pp. 7761–7813, Jan. 2013.[12] T. Richardson, R. Urbanke et al. , “Multi-edge type LDPC codes,” in
Workshop honoring Prof. Bob McEliece on his 60thbirthday, California Institute of Technology, Pasadena, California , 2002, pp. 24–25.[13] C. H¨ager, H. D. Pfister, F. Br¨annstr¨om et al. , “Density evolution for deterministic generalized product codes on the binaryerasure channel,” arXiv preprint arXiv:1512.00433 , Dec. 2015.[14] R. Pedarsani, D. Yin, K. Lee, and K. Ramchandran, “Phasecode: Fast and efficient compressive phase retrieval based onsparse-graph codes,”
IEEE Transactions on Information Theory , 2017.715] F. Ong, S. Pawar, and K. Ramchandran, “Fast and efficient sparse 2D Discrete Fourier Transform using sparse-graph codes,” arXiv preprint arXiv:1509.05849 , Sep. 2015.[16] D. L. Donoho, A. Maleki, and A. Montanari, “Message passing algorithms for compressed sensing: I. motivation andconstruction,” in
IEEE Inf. Theory Workshop on Inf. Theory . IEEE, Jan. 2010, pp. 1–5.
A Proofs for Bounds
Here, we provide proofs for the bounds on L -smearing. A.1 Proof of Lemma 2
Proof.
Consider the depth neighborhood of a random node, as shown in Fig. 3. We define the following events: let A i bethe event that node B i is recovered by time t . Since we are not recursing here, we drop the references to t . Thus, what we areinterested in is: q t = P (cid:32) L (cid:91) i =1 P ( A i ) (cid:33) = P ( A ∪ ( A \ A ) ∪ ( A \ A ) ∪ · · · ∪ ( A L \ A L − ))= P ( A ) + P ( A \ A ) + · · · + P ( A L \ A L − )= L (cid:88) i =1 P ( A i ) − L − (cid:88) i =1 P ( A i , A i +1 ) (8)Now, we note that P ( A ) = P ( A k ) = d t s ( L − t and P ( A i ) = d t s + t ( L − i ) s ( i − t where < i < L . Additionally, we can seethat P ( A i , A i +1 ) = d t s ( i ) t s ( L − i ) t . Plugging these into equation ?? gives the result. A.2 Proof of Lemma 3
Proof.
First, we note that d t ≥ s ( j ) t ≥ s ( i ) t (9)if i ≥ j ≥ by definition. Additionally, we can see that f (0 , , . . . ,
0) = 0 and f (1 , , . . . ,
1) = 1 . Now, we have: ∂f∂d t = 2 s ( L − t + L − (cid:88) i =2 s ( i − t s ( L − i ) t − L − (cid:88) i =1 s ( i ) t s ( L − i ) t = s ( L − t + L − (cid:88) i =1 s (1) t ( s ( L − i − t − s ( L − i ) t ) + s ( L − t − s ( L − t s (1) t ≥ s ( L − t + s ( L − t (1 − s (1) t ) ≥ where the first inequality follows by Equation 9 and the last inequality follows since s (1) t ≤ and s ( L − t ≥ . Additionally, wecan see that for ≤ i < L − : ∂f∂s ( i ) t = 2 d t ( s ( L − i − t − s ( L − i ) t ) ≥ where the inequality follows from Equation 9. The argument follows similarly for ∂f∂s ( L − t . Thus, since the partial derivatives areall non-negative, the result follows. B -smearing ( S [1 , , ) In this section, we provide exact thresholds for the ensemble drawn from S [1 , , , illustrate why it is difficult to generalize, anddraw out the structure in the smearing patterns. 8e introduce the notation p t to denote the probability of an edge from a node to a bin in a stage with 1 smearing is not removed at time t . Also note that there are streams missing from the diagram, the symmetric picture for nodes “hanging off theedge” are not shown. We can see that (from Lemma 2): q t = P ( A ∪ A ∪ A )= P ( A ) + P ( A ) + P ( A ) − P ( A ∩ A ) − P ( A ∩ A ) − P ( A ∩ A ) + P ( A ∩ A ∩ A )= d t (2 s (2) t + ( s (1) t ) − s (2) t s (1) t ) (10)Now, exactly as in section 2.1, we can see that: d t = e − λp t − (11)Now we analyze s (1) t . We omit the detail before the Taylor series approximation for readability. Now, note that the nodes inthe stream can either all be cleared from the other stages, or they can be cleared by the bin shared by all streams tracked by s (2) t ,call this bin B . Additionally call the bin shared by the stream thrown to the outermost bin as B . The probability that the streamis cleared from the other stage is e − p t − . The only way the stream could have been cleared by B is if there were exactly noderemaining at time t − and B knows it is a singleton at time t − . Note that B is contaminated by the stream of balls thrownto the outermost bin and this stream must have been cleared by time t − . This can happen in not necessarily disjoint ways.First, B is “clear from below” and all the balls in the stream thrown to the outermost bin were peeled by t − . This happenswith probability s (1) t − e − λp t − (12)Second, exactly one node is missing from the stream of balls thrown to B , the stream of nodes thrown to the outermost bin wasempty at time t − , and B knows it is a singleton at time t − . This happens with probability: (cid:0) e − λp t − + s (2) t − λp t − e − λp t − (cid:1) · λp t − e − λp t − (13)Finally, one ball can be missing from the stream of balls thrown to the outermost bin. Then, B must be a singleton at time t − ,so we have the probability: s (2) t − λp t − e − λp t − (14)Finally, we add a term to correct for overcounting. The overlap between the first and last terms is: s (2) t − λp t − e − λp t − (15)Note that if at time t − , there were exactly node in the stream of nodes thrown to B , and both B and the bin below weresingletons, then the first and last terms overlap. Also note that these two streams are exactly s (2) t − from the perspective of B andthe bin below, and the term follows.Thus, we can see that: s (1) t = e − λp t − + λp t − e − λp t − · (cid:20) s (1) t − e − λp t − + (cid:0) e − λp t − + s (2) t − λp t − e − λp t − (cid:1) · λp t − e − λp t − + s (2) t − λp t − e − λp t − − s (2) t − λp t − e − λp t − (cid:21) (16)Finally, we give: s (2) t = e − λp t − + λp t − e − λp t − · (cid:20) s (1) t − e − λp t − + (cid:0) e − λp t − + s (2) t − λp t − · e − λp t − (cid:1) λp t − e − λp t − + s (2) t − λp t − e − λp t − + s (2) t − e − λp t − − s (2) t − e − λp t − − s (2) t − λp t − e − λp t − (cid:21) (17)We again note that these recursions show tight agreement with simulations. While they are difficult to digest, there is sig-nificant structure in the recursions. Analyzing the structure from whether or not a bin is cleared leads us to the bounds given inSection 2.2. Additionally, one can see that whereas smearing involved step of memory, smearing involves steps of mem-ory, and it becomes clear that in general L smearing will involve L − steps of memory. The amount of memory in addition tothe smearing length results in complex recursions. Hence, a simple lower bound that is easy to generalize is given in Section 2.2.9 Proof of Lemma 4
In order to describe the computational complexity, we first give pseudocode for the decoding algorithm:
Algorithm 1:
Basis-aware Peeling
Input :
Coupled bipartite graph G Output: X while singletons remain dofor B in bins doif B is singleton in a good stage then Peel( B ); endif B is singleton in a bad stage then AddToHypothesisList( B ); endendendAlgorithm 2: AddToHypothesisList
Input:
Bin BL ← Location( B ); V ← Value( B ); for B (cid:48) in bins connected to variable node ( L, V ) dofor Basis b in set of filters do B (cid:48) ← Hypothesis ( b, L, V ) ; if Singleton created then
Peel( B ); endendend Now, note that the creation of the bipartite graph is done in O ( K log K ) time and is disjoint from the decoding process.Decoding uses an iterative decoder with a constant number of iterations [5]. Additionally, the number of bins iterated over islinear in K . We now note the size of the hypothesis list in each bin is O ( K ) . Thus, if the list is stored using a data structure with O (log K ) insertion and deletion, such as a red-black tree, the iterative decoding complexity is O ( K log K ))