Density Evolution Analysis of Node-Based Verification-Based Algorithms in Compressive Sensing
Yaser Eftekhari, Anoosheh Heidarzadeh, Amir H. Banihashemi, Ioannis Lambadaris
DDensity Evolution Analysis of Node-BasedVerification-Based Algorithms in Compressed Sensing
Yaser Eftekhari, Anoosheh Heidarzadeh, Amir H. Banihashemi, Ioannis LambadarisDepartment of Systems and Computer Engineering, Carleton University, Ottawa, ON,CanadaE-mails: { eft-yas, anoosheh, Amir.Banihashemi, Ioannis } @sce.carleton.ca Abstract
In this paper, we present a new approach for the analysis of iterative node-based verification-based (NB-VB)recovery algorithms in the context of compressive sensing. These algorithms are particularly interesting due totheir low complexity (linear in the signal dimension n ). The asymptotic analysis predicts the fraction of unverifiedsignal elements at each iteration (cid:96) in the asymptotic regime where n → ∞ . The analysis is similar in nature to thewell-known density evolution technique commonly used to analyze iterative decoding algorithms. To perform theanalysis, a message-passing interpretation of NB-VB algorithms is provided. This interpretation lacks the extrinsicnature of standard message-passing algorithms to which density evolution is usually applied. This requires a numberof non-trivial modifications in the analysis. The analysis tracks the average performance of the recovery algorithmsover the ensembles of input signals and sensing matrices as a function of (cid:96) . Concentration results are devised todemonstrate that the performance of the recovery algorithms applied to any choice of the input signal over anyrealization of the sensing matrix follows the deterministic results of the analysis closely. Simulation results are alsoprovided which demonstrate that the proposed asymptotic analysis matches the performance of recovery algorithmsfor large but finite values of n . Compared to the existing technique for the analysis of NB-VB algorithms, whichis based on numerically solving a large system of coupled differential equations, the proposed method is muchsimpler and more accurate. Index Terms – Density evolution, compressive sensing, iterative recovery algorithms, sparse sensing matrix, sparsegraphs, message-passing algorithms.
I. I
NTRODUCTION
Compressed sensing was introduced with the idea to represent a signal v ∈ R n with k nonzero elementswith measurements c ∈ R m , where k < m (cid:28) n , and yet to be able to recover back the originalsignal v [1], [2]. In the measuring process, also referred to as encoding , signal elements are mapped tomeasurements through a linear transformation represented by the matrix multiplication c = Gv , wherethe matrix G ∈ R m × n is referred to as the sensing matrix . This linear mapping can also be characterizedby a bipartite graph [3], referred to as the sensing graph . a r X i v : . [ c s . I T ] A p r In the recovery process, also referred to as decoding , based on the knowledge of the measurements andthe sensing matrix, we estimate the original signal. Decoding process is successful if v is estimatedcorrectly. Three performance measures namely, density ratio γ (cid:44) k/n , compression ratio r c (cid:44) m/n , andoversampling ratio r o (cid:44) m/k are used in order to measure and compare the performance of the recoveryalgorithms in the context of compressed sensing. The sensing matrix in compressive sensing can be either dense or sparse . A sensing matrix is considereddense if it has few, if not none, zero entries. Sparse matrices, on the other hand, have few nonzero entriesin each row and column. One major difference between these two types of matrices is the encodingcomplexity associated with each class. For sparse matrices, the number of operations needed to calculatethe measurements is considerably lower than the one needed for dense matrices.Decoding algorithms can be classified based on the class of sensing matrix they use. The decodingalgorithms in each class have certain properties in common, which follows. (For a comprehensive studyon the topic we refer the interested readers to [7].) Decoding algorithms associated with dense matriceshave, generally, high complexity (between O ( n ) and O ( n ) ) compared to the lower complexity ofalgorithms utilizing sparse matrices (between O ( n ) and O ( n ) ). To have a better feeling about thecomplexity and running time of these two classes of algorithms, we have included (Section VII, Fig.3) the comparison between two standard recovery algorithms for dense matrices ( (cid:96) minimization andweighted (cid:96) minimization) and one algorithm (SBB) for sparse matrices. As can be seen, the decodingalgorithm on sparse matrices is faster by about two orders of magnitude. Decoding algorithms for densematrices are mostly based on linear or convex programming [1], [2], [8], [9]. The reason is that randomdense matrices satisfy restricted isometry property (RIP) with overwhelming probability [10], [11]. TheRIP was introduced by Cand`es and Tao in [12] as the main restriction on the sensing matrix so that therecovery based on linear programming will be able to successfully recover the signal. Sparse matrices,on the other hand, do not satisfy RIP unless m = Ω( k ) [13]. In fact, many of the decoders based onsparse sensing matrices are iterative [3], [14]–[26]. Although more computationally complex, decodingalgorithms on dense matrices tend to recover signals with larger number of nonzero elements (higherdensity ratio) compared to decoders on sparse matrices. Nevertheless, the high complexity of decodingalgorithms on dense matrices hinders their application to high-dimensional signal recovery (signals withlarge n ).Focusing on recovery algorithms based on sparse graphs, we can further divide them into two major groups.In one group, we have algorithms that use group testing and similar techniques from estimation theory For successful decoding clearly we need r o ≥ . It is desirable to have this parameter as small as possible. Indeed, in [4] authors provedthat r o = 1 is achievable in the asymptotic case ( n → ∞ ). This means that the highest density ratio that an algorithm can possibly handleis γ ∗ = r c . Authors in [5] have shown that if the sensing matrix consists of i.i.d. Gaussian elements, then a decoder based on the (cid:96) normcan recover the original signal with m = k + 1 measurements; i.e., r o ≈ . To find the solution based on the (cid:96) recovery, however, one hasto perform an exhaustive search, which is computationally too complex [6]. Authors in [7] extended the definition of RIP and showed that sparse matrices satisfy a generalized RIP constraint. The generalized RIPsuffices for the linear programming decoders to succeed in recovering sparse signals. However, the resulting bound on the reconstructionerror is weaker compared to the case where the sensing matrix satisfies the original RIP condition. [14]–[17]. These are referred to as combinatorial algorithms . In the other group, recovery algorithmswork with the bipartite graph associated with the sensing matrix by passing messages over the edges ofthe graph [3], [18]–[26]. These are referred to as message-passing algorithms. Combinatorial algorithms,generally, assume that the decoder knows the size of the support set, k [14]–[17]. These algorithms,have two main steps. In the first step, the algorithm outputs an estimate which has more nonzero valuesthan the original signal. In the next step, knowing the parameter k , the estimate is pruned so that theoutput estimate has the same support size as the original signal. Combinatorial algorithms are, in general,more computationally complex than message-passing algorithms. For example, the algorithm introducedin [14] has complexity O ( k polylog ( n )) , which translates to O ( n polylog ( n )) in the regime where k scales linearly with n . Message-passing algorithms, on the other hand, have computational complexity O ( n ) .In this work, we are interested in low-complexity recovery algorithms that exploit the sparsity of the sensingmatrix. In particular, we are interested in message-passing recovery algorithms. In [19], the authors proposea simple message-passing algorithm to reconstruct non-negative signals. This algorithm assumes lowerand upper bounds for the signal elements. It then shrinks the difference between the two bounds throughiterations. The important feature of the recovery algorithm introduced in [19] is its uniform guarantee onsignal reconstruction. Another approach in message-passing algorithms is to assume a prior distribution forthe signal elements and try to maximize the a-posteriori distribution of the elements based on the observedmeasurements. In [26], the authors assume Gaussian mixture priors. The main problem associated withthis approach is that the length of the messages passed over the edges of the graph grow exponentiallyfast with the number of iterations. In another work [25], the authors assume Jeffreys’ priors [27] andaim at recovering the support set of the original signal using message-passing algorithms. Then, theyapply well-known least-square algorithms, such as LSQR [28], to estimate the value of signal elements.Moreover, in [25], it is assumed that the size of the support set, k , is known. Algorithms discussed so farare either restrictive, in the sense that they assume some knowledge of the support set at the decoder, orhave a high computational complexity that makes them impractical in applications with large n .In this paper, we are interested in a sub-class of message-passing algorithms called Verification-Based (VB) algorithms. These algorithms were originally introduced in the context of channel coding with non-binary alphabet [29]. Thanks to the connection between compressive sensing and linear channel codesover real numbers noted in [21], the authors in [18] and [20] used VB algorithms in the context ofcompressed sensing. This class of algorithms has certain properties that make it perhaps one of themost interesting classes of recovery algorithms in compressed sensing. The VB algorithms recover signalelements in iterations. When an element is recovered, its value is kept unchanged in future iterations.The algorithms in this class have decoding complexity O ( n ) , which makes them suitable for applicationsinvolving recovery of signals with large n . Moreover, these algorithms operate on sparse sensing graphs,which translates to less computations in the encoding process. Another main advantage of VB algorithms is that they are not sensitive to the distribution of nonzero elements of the sensing matrix as well as thedistribution of nonzero elements of the signal, if certain conditions are satisfied. We will elaborate on thistopic further in section III. These properties make the VB algorithms a suitable choice for low-complexityrecovery of sparse signals. The VB algorithms are, however, sensitive to the presence of noise in themeasured data. In this work, our main focus is on the study of the noiseless case. This case is importantbecause i) the noise-free analysis of recovery algorithms can serve as an upper bound for the performanceof the noisy versions, and ii) noiseless compressive sensing has its own applications such as those in [24],[30]. For the sake of being thorough and to demonstrate the potential of VB algorithms in recoveringsignals from noisy measurements, we will also comment on using standard thresholding techniques todeal with noisy measurements in Section VI. An in-depth analysis of the approach, however, is beyondthe scope of this paper.Another interesting feature of VB algorithms is that their performance can be analyzed in the asymptoticcase ( n → ∞ ). Assume a probabilistic input model, in which a signal element is nonzero (and takes avalue from a certain distribution) with probability α and is zero with probability − α . In the sequel,we refer to parameter α as the density factor . Furthermore, let α ( (cid:96) ) denote the probability that a signalelement is nonzero and unverified before iteration (cid:96) over the ensemble of all sensing graphs and inputsof interest. So, α (0) = α . If lim (cid:96) →∞ α ( (cid:96) ) = 0 , then the algorithm is called successful for the initial densityfactor α . On the other hand, if there exists (cid:15) > , such that lim (cid:96) →∞ α ( (cid:96) ) > (cid:15) , then the algorithm is saidto fail for the initial density factor α . Using the combinatorial arguments in [31] and [32], one can seethat the algorithms complete the recovery once the probability α ( (cid:96) ) → .Authors in [20], [22], [23], [29] have shown that for each VB recovery algorithm in the asymptotic regimeas n → ∞ and (cid:96) → ∞ , a limiting value exists for α , before which the recovery algorithm is successfuland beyond which it is not. We refer to this limit as the success threshold . The success threshold serves asan asymptotic measure of performance for VB algorithms. It can also be used to estimate the performanceof these algorithms for finite but large values of n . To this end, researchers have analyzed VB algorithmsin the asymptotic regime ( n → ∞ , (cid:96) → ∞ ) in order to find the success threshold associated with eachVB algorithm. There are two categories of VB algorithms: node-based (NB) and message-based (MB) [23]. The two categories yield different success thresholds and are analyzed using different techniques.In general, NB algorithms have higher success thresholds and are harder to analyze. We elaborate on thedifferences between the two categories and their corresponding analytical tools in Section II-B. The focusof this work is on the analysis of NB algorithms.Asymptotic analysis of VB algorithms can be found in [20], [23], [29]. Algorithms considered in [20],[29] are of MB type, while the authors in [23] considered the NB type recovery algorithms. Moreover, fortheir analysis, the authors of [20] made the assumption that k/n → as n → ∞ . From a practical point It is easy to prove that the probability of a zero-valued signal element being unverified at iteration (cid:96) is upper bounded by d c α ( (cid:96) ) − α ( (cid:96) ) .Hence, when α ( (cid:96) ) tends to zero, this probability also tends to zero. of view, however, it is more desirable to analyze the cases in which the number of nonzero elements ofthe signal, k , scales linearly with its dimension n . In fact, in this paper, we show that VB algorithms arecapable of recovering signals whose density ratio is a nonzero constant, i.e., k/n remains constant, as n is increased.The analysis of NB-VB algorithms discussed in [23] results in a system of coupled differential equations.Due to the high complexity of solving the resulting differential equations, the authors used numericalmethods to approximate the asymptotic results. They further assumed that the algorithms resolve atmost one signal element in each iteration. Therefore, the number of iterations needed for the numericalanalysis equals n ; the number of signal elements. The challenges associated with the analysis of [23] aretwofold: 1) as the analysis is only valid for n → ∞ , one has to choose very large n for the numericalapproximation, which directly translates to long running time and high computational complexity, and 2)since the numerical approach is used to approximately solve the differential equations, the approximationerrors can potentially propagate through the iterations. This makes it hard to evaluate how close thesuccess threshold reported by this analysis (even for large values of n ) is to the real success threshold. Incomparison, the analysis proposed in this paper is much faster (by about two orders of magnitude for thetested cases) and is more robust against numerical errors. The general approach for the analysis is alsodifferent and is based on basic probability theory.The goal of this work is to develop a low-complexity framework for the asymptotic analysis (as n →∞ , (cid:96) → ∞ ) of NB-VB algorithms over sparse random sensing graphs and extend it to include recoveryalgorithms of similar nature such as that of [3]. In our analysis, we assume that the measurements arenoiseless. The main analytical tool used in this paper is probability theory. We demonstrate that therecovery algorithms can be described by a first order time-varying Markov chain. We thus track thedistribution of the states of this Markov chain through iterations in the analysis. The purpose is to findthe transition probabilities between different states of the Markov chain as the iterations progress. Thecomputational complexity of the proposed analysis thus increases linearly with the number of iterations.The calculation of transition probabilities includes simple mathematical operations, more specificallyaddition and multiplication, as opposed to solving complex systems of coupled differential equations,as is the case in [23]. One should however note that for a sensing graph in which each signal elementaffects d v measurements and each measurement is a linear combination of d c signal elements, where d v and d c are fixed and positive integers, the number of states in the proposed analysis is O ( d v + d c ) ,which is in the same order as the number of differential equations in [23] is. As part of our asymptoticanalysis, we also prove concentration results which certify that the performance of a recovery algorithmfor a random choice of the input signal and the sensing matrix is very close to what is predicted by thedensity evolution results at the limit of n → ∞ .Using the proposed analysis, we can determine the distribution of the decoder states at any desirediteration. By tracking the distribution of the decoder states with iterations, we then find the success threshold of different NB-VB algorithms. Moreover, using our approach, we perform a comprehensivestudy and comparison of performance of different VB recovery algorithms over a variety of sparse graphs.Our simulations show that the behavior of VB algorithms, when applied to signals with large lengths (inthe order of ), are in good agreement with the asymptotic analytical results.The rest of the paper is organized as follows. In section II, we introduce the class of bipartite graphsand inputs signals of interest in this paper. We also provide a more detailed description of VB algorithmsin this section. In section III, the decoding process in each VB algorithm is discussed further. Also inthis section, we discuss the important notion of false verification and its probability in VB algorithms. Amessage-passing interpretation of the recovery algorithms is presented in section IV. In this section, wealso make a more detailed distinction between NB and MB recovery algorithms. The analysis frameworkwill be introduced in section V. We propose a simple modification of VB algorithms to deal with noisymeasurements in section VI. Simulation results will be presented in section VII. Appendices I, II, and IIIare devoted to the derivation of the transition probabilities. Appendix IV consists of some bounds neededfor the concentration theorem, presented and proved in section V.II. B ACKGROUND
A. Ensembles of Sensing Graphs and Inputs A bipartite graph (or bigraph ) G ( V ∪ C, E ) is defined as a graph whose set of vertices V ∪ C is dividedinto two disjoint sets V and C , so that every edge in the set of edges E connects a vertex in V to one in C . Corresponding to each such graph, a biadjacency matrix A ( G ) of size | C | × | V | is formed as follows:the entry a ij is if there exists an edge e ij ∈ E connecting the vertex c i ∈ C to the vertex v j ∈ V ; andis , otherwise.Let d v and d c be two positive integers. Consider a bigraph G ( V ∪ C, E ) with | V | = n and | C | = m , so thateach vertex in V ( C ) is incident to d v ( d c ) vertices in C ( V ). Clearly, nd v = md c . We refer to this bigraphas an ( n, d v , d c ) - biregular graph . The biadjacenecy matrix A ( G ) associated to an ( n, d v , d c ) -biregulargraph has d c ’s in each row and d v ’s in each column.Moreover, a bipartite weighted graph (or weighted bigraph ) G (cid:48) ( V ∪ C, W ( E )) is a generalization of thebigraph G ( V ∪ C, E ) in the sense that a weight w ij := w ( e ij ) ∈ R \{ } is associated with each edge e ij ∈ E . The biadjacency matrix A ( G (cid:48) ) corresponding to the weighted bigraph G (cid:48) is acquired from thebiadjacency matrix A ( G ) of the underlying bigraph G by replacing nonzero a ij values in A ( G ) with w ij .A regular bipartite weighted graph (or weighted biregular graph ) is defined similarly.For given parameters d v , d c and n ( m = nd v /d c ), let G n ( d v , d c ) denote the ensemble of all ( n, d v , d c ) -biregular graphs. Let us assume an arbitrary, but fixed, labeling scheme for vertex sets V and C over theensemble. Further, let W be a matrix of size m × n of weights w drawn i.i.d. according to a distribution f ( w ) , and W m × nf be the ensemble of all such matrices. Now for any biregular graph G ( V ∪ C, E ) ∈G n ( d v , d c ) and any weight matrix W ∈ W m × nf , we form the corresponding ( n, d v , d c ) -weighted biregulargraph G (cid:48) ( V ∪ C, W ( E )) as follows. To every edge e ij ∈ E , ≤ i ≤ m, ≤ j ≤ n , connecting i th vertexfrom C and j th vertex from V , we assign the weight w ( e ij ) = w ij ; i.e., the weight in row i and column j of the weight matrix W . Thus, we construct the ensemble of all ( n, d v , d c ) -weighted biregular graphs,denoted by G nf ( d v , d c ) , by freely combining elements in G n ( d v , d c ) and W m × nf .Thus far, we have described the ensemble of graphs that are of interest in this work. In what fallows,we describe the ensemble of inputs of interest. Let α ∈ [0 , be a fixed real number and v be a vectorof length n with elements v i drawn i.i.d. according to a probability distribution function defined asfollows: the element is zero with probability − α , or follows a distribution g with probability α (i.e., Pr[ v i = v ] = αg ( v ) + (1 − α ) δ ( v ) , where δ is the Dirac delta function). We denote the ensemble of allsuch vectors by V ng ( α ) . In compressive sensing, each measurement is a linear combination of the signal elements. With a slightabuse of notation, we use c i ( v j ) for both the label and the value of the i th measurement (the j thsignal element). We denote by c and v , the column vectors of the measurements c i ’s ( ≤ i ≤ m ),and the signal elements v j ’s ( ≤ j ≤ n ), respectively. The underlying system of linear combinationscan then be represented by the matrix multiplication c = Gv . In this paper, the sensing matrix G is thebiadjacency matrix of a weighted bigraph G ( V ∪ C, W ( E )) drawn uniformly at random from the ensemble G nf ( d v , d c ) . Henceforth, we refer to the graph G as the sensing graph . Moreover, the signal vector v isdrawn uniformly at random from the ensemble V ng ( α ) . The sets of signal elements and measurements arerespectively mapped to the vertex sets V and C ( | V | = n , | C | = m ). The coefficient of the j th signalelement ( v j ∈ V ) in the linear combination associated with the i th measurement c i ∈ C , the entry g ij in G , is the entry w ij of the biadjacency matrix A ( G ) of G .Following the terminology frequently used in the context of coding, we refer to the sets V and C asthe variable nodes and check nodes , respectively. We will interchangeably use the terms variable nodesand signal elements as well as check nodes and measurements. The main focus of this paper is on theweighted biregular graphs. The results however, can be generalized to irregular graphs. B. Previous Work on VB Algorithms
Luby and Mitzenmacher [29] proposed and analyzed two iterative algorithms over bigraphs for packet-based error correction in the context of channel coding. In these algorithms, a variable node can be in one It is worth noting that the expected fraction of nonzero elements in such a vector is α . Using a Chernoff bound, it can be shown that theactual fraction of nonzero elements in a randomly chosen vector from this ensemble is tightly concentrated around its expected value ( α )with high probability. In the context of coding, a linear code can be represented by a bigraph, where the two sets of nodes represent the code symbols, and thelinear constraints that the symbols have to satisfy [33]. of the two states: “verified” or “unverified”. Under certain circumstances, a variable node is verified and avalue is assigned to it. This node then contributes to the verification of other variable nodes. The decodingprocess continues until either the entire set of unverified variable nodes is verified, or the process makesno further progress while there are still some unverified variables. Due to the verification nature of theprocedure, the two algorithms in [29] are called verification-based (VB) algorithms. If the assigned valueto a variable node at a certain iteration is different from its true value, a false verification has occurred. Insection III, we discuss sufficient conditions for VB algorithms so that the probability of false verificationis zero.The verification process in VB algorithms can be seen as a message-passing procedure. In general, avariable node sends its current state (either verified or unverified) to its neighboring check nodes alongwith its value (if verified). A check node processes the received messages and subsequently sends somemessages to its neighboring variable nodes. Each unverified variable node decides on its next state, eitherverified or unverified, based on the received messages from check nodes. The process of passing messagesbetween variable nodes and check nodes continues until all variable nodes are verified, or no variablenode changes its state.In message-passing algorithms, a node can take two approaches in order to produce a message based onthe set of received messages. In the first approach, the outgoing message is a function of all receivedmessages. In this case, all messages leaving a node at a certain iteration are the same. In the secondapproach, the message passed from node a to node b in the bigraph, is a function of all the receivedmessages by node a except the received message from node b . Therefore, the outgoing messages of anode at a certain iteration may be different, depending on the received messages. In the context of VBalgorithms, the first approach is known as node-based (NB) , while the second approach is called message-based (MB) [20], [23]. So, for a NB-VB algorithm, the state of a variable node is reported identicallyby all its outgoing messages, while in an MB-VB algorithm, different states may be reported by differentoutgoing messages from a variable node.As noted in [23], the authors in [29] defined the two VB algorithms using the NB representation butanalyzed them using the MB representation. In [23], the authors proved that for one of the VB algorithms,the NB and MB versions perform the same, but for the other VB algorithm, the NB version outperforms theMB one. In compressed sensing, this implies that NB versions, in general, have higher success thresholds;i.e., can successfully recover signals with larger density ratios [21].A well-known method to analyze iterative message-passing algorithms in coding theory is density evolution[35]. In density evolution, the distribution of messages is tracked with the iteration number. The evolutionof the message distributions with iterations will then reveal important properties of the decoding algorithmsuch as decoding threshold and convergence speed [35]. The derivation of the message distribution In the context of iterative decoding algorithms, NB and MB approaches are known as non-extrinsic and extrinsic message-passing,respectively [34]. however, requires the independence among the incoming messages to a node. The analysis is thus onlyapplicable to extrinsic message-passing algorithms (MB decoders). To extend density evolution to NBalgorithms, Zhang and Pfister [23] used a system of differential equations as the analytical tool. Applyingtheir analysis to ( d v , d c ) graphs, the number of differential equations is O ( d v + d c ) . This rapidly becomestoo complex to handle for large values of d v and d c . Numerical methods were used in [23] to solve thesystem of differential equations and consequently evaluate the performance of the NB algorithms. It isimportant to note that the authors in [23] analyzed a serial version of NB-VB algorithms, i.e., the versionwhich allows only one variable node to be verified in each iteration. The total number of iterations intheir calculations is thus equal to the size of the signal n . Another important issue regarding the analysisof [23] is that while the validity of the analysis is proved for the asymptotic scenario of n → ∞ , thenumerical results are highly dependent on the selected, large but still finite, value of n .III. VB A LGORITHMS , V
ERIFICATION R ULES AND F ALSE V ERIFICATION
A. VB Algorithms and Verification Rules
In compressive sensing, the decoder receives the vector of measurements and aims at estimating the originalsignal based on the knowledge of measurements and the sensing graph. In this section, we discuss fourVB decoding algorithms.The first algorithm, here referred to as “Genie”, is a benchmark VB algorithm in which the support set ofthe signal is known to the decoder. We use the Genie algorithm and its analysis to motivate and explainthe analytical framework. The success threshold associated with this algorithm serves as an upper boundfor the performance of other VB algorithms.In other recovery algorithms, the decoder has no information about the support set. The next two decodersconsidered in this paper, are the two main VB decoding algorithms in the context of CS. The first algorithmis referred to as LM and is the first algorithm discussed in [20]; LM1. The second main VB algorithm isthe algorithm introduced in [18], which is the same as the second algorithm discussed in [20]; LM2. Werefer to this algorithm as SBB.By the description given in Section II-B, the algorithm in [3], here referred to as XH, also falls into thecategory of VB algorithms, and can be also analyzed using the proposed framework. The details of theanalysis for this algorithm however, is not included in this paper. We just report some numerical resultson the success threshold and the convergence speed of this algorithm in Section VII.In what follows, we give the general description of the aforementioned VB algorithms, as found in theliterature ( [3], [20], [22], [23]). We then use this description to discuss the issue of false verification. In The number of differential equations for a (3,6) bigraph is about 30. The Genie algorithm is similar to the peeling algorithm over the BEC proposed in [31]. Section IV, we present the equivalent message-passing description of the VB algorithms.In the VB algorithms, when a variable node is verified at an iteration, its verified value is subtracted fromthe value of its neighboring check nodes. The variable node, then, is removed from the sensing bigraphalong with all its adjacent edges. Hence, all the neighboring check nodes of the verified variable nodeface a reduction in their degree. In the next iteration, some variable nodes may be verified based on thedegree and/or the value of their neighboring check nodes. The rules based on which the variable nodesare verified at each iteration are called verification rules and are as follows: • Zero Check Node (ZCN): If a check node has a zero value, all its neighboring variable nodes areverified with a zero value. • Degree One Check Node (D1CN): If a check node has degree 1 in a graph, its unique neighboringvariable node is verified with the value of the check node. • Equal Check Nodes (ECN): Suppose we have N check nodes with the same nonzero value, then 1)all variable nodes neighboring a subset of these N check nodes (not all of them) are verified withthe value zero; 2) if there exists a unique variable node neighboring all N check nodes, then it isverified with the common value of the check nodes.Verification rules ZCN and ECN are responsible for verifying variable nodes not in the support set. Since,the Genie algorithm has the complete knowledge of the support set, it has no need to apply these tworules. Hence, D1CN is the only rule used by the Genie. Other VB algorithms, each uses a combination ofverification rules in order to verify and resolve unverified variable nodes. Assuming zero probability forfalse verification, the order in which the rules are applied does not affect the overall performance of thealgorithm; it will only change the order in which variable nodes are verified. Verification rules adoptedby different algorithms are summarized in Table I. TABLE IV
ERIFICATION RULES ADOPTED IN EACH VB ALGORITHM
ZCN D1CN ECNGenie Not Needed ! Not NeededLM ! ! %
SBB ! ! ! XH ! % ! Based on Table I, SBB applies the union of all rules to verify variable nodes. Therefore, this algorithm isexpected to have the highest success threshold amongst the practical VB algorithms discussed here. Thisis verified in Section VII.The ECN rule as stated above can not be easily captured in the analysis. Based on our extensive simulations,we conjecture that this recovery rule can be modified to read as follows (without affecting the asymptoticbehavior of the recovery algorithms): Modified Equal Check Nodes (MECN): Suppose we have N check nodes with the same nonzero value.Then if there exists a unique variable node neighbor to all such check nodes, it is verified with the commonvalue of the check nodes. It is only in this case that any other variable node connected to such checknodes is verified as zero. B. False Verification
Let K denote the set of nonzero variable nodes in the signal; the support set. Also, let M ( c ) denote theset of variable nodes neighbor to a check node c . Now, consider the following facts:( ) Let C be an arbitrary subset of check nodes. If all the check nodes in C are neighbor to the samesubset of nodes in K , then all these check nodes have the same value.( ) Any check node with no neighbor in K has a zero value.Verification rules ZCN and ECN in VB algorithms are designed based on the following assumptions:( (cid:48) ) Let C (cid:48) be any arbitrary subset of check nodes with the same value. Then all these check nodes areneighbor to the same subset of K .( (cid:48) ) For any zero valued check node, none of its neighboring variable nodes belong to the set K .It is worth noting that the assumptions ( (cid:48) ) and ( (cid:48) ) are the converses of the facts (1) and (2), respectively.To any choice of distributions f and g for nonzero weights of the sensing graph and nonzero signalelements, respectively, corresponds a certain probability that the converses fail to hold. Those distributionswhich make the converses hold true with probability (almost surely), are of interest in this paper. In thefollowing theorem, we give an example of distributions that make the statements ( (cid:48) ) and ( (cid:48) ) hold truealmost surely. Theorem 1.
Let c i and c j be two distinct check nodes and V i and V j be their corresponding set ofneighboring variable nodes in K ; i.e., V i = M ( c i ) ∩ K and V j = M ( c j ) ∩ K . Suppose that at least oneof the distributions f or g described before is continuous. Then the statements ( (cid:48) ) and ( (cid:48) ), describedabove, are correct with probability one for c i and c j . To prove the theorem, we need the following fact.
Fact 1.
Let x i and x j be two independent samples drawn from a continuous distribution. It follows that: Pr ( x i = x j ) = 0 . Stated differently, no two independent samples of a continuous distribution will have the same value,almost surely. Moreover, for any constant c , we have Pr ( x i = c ) = 0 . Proof of Theorem 1:
Let G = [ w ij ] be the sensing matrix. The value of a check node c i is (cid:80) j : v j ∈M ( c i ) w ij v j . Therefore, if at least one of the following conditions is satisfied, then the value ofa check node follows a continuous distribution:1) the nonzero elements of the sensing matrix (weights associated to the edges of the sensing graph),follow a continuous distribution f .2) the value of a nonzero variable node (a variable node in the support set K ) follows a continuousdistribution g .Therefore, according to Fact 1, the subset of check nodes with the same value can not be independentalmost surely. Their dependence implies that they are neighbor to the same set of nonzero variable nodes.This proves the statement ( (cid:48) ). Similarly, if a check node is neighbor to at least one nonzero variablenode, its value follows a continuous distribution, which according to Fact 1 is zero with probability zero.This proves the statement ( (cid:48) ).So, the continuity of f or g is a sufficient condition to have the probability of false verification equalto zero. In the rest of the paper, we assume that the statements ( (cid:48) ) and ( (cid:48) ) are correct with probabilityone and consequently, the probability of false verification for a variable node in any iteration of the VBalgorithms is zero. Using the union bound, one can see that the probability of false verification in anyiteration and also in the whole recovery algorithm is zero.IV. VB R ECOVERY A LGORITHMS AS M ESSAGE -P ASSING A LGORITHMS
A. Definitions and Setup
There are a number of VB decoding algorithms that can be formulated as node-based message-passing (NB-MP) algorithms. These are the algorithms that are of interest to us in this paper. Each algorithmworks in iterations through exchanging messages between the check nodes and the variable nodes alongthe edges in the graph. Any message sent from a variable node to its neighboring check nodes belongs toan alphabet set M : { , } × R . The first coordinate of such a message is a status flag, sometimes referredto as “recovery flag”, taking binary values. The flag indicates the verification status of the variable node.If this flag is , then the variable node is not verified. If, on the other hand, the flag is , then the variablenode has been verified. In this case, the second coordinate, which is a real number, is interpreted as theverified value of the variable node.Similarly, any message sent from a check node to all its neighboring variable nodes belongs to an alphabetset O : Z + × R . The first coordinate of such a message indicates the number of unverified variable nodesneighbor to the check node. The first coordinate is in fact the degree of the check node in the subgraphinduced by the unverified variable nodes. The second coordinate indicates the current value of the checknode, i.e., the result of the linear combination of the unverified neighboring variable nodes. The edges, in NB-MP algorithms, do not simply forward messages from check nodes to variable nodes andvice versa. Instead, based on the traveling direction of the message, edges multiply or divide the secondcoordinate of the message by their associated weight. More specifically, if the message is sent from avariable node to a check node, its second coordinate is multiplied by the weight. The second coordinateof the message is divided by the weight, if the message is sent from a check node to a variable node. So,although messages generated by a node (either variable node or check node) are sent identically over alladjacent edges, the fact that the edges may have different weights will result in different messages beingreceived at the destination nodes. All such messages are independent if the weights associated with thecorresponding edges are independent.Any iteration (cid:96) ≥ in NB-VB algorithms, consists of two rounds, each with two half-rounds. In eachround, every check node processes all received messages from the previous round together with itsassociated measurement and sends out a message from the alphabet O to all its neighboring variablenodes (first half-round). In the second half-round, each (unverified) variable node decides on its nextstate by processing all its received messages. Whichever the decision is, the variable node sends backa message, from the alphabet M , to all its neighboring check nodes. So, a round starts with checknodes processing the received messages from neighboring variable nodes, proceeds with the transmissionof messages from check nodes to variable nodes, continues by variable nodes processing the receivedmessages from neighboring check nodes, and ends with the transmission of messages from variable nodesto check nodes. The two rounds in each iteration follow the same general structure. They only differ inthe processing carried out in the variable nodes.Let Φ (1 ,(cid:96) ) v : O d v → M and Φ (2 ,(cid:96) ) v : O d v → M , (cid:96) ∈ N , represent the mappings used at any unverified variable node to map the incoming messages to the outgoing message in the first and the second roundof iteration (cid:96) , respectively. Obviously, due to the verification-based nature of the algorithms, when avariable node becomes verified at an iteration, its outgoing message remains unchanged, irrespective ofits incoming messages. In contrast to the variable nodes, the mapping function used in check nodes isidentical for both the first and the second round of each iteration. Every check node c i , i ∈ [ m ] hasan associated received measurement c i , a random variable taking values in R . So, we use the notation Φ ( (cid:96) ) c : R × M d c → O , (cid:96) ∈ N , to denote the mapping function used in all check nodes at iteration (cid:96) . Forthe sake of completeness, let Φ (0) v = Φ (2 , v : O d v → M and Φ (0) c : R → O represent the mappings used,respectively in all variable nodes and check nodes at iteration . This iteration consists of only one round.For the VB algorithms under consideration, the mapping functions in the variable nodes and check nodesare not a function of the iteration number. Therefore, we omit the superscript ( (cid:96) ) henceforth.In what follows, we describe VB algorithms of Section III as message-passing algorithms with the general structure explained above. B. Message-Passing Description of Recovery Algorithms
To describe the four VB recovery algorithms using the message-passing approach, we need to definethe mappings Φ (1) v , Φ (2) v and Φ c . Mapping Φ (1) v embeds the verification rules D1CN and ECN, while themapping Φ (2) v embeds the ZCN rule. To make the description of mappings Φ v and Φ c simpler, we introducesome notations to represent the incoming messages to variable and check nodes from the alphabet sets O and M , respectively. A message o ∈ O , incoming to a variable node, is an ordered pair of elements ( d, ξ ) , where d ∈ Z + , ξ ∈ R . A message m ∈ M , incoming to a check node, is an ordered pair ofelements ( s, ω ) , where s ∈ { , } , ω ∈ R . Moreover, we assume that there is an arbitrary numbering foredges adjacent to a node (either variable node or check node). So, we use the notations o i , i ∈ [ d v ] and m j , j ∈ [ d c ] , to denote the incoming messages to variable nodes and check nodes, respectively.At iteration zero, all variable nodes are unverified and there is no received message at the check nodes.At this stage, all check nodes send their corresponding measurements along with their degree ( d c ) to theirneighboring variable nodes. For the following iterations (cid:96) ≥ , the mapping function at any check node c i is as follows: Φ c ( c i , m , · · · , m d c ) = ( d c − d c (cid:88) i =1 s i , c i − d c (cid:88) i =1 s i ω i ) , where, c i is the measurement associated with the check node c i , and m i = ( s i , ω i ) is the message receivedalong the i th edge. The mapping functions Φ (1) v , Φ (2) v are algorithm dependent and are discussed for eachVB algorithm separately next.The decoder stops at an iteration (cid:96) , (cid:96) ≥ , if the algorithm makes no further progress, i.e., the set ofverified variable nodes are the same for the two consecutive iterations (cid:96) − and (cid:96) . Equivalently, thealgorithm stops if the messages sent from variable nodes to check nodes, and also from check nodes tovariable nodes, are the same for two consecutive iterations (cid:96) and (cid:96) − . At this point, if the decoder isable to verify all the variable nodes, then the decoding is called successful. Otherwise, the decoder willdeclare a failure. Genie
In this algorithm, the decoder has the knowledge of the support set K . So, the verification rules ZCNand ECN are not needed for this algorithm. Hence, each iteration in this algorithm consists of only oneround, in which one verification rule (D1CN) is applied to all variable nodes. For variable nodes not in It is worth mentioning that the message-passing description of the NB-VB algorithms, presented in Section IV-B, is only valid for thecases in which the nonzero weights of the sensing graph are drawn from an uncountable or countably infinite alphabet set. If the elementsof the sensing matrix are drawn from a finite alphabet set, such as binary and , the outgoing messages from a check node should alsoinclude the list of all unverified variable nodes neighbor to the check node. The mapping function in the variable nodes should also changein order to use the extra information in the incoming messages. the support set, the outgoing message in all iterations is fixed and equals m = (1 , .For any variable node in the support set, the mapping Φ v ( o , · · · , o d v ) is defined based on the followingrules. • If among all received messages (from the neighboring check node), there exists only one message,say o i , i ∈ [ d c ] , such that o i = (1 , ξ i ) , ξ i ∈ R , then Φ v ( o , · · · , o d v ) = (1 , ξ i ) . In this case, the variablenode is verified with the value ξ i . • If multiple messages exist in the form (1 , ξ ) (any ξ ∈ R ), then choose one at random, say o i =(1 , ξ i ) , ξ i ∈ R , and set Φ v ( o , · · · , o d v ) = (1 , ξ i ) . In this case, the variable node is verified with thevalue ξ i . • If none of the above happens, then Φ v ( o , · · · , o d v ) = (0 , . In this case, the variable node is stillunverified. LM For any unverified variable node in this algorithm, the mappings Φ (1) v and Φ (2) v are defined according tothe following rules. • Φ (1) v ( o , · · · , o d v ) : – If among all received messages (from the neighboring check nodes), there exists only onemessage, say o i , i ∈ [ d c ] , such that o i = (1 , ξ i ) , ξ i ∈ R , then Φ (1) v ( o , · · · , o d v ) = (1 , ξ i ) . Inthis case, the variable node is verified with the value ξ i . – If multiple messages exist in the form (1 , ξ ) (any ξ ∈ R ), then choose one at random, say o i = (1 , ξ i ) , ξ i ∈ R , and set Φ (1) v ( o , · · · , o d v ) = (1 , ξ i ) . In this case, the variable node is verifiedwith the value ξ i . – If none of the above happens, then Φ (1) v ( o , · · · , o d v ) = (0 , . In this case, the variable node isstill unverified. • Φ (2) v ( o , · · · , o d v ) : – If there exists at least one message o i such that o i = ( d i , (for any d i ∈ Z + ), then Φ (2) v ( o , · · · , o d v ) = (1 , . In this case, the variable node is verified with the value equal to . – If no incoming message exists in the form ( d i , (for any d i ∈ Z + ), then Φ (2) v ( o , · · · , o d v ) = (0 , . In this case, the variable node is still unverified. SBB For any unverified variable node in this algorithm, the mappings Φ (1) v and Φ (2) v are defined by the followingrules. The original recovery algorithm introduced in [18] has a computational complexity of O ( m · log m ) (which translates to O ( n · log n ) when using biregular graphs of fixed degrees). It is easy to prove that the message-passing description provided here does not change therecovery capability of the algorithm but results in the reduction of decoding complexity from O ( n · log n ) to O ( n ) . • Φ (1) v ( o , · · · , o d v ) : – If among all received messages (from the neighboring check nodes), there exists only onemessage, say o i , i ∈ [ d c ] , such that o i = (1 , ξ i ) , ξ i ∈ R , then Φ (1) v ( o , · · · , o d v ) = (1 , ξ i ) . Inthis case, the variable node is verified with the value ξ i . – If there exist N messages ( ≤ N ≤ d v ) o i = ( d i , ξ i ) , o i = ( d i , ξ i ) , · · · , o i N = ( d i N , ξ i N ) ,such that ξ i = ξ i = · · · = ξ i N , then Φ (1) v ( o , · · · , o d v ) = (1 , ξ i ) . In this case, the variable nodeis verified with the common value ξ i . – If a variable node is verified to different values according to verification rules above, then chooseone at random and generate the outgoing message accordingly. – If none of the above happens, then Φ (1) v ( o , · · · , o d v ) = (0 , . In this case, the variable node isstill unverified. • Φ (2) v ( o , · · · , o d v ) : – If there exists at least one message o i such that o i = ( d i , (for any d i ∈ Z + ), then Φ (2) v ( o , · · · , o d v ) = (1 , . In this case, the variable node is verified with the value equal to . – If no incoming message exists in the form ( d i , (for any d i ∈ Z + ), then Φ (2) v ( o , · · · , o d v ) = (0 , . In this case, the variable node is still unverified. XH For any unverified variable node in this algorithm, the mappings Φ (1) v and Φ (2) v are defined according tothe following rules. • Φ (1) v ( o , · · · , o d v ) : – If there exist M messages ( (cid:100) d v / (cid:101) ≤ M ≤ d v ) o i = ( d i , ξ i ) , o i = ( d i , ξ i ) , · · · , o i M =( d i M , ξ i M ) , such that ξ i = ξ i = · · · = ξ i M , then Φ (1) v ( o , · · · , o d v ) = (1 , ξ i ) . In this case, thevariable node is verified with the common value ξ i . – If a variable node is verified to different values according to the verification rule above, i.e., iftwo groups of messages both at least of size d v / satisfy the above condition, then choose oneat random and generate the outgoing message accordingly. – If none of the above happens, then Φ (1) v ( o , · · · , o d v ) = (0 , . In this case, the variable node isstill unverified. • Φ (2) v ( o , · · · , o d v ) : – If there exists at least one message o i such that o i = ( d i , (for any d i ∈ Z + ), then Φ (2) v ( o , · · · , o d v ) = (1 , . In this case, the variable node is verified with the value equal to . – If no incoming message exists in the form ( d i , (for any d i ∈ Z + ), then Φ (2) v ( o , · · · , o d v ) = (0 , . In this case, the variable node is still unverified. We note that the message received by the variable node equals the message sent from the check node divided by the weight of theconnecting edge. Therefore, receiving two messages with the same value would imply that, almost surely, the unverified variable node underconsideration is the unique nonzero variable node neighbor to the N check nodes. Other unverified variable nodes neighbor to these N check nodes do not belong to the support set and should be verified with a value equal to zero. This, however, happens in the next round. C. A Short Note on False Verification
In the above description of recovery algorithms, there may be cases where a variable node can be verifiedto different values by different rules. Using the same assumption made in Section III-B, it is easy to seethat the probability of this event is equal to zero. In such cases, we have thus assumed that the variablenode is verified by one of the rules selected randomly. Clearly, the probability of false verification as aresult of such selection is zero.V. A
SYMPTOTIC A NALYSIS F RAMEWORK
In this section, we first show that (i) the performance of a realization of the sensing graph, with a certainselection of the edge weights for the recovery of a realization of the input signal concentrates around theaverage performance of the ensemble (where the average is taken over all the elements in the ensemble G nf ( d v , d c ) × V ng ( α ) , for given probability distribution functions f, g , and given constant parameters d v , d c and α ), as n tends to infinity, and (ii) the average performance of the ensemble, as n goes to infinity,converges to the performance of the cycle-free case defined as follows. Let N (cid:96)v be the neighborhood of node v of depth (cid:96) , i.e., the subgraph consisting of the variable node v and all those nodes that are connected to v with any path of length less than or equal to (cid:96) . We say thatwe are working under the cycle-free assumption when for a fixed (cid:96) , and for every v , N (cid:96)v is tree-like.Similar to [35], the concentration result presented here is valid for any iteration number (cid:96) . We thus oftenomit the superscript (cid:96) in the notations.Following the concentration results, we present the asymptotic analysis of the Genie algorithm. Theanalysis provides us with the average ensemble performance for the asymptotic case of n → ∞ . We thengeneralize the concepts used in the analysis of Genie and analyze LM and SBB algorithms. The analysisof XH is similar and is omitted to prevent redundancy. However, we shall report the thresholds of thisdecoder for different biregular graphs in section VII. A. Concentration Results and Convergence to Cycle-Free Case
Consider a weighted graph selected at random from G nf ( d v , d c ) . Such a graph can be represented by itscorresponding unweighted graph G (from the ensemble G n ( d v , d c ) ) along with its vector of edge weights w (the vector of elements, with an arbitrary fixed order, in an m × n matrix W from the ensemble W m × nf ).Also consider an input signal vector v chosen randomly from V ng ( α ) . Suppose that a VB algorithm isapplied to the measurement vector c = Gv to recover v iteratively, where G is the biadjacency matrix Our method of proof is very similar to that of [35], though due to the differences in the nature of the problems (channel coding vs.compressed sensing) and the difference in the update rules at the graph nodes, some arguments are revised and some new components areadded to the proof. of the chosen weighted graph. For this scenario, let β ( (cid:96) ) ( := β ( (cid:96) ) ( G, w , v ) ) be the fraction of unverifiednonzero variable nodes at the beginning of iteration (cid:96) , i.e., the fraction of variable to check node messagespassed along the edges of the chosen weighted graph with unverified status (sent by nonzero variablenodes); further, let E [ β ( (cid:96) ) ] denote the expected value of β ( (cid:96) ) , where the expectation is taken over theensembles G nf ( d v , d c ) and V ng ( α ) . Now, consider the corresponding cycle-free case, and let α ( (cid:96) ) be theexpected number of messages with unverified status passed along an edge emanating from a nonzerovariable node with a tree-like neighborhood of depth at least (cid:96) at the (cid:96) th iteration. Here, again, theexpectation is taken over the ensembles W m × nf and V ng ( α ) .In the subsequent subsection, we will show how α ( (cid:96) ) can be calculated. It should be clear that α ( (cid:96) ) , beingdefined as the “average” over the ensemble of weights and input vectors, is the same as the “probability”that a message from a non-zero variable node with a tree-like neighborhood of depth at least (cid:96) , at the (cid:96) th iteration, carries an unverified status. In this section, we use the interpretation of α ( (cid:96) ) as an average.The interpretation of α ( (cid:96) ) as a probability will be used in the analysis section. In the following, we willshow that over all realizations, with high probability, β ( (cid:96) ) does not deviate much from E [ β ( (cid:96) ) ] , and E [ β ( (cid:96) ) ] ,itself, is not far from α ( (cid:96) ) , as n tends to infinity. Theorem 2.
Over the probability space of all weighted graphs G nf ( d v , d c ) , and all signal inputs V ng ( α ) ,for a fixed (cid:96) , letting β ( (cid:96) ) and α ( (cid:96) ) be defined as above, there exist positive constants µ ( d v , d c , (cid:96) ) and γ ( d v , d c , (cid:96) ) , such that (i) for any (cid:15) > , Pr (cid:2)(cid:12)(cid:12) β ( (cid:96) ) − E [ β ( (cid:96) ) ] (cid:12)(cid:12) > (cid:15)/ (cid:3) ≤ e − (cid:15) n/µ , (1) and (ii) for any (cid:15) > , and n > γ/(cid:15) , (cid:12)(cid:12) E [ β ( (cid:96) ) ] − α ( (cid:96) ) (cid:12)(cid:12) < (cid:15)/ . (2)Note that combining (1) and (2), the following holds: for any (cid:15) > , and n > γ/(cid:15) , Pr (cid:2)(cid:12)(cid:12) β ( (cid:96) ) − α ( (cid:96) ) (cid:12)(cid:12) > (cid:15) (cid:3) ≤ e − (cid:15) n/µ . Hereafter, for ease of notation, we drop the superscript (cid:96) , as we are studying the parameters of interestat a fixed iteration (cid:96) . Proof:
We start by proving (1). Let the triple T := ( G, w , v ) represent one particular realization G , w , and v of the ensembles of graphs, edge weights and input vectors, respectively. For any i , ≤ i ≤ (2 d v + 1) n := ξ , the i th element of T , with an arbitrary order which is fixed during the discussion, isreferred to as the i th coordinate of T . Consider two arbitrary realizations T (cid:48) and T (cid:48)(cid:48) . Let the symbol “ ≡ i ,”for all ≤ i ≤ ξ , be a binary relation as follows: T (cid:48) ≡ i T (cid:48)(cid:48) implies T (cid:48) ≡ i − T (cid:48)(cid:48) , where T (cid:48) ≡ i T (cid:48)(cid:48) if the first i coordinates of the triples T (cid:48) and T (cid:48)(cid:48) are the same. Suppose that we first expose the nd v edges ofthe graph one at a time. In the next nd v steps, we proceed with exposing the nd v edge weights one byone, and finally, we expose the n variable nodes’ input values one at a time. Then, T (cid:48) ≡ i T (cid:48)(cid:48) if and onlyif the information that we reveal in the first i steps of exposure for both triples is the same.We construct a martingale sequence β , β , ..., β ξ , by defining β i ( T ) = E [ β ( T (cid:48) ) | T (cid:48) ≡ i T ] . ( β is a constant, i.e., the expected value of β ( T (cid:48) ) over all graphs G (cid:48) , all edge weights w (cid:48) , and all inputvectors v (cid:48) ; β ξ is β ( T ) itself.) By the application of Azuma’s inequality (see [36, Chapter 7]), one cangive an upper bound on Pr [ | β ( T ) − E [ β ( T (cid:48) )] | > (cid:15)/
2] = Pr [ | β ξ − β | > (cid:15)/ , so long as for all ≤ i < ξ , | β i +1 − β i | ≤ ∆ i , for some bounded ∆ i , as n tends to infinity.In the following, we find ∆ i , for all ≤ i < ξ , for the Genie algorithm, and by following similar steps,one can find ∆ i for the other VB algorithms. We will explain the details of the proof for Genie, and theproofs for the other algorithms will not be presented as the method of the proofs is quite similar.First, consider the steps where we expose the edges of the graph, i.e., for all ≤ i < nd v . We want toupper bound | β i +1 ( T ) − β i ( T ) | . Let G ( G, i ) be the set of graphs in the ensemble G n ( d v , d c ) , such that their first i edges are the same as theedges in G , and let G j ( G, i ) be the subset of G ( G, i ) whose ( i + 1) th edge from the perspective of variablenodes is the j th edge from the perspective of check nodes. It should be clear that G ( G, i ) = ∪ j ∈ J i G j ( G, i ) ,where J i is the set of indices of those edges from the perspective of check nodes that have not been exposedbefore revealing the ( i + 1) th edge from the perspective of variable nodes. Thus, by definition, β i ( T ) = E [ β ( T (cid:48) ) | G (cid:48) ∈ G ( G, i )]= (cid:88) j ∈ J i E [ β ( T (cid:48) ) | G (cid:48) ∈ G j ( G, i )] · Pr [ G (cid:48) ∈ G j ( G, i ) | G (cid:48) ∈ G ( G, i )] , (3)and β i +1 ( T ) = E [ β ( T (cid:48) ) | G (cid:48) ∈ G j ( G, i )] , An example: If, for i = nd v , we have T (cid:48) ≡ i T (cid:48)(cid:48) , then the two realizations T (cid:48) and T (cid:48)(cid:48) have the same graph, but may have differentedge weights and different input vectors. for some j ∈ J i . Since (cid:80) j ∈ J i Pr[ G (cid:48) ∈ G j ( G, i ) | G (cid:48) ∈ G ( G, i )] = 1 , by using (3), one can show that β i ( T ) ≥ min j ∈ J i E [ β ( T (cid:48) ) | G (cid:48) ∈ G j ( G, i )]= E [ β ( T (cid:48) ) | G (cid:48) ∈ G k ( G, i )] , (4)for some k ∈ J i . Therefore, | β i +1 ( T ) − β i ( T ) |≤ max j ∈ J i | E [ β ( T (cid:48) ) | G (cid:48) ∈ G j ( G, i )] − β i ( T ) | (5) ≤ max j,k ∈ J i | E [ β ( T (cid:48) ) | G (cid:48) ∈ G j ( G, i )] − E [ β ( T (cid:48) ) | G (cid:48) ∈ G k ( G, i )] | . (6)Then, for all ≤ j, k ≤ md c , we need to bound the right hand side of the above inequality.Let φ j,k : G j ( G, i ) → G k ( G, i ) be a map such that for any given graph H ∈ G j ( G, i ) , the graph H (cid:48) := φ j,k ( H ) is the same as H , except in one pair of edges, i.e., if the j th and the k th edges in H from theperspective of check nodes are the ( i + 1) th and the i (cid:48) th edges from the perspective of variable nodes,respectively, then, these two edges will be the i (cid:48) th and the ( i + 1) th edges in H (cid:48) from the perspectiveof variable nodes, respectively. By construction, φ j,k is a bijection, and it preserves probabilities. Thus, E [ β ( G (cid:48) , w (cid:48) , v (cid:48) ) | G (cid:48) ∈ G k ( G, i )] = E [ β ( φ j,k ( G (cid:48) ) , w (cid:48) , v (cid:48) ) | G (cid:48) ∈ G j ( G, i )] . Further, since the graphs H and H (cid:48) defined as above are different only in one pair of edges, we will show later that for each VB algorithmof interest, regardless of the choices of w and v , the difference between β ( H, w , v ) and β ( H (cid:48) , w , v ) isbounded from above. To be precise, in the case of the Genie algorithm, for all edge weights w , and forall inputs v , we can write | β ( H, w , v ) − β ( φ j,k ( H ) , w , v ) | ≤ d (cid:96)v /n. (7)The proof of inequality (7), and similar results for the cases of LM and SBB algorithms are given inAppendix IV-B. By (7), any pair β ( H, w , v ) and β ( φ j,k ( H ) , w , v ) has a bounded difference and thus,for any j, k ∈ J i , | E [ β ( T (cid:48) ) | G (cid:48) ∈ G j ( G, i )] − E [ β ( T (cid:48) ) | G (cid:48) ∈ G k ( G, i )] | = (cid:12)(cid:12)(cid:12) E [ β (cid:48) | G (cid:48) ∈ G j ( G, i )] − E [ β (cid:48) φ j,k | G (cid:48) ∈ G j ( G, i )] (cid:12)(cid:12)(cid:12) = (cid:12)(cid:12)(cid:12) E [ β (cid:48) − β (cid:48) φ j,k | G (cid:48) ∈ G j ( G, i )] (cid:12)(cid:12)(cid:12) ≤ E [ | β (cid:48) − β (cid:48) φ j,k || G (cid:48) ∈ G j ( G, i )] ≤ max G (cid:48) , w (cid:48) , v (cid:48) | β ( G (cid:48) , w (cid:48) , v (cid:48) ) − β ( φ j,k ( G (cid:48) ) , w (cid:48) , v (cid:48) ) |≤ d (cid:96)v /n, (8) Arguments similar to what was used in upper bounding the difference between β ( T ) and β ( T (cid:48) ) , when G and G (cid:48) are different in oneand only one pair of edges, will be used to give an upper bound on β ( T ) and β ( T (cid:48) ) , when ( G, w ) = ( G (cid:48) , w (cid:48) ) , but v and v (cid:48) are differentfor one and only one variable node. Such a bound will be used later on in the proof. where β (cid:48) := β ( G (cid:48) , w (cid:48) , v (cid:48) ) , and β (cid:48) φ j,k := β ( φ j,k ( G (cid:48) ) , w (cid:48) , v (cid:48) ) . Thus, combining (6) and (8), for all ≤ i
In the Genie algorithm, the support set is known. Therefore, the set of all variable nodes can be dividedinto two disjoint sets: verified R and unverified K . At iteration zero, variable nodes in the support set areunverified, and the zero-valued variable nodes (variable nodes not in the support set) belong to the verifiedset (with a verified value equal to zero). In future iterations, during the verification process, variable nodesare removed from the set K and added to the set R . We use the notation K ( (cid:96) ) and R ( (cid:96) ) to denote the setof unverified and verified variable nodes at (the beginning of) iteration (cid:96) , respectively. We also use thesuperscript (cid:96) to indicate the iteration number for all the other sets in this section in the same way.Each iteration of the Genie algorithm consists of only one round (two half-rounds). At a generic iteration (cid:96) , in the first half-round, check nodes process the received messages from variable nodes sent at iteration (cid:96) − and generate outgoing messages to be delivered to variable nodes. As we shall see, check nodesare grouped based on their degree also reflected in their outgoing messages. A check node with degree j before half-round 1 may have degree i ≤ j after half-round 1. The set of check nodes with degree i after the (processing in the) first half-round of iteration (cid:96) , is represented with N ( (cid:96), i . In the second half-round, the variable nodes process the incoming messages and generate outgoing messages accordingly.Variable nodes, are also grouped based on the number of neighboring check nodes of degree . The group of unverified variable nodes with i neighboring check nodes of degree after the (processing in the)second half-round of iteration (cid:96) is represented with K ( (cid:96), i . Note that the grouping of check nodes remainsunchanged during the second half-round of the same iteration. In a similar way, the grouping of variablenodes remains unchanged during the first half-round of the next iteration.In the first half-round of iteration zero, every check node sends its corresponding measurement value alongwith its degree, d c . In the second half-round, variable nodes in R (0) return a verified message with a valueequal to , while variable nodes in K (0) return a message with the status bit indicating their unverifiedstatus. At iteration , the set K (0 , includes all unverified variable nodes K (0) . Therefore, at this stage,no additional variable node can be verified because all incoming messages to unverified variable nodeshave d c in their first coordinate.In the first half-round of iteration 1, received messages from variable nodes are processed at checknodes and outgoing messages to variable nodes are generated. Each such message has the following twoproperties: 1) the second coordinate of the message is the same as the second coordinate of the messagesent over the same edge in the same half-round of iteration 0, and 2) the first coordinate of the messageis at most the same as the first coordinate of the message sent over the same edge in the same half-roundof iteration 0. The second coordinates are the same because no variable node from the support set wasverified at iteration 0. The first coordinates may be different because the variable nodes not in the supportset ( R (0) ) have been revealed thus reducing the number of unverified variable nodes connected to thecheck nodes. We use the notation N (1) i ↓ j to refer to the set of check nodes N (0 , i that are categorized in N (1 , j . The arrow points downward to emphasize that j ≤ i . Note that for iteration , i = d c .In the second half-round of iteration 1 and after receiving the messages from check nodes, variable nodesin K (0 , should be regrouped in K (1 , j , ≤ j ≤ d v . We denote by K (1) i ↑ j the set of variable nodes in K (0 , i joining the set K (1 , j . In this case, j ≥ i , hence the use of the arrow pointing up. Based on the verificationrule for the Genie, at any iteration (cid:96) if an unverified variable node is neighbor to at least one check nodein the set N ( (cid:96), , it will be verified. So, variable nodes in the set (cid:83) d v j =1 K (1 , j are verified by the end ofiteration 1. Therefore, the new sets R (2) and K (2) to be used at iteration 2 are calculated as follows. R (2) = R (1) ∪ (cid:110) K (1 , , K (1 , , · · · , K (1 , d v (cid:111) , K (2) = K (1 , . The process of sending, receiving, and processing messages between check nodes and variable nodes aswell as the verification process continues in next iterations in the same fashion as we discussed above. Insummary, in a generic iteration (cid:96) , we have the following relationships: N ( (cid:96) − , i = i (cid:88) j =0 N ( (cid:96) ) i ↓ j , i = 0 , , · · · , d c , N ( (cid:96) )1 ↓ = 0 , N ( (cid:96), j = d c (cid:88) i = j N ( (cid:96) ) i ↓ j , j = 0 , , · · · , d c . R ( (cid:96) +1) = R ( (cid:96) ) ∪ (cid:110) K ( (cid:96), , K ( (cid:96), , · · · , K ( (cid:96), d v (cid:111) , K ( (cid:96) +1) = K ( (cid:96), . K ( (cid:96) − , = d v (cid:88) j =0 K ( (cid:96) )0 ↑ j , K ( (cid:96), j = K ( (cid:96) )0 ↑ j , j = 0 , , · · · , d v . By tracking the set K ( (cid:96) ) with iterations, we can decide on the success or failure of the algorithm. If thesize of the set K ( (cid:96) ) shrinks to zero as (cid:96) → ∞ , then the algorithm is successful. On the other hand, ifthere exists an (cid:15) > such that |K ( (cid:96) ) | ≥ (cid:15) , ∀ (cid:96) ≥ , then the algorithm is not successful. The success orfailure of an algorithm depends on the parameters of the graph ( d v and d c ) as well as the initial size ofthe support set |K (0) | .To be able to use the concentration results discussed before and analyze the Genie in the asymptotic case,we track the probability α ( (cid:96) ) that a variable node belongs to the set K ( (cid:96) ) . Hence, we focus on a tree-likegraph with random weights and a random input signal (random support set and random nonzero values).Let p ( (cid:96), N i , ≤ i ≤ d c , denote the probability that a check node belongs to the set N ( (cid:96), i . Furthermore,let p ( (cid:96), K j , ≤ j ≤ d v , denote the probability that an unverified (nonzero) variable node belongs to the set K ( (cid:96), j . In what follows, we show the step-by-step procedure to find the update rules for probabilities p ( (cid:96), N i , p ( (cid:96), K j , and α ( (cid:96) +1) , in terms of probabilities α ( (cid:96) − , α ( (cid:96) ) , p ( (cid:96) − , N i , and p ( (cid:96) − , K j . The formulas are calculatedusing probabilistic/counting arguments. The details can be found in Appendix I.1) Find the set of probabilities p ( (cid:96), N i , i = 0 , · · · , d c from: p ( (cid:96), N i = d c (cid:88) j = i p ( (cid:96) − , N j p ( (cid:96) ) N j ↓ i , ≤ i ≤ d c , where, p ( (cid:96) ) N j ↓ i = (cid:18) jj − i (cid:19) (cid:0) A ( (cid:96) ) (cid:1) j − i (cid:0) − A ( (cid:96) ) (cid:1) i , ≤ j ≤ d c , ≤ i ≤ j,p ( (cid:96) ) N ↓ = 1 , p ( (cid:96) ) N ↓ = 0 , p ( (cid:96) ) N ↓ = 1 ,A ( (cid:96) ) = 1 − (cid:0) − p ( (cid:96) ) (cid:1) d v − , p ( (cid:96) ) = p ( (cid:96) − , N α ( (cid:96) − d c .
2) Find the set of probabilities p ( (cid:96), K j from: p ( (cid:96), K j = (cid:18) d v j (cid:19) (cid:0) p ( (cid:96) +1) (cid:1) j (cid:0) − p ( (cid:96) +1) (cid:1) d v − j , ≤ j ≤ d v .
3) Based on the set of probabilities p ( (cid:96), K i , find the probability α ( (cid:96) +1) from: α ( (cid:96) +1) = α ( (cid:96) ) (cid:32) − d v (cid:88) i =1 p ( (cid:96), K i (cid:33) = α ( (cid:96) ) p ( (cid:96), K . TABLE IIS
ETS THAT CHANGE IN EACH HALF - ROUND OF EACH ROUND AT ANY ITERATION
R1 R2HR1 HR2 HR1 HR2 N k,i → N k,j K i → K j N i,k → N j,k ∆ i → ∆ j The initial set of probabilities p (1 , N i for Genie are as follows: p (1 , N i = (cid:18) d c i (cid:19) (cid:0) α (0) (cid:1) i (cid:0) − α (0) (cid:1) d c − i , ≤ i ≤ d c . The set of probabilities p (1 , K j are calculated following step (2) by noticing that α (1) = α (0) . C. Notation and Setup for the Analysis of LM and SBB
In LM and SBB, at the beginning of any iteration (cid:96) , the set of all variable nodes can be divided into threedisjoint sets: K ( (cid:96) ) , R ( (cid:96) ) , and ∆ ( (cid:96) ) . The set K ( (cid:96) ) consists of all unverified nonzero variable nodes, whilethe set ∆ ( (cid:96) ) consists of all unverified zero-valued variable nodes. The set R ( (cid:96) ) includes all variable nodesrecovered up to iteration (cid:96) . Clearly, the decoder can not make the distinction between variable nodes inthe sets K ( (cid:96) ) and ∆ ( (cid:96) ) . The distinction between the two sets, however, is very useful in the analysis.Furthermore, at any iteration (cid:96) , we partition the set of all check nodes into subsets N ( (cid:96) ) i,j . The index i indicates the number of neighboring variable nodes in the set K ( (cid:96) ) while the index j indicates the numberof neighboring variable nodes in the set ∆ ( (cid:96) ) . Note that: 1) the degree of each check node in the subgraphinduced by unverified variable nodes at iteration (cid:96) (reflected in the outgoing message of the check node),is i + j , and 2) the second coordinate of messages received by a variable node in the support set from asubset of check nodes all in the sets N ( (cid:96) )1 ,j , ≤ j ≤ d c − , is the same.In algorithms LM and SBB, each iteration consists of two rounds, each with two half-rounds. Theconfiguration of the sets at the end of each half-round (1 or 2), each round (R1 or R2), and each iteration( (cid:96) ), is specified using the following 4 superscripts: ( (cid:96), R , , ( (cid:96), R , , ( (cid:96), R , , and ( (cid:96), R , . In thefirst half-rounds (any round and any iteration), messages are passed from check nodes to variable nodes,while in the second half-rounds, messages are passed from variable nodes to check nodes. Also, with thedefinition of mapping functions Φ (1 ,(cid:96) ) v and Φ (2 ,(cid:96) ) v , the set of verified variable nodes in the first and thesecond rounds belong to the sets K ( (cid:96) ) and ∆ ( (cid:96) ) , respectively. We have summarized in Table II the setsthat are affected in each half-round (HR) of each round (R) at any iteration.In the Genie algorithm, the set K ( (cid:96) ) i represents the set of unverified variable nodes in the support set thathave i neighboring check nodes of degree . The definition of this set in the LM and SBB algorithms is different, as explained in the following. The set K ( (cid:96) ) i in the LM algorithm represents the set of unverifiedvariable nodes in the support set with i neighboring check nodes in the set N ( (cid:96) )1 , . To define this set inthe SBB algorithm, let N ( (cid:96) ) i := (cid:83) d c − ij =0 N ( (cid:96),R , i,j . With this notation, the set K ( (cid:96) ) i in the SBB algorithm isdefined as the set of unverified variable nodes in the support set with i neighboring check nodes in theset N ( (cid:96) )1 .In order to track the evolution of the unverified support set, it is imperative to characterize the set ofvariable nodes in the support set that are recovered in each iteration. In Theorems 3 and 4 below, wecharacterize the verification of unverified nonzero variable nodes in the set K ( (cid:96) ) in each iteration (cid:96) for thetwo algorithms LM and SBB. The theorems are proved in Appendices II and III. Theorem 3.
In the first round of any iteration (cid:96) in the LM algorithm, a nonzero variable node v ∈ K ( (cid:96) ) is verified if and only if it belongs to the set (cid:83) d v i =1 K ( (cid:96),R , i . Theorem 4.
In the first round of any iteration (cid:96) in the SBB algorithm, a nonzero variable node v ∈ K ( (cid:96) ) is verified if and only if it belongs to the set (cid:83) d v i =2 K ( (cid:96),R , i ∪ ˆ K ( (cid:96),R , , where the set ˆ K ( (cid:96),R , consists ofall variable nodes in the set K ( (cid:96),R , connected to the set N ( (cid:96),R , , . In the LM and SBB algorithms, unverified variable nodes with zero values are verified in R2. Note thata check node is zero-valued if it belongs to the set N ( (cid:96),R , ,j , ≤ j ≤ d c . Therefore, for the verificationof zero-valued variable nodes in the second round of iteration (cid:96) , we divide the set of variable nodes in ∆ ( (cid:96) ) into subsets ∆ ( (cid:96) ) i , ≤ i ≤ d v with the following definition: a variable node in the set ∆ ( (cid:96) ) i has i neighboring check nodes in the set (cid:110)(cid:83) d c j =1 N ( (cid:96),R , ,j \ (cid:83) d c j =1 N ( (cid:96),R , ,j (cid:111) , i.e., the set of all check nodes whichbecame zero-valued after HR1 of R2. In Theorem 5 below, we characterize the verification of unverifiedzero-valued variable nodes in the set ∆ ( (cid:96) ) at HR2-R2 in each iteration (cid:96) of LM and SBB algorithms. Theorem 5.
In the second half-round of the second round of any iteration (cid:96) in the LM and SBB algorithmsa zero-valued variable node v ∈ ∆ ( (cid:96) ) is verified if and only if it belongs to the set (cid:83) d v i =1 ∆ ( (cid:96) ) i . We denote by N ( (cid:96),R k,i ↓ j the set of check nodes that are moved from N ( (cid:96) − ,R , k,i to N ( (cid:96),R , k,j in HR1-R1of iteration (cid:96) . Similarly, the set of check nodes that are moved from N ( (cid:96),R , i,k to N ( (cid:96),R , j,k in HR1-R2 ofiteration (cid:96) are denoted by N ( (cid:96),R i ↓ j,k . Since variable nodes in K and ∆ are verified through iterations, wealways have j ≤ i and hence the use of notation i ↓ j . Moreover, we denote the set of variable nodesthat are moved from K ( (cid:96) − ,R , i to K ( (cid:96),R , j and from ∆ ( (cid:96) − ,R , i to ∆ ( (cid:96),R , j in HR2-R1 and HR2-R2 ofiteration (cid:96) by K ( (cid:96),R i ↑ j and ∆ ( (cid:96),R i ↑ j , respectively. The notation i ↑ j implies that j ≥ i .The sets that fully describe the state of the decoder at the beginning of iteration (cid:96) are: K ( (cid:96) ) , R ( (cid:96) ) , ∆ ( (cid:96) ) , N ( (cid:96) − ,R , i,j , K ( (cid:96) − ,R , i , and ∆ ( (cid:96) − ,R , i . For the analysis, we track the probability that a node (variablenode or check node) belongs to a certain set at each half-round, round, or iteration. We use the notation α ( (cid:96) ) to denote the probability that a variable node belongs to the set K ( (cid:96) ) . For the rest of the sets, we use the standard notation of probabilities that was applied in the analysis of the Genie algorithm. For instance,we denote the probability that a check node belongs to the set N ( (cid:96),R , i,j by p ( (cid:96),R , N i,j .In the analysis, the goal is to find the recursive equations that relate these probabilities for consecutiveiterations. As we shall see, the analysis of the decoding process for LM and SBB results in a system ofcoupled recursive update rules. Moreover, we show that the update rules at iteration (cid:96) are functions ofprobabilities at iteration (cid:96) − . Hence, the complexity of the analysis scales linearly with the number ofiterations. In the following two sections, we present the update rules for the LM and SBB algorithms.The derivation of formulas are discussed in detail in Appendices II and III. D. Update Rules for the LM Algorithm
In one iteration (cid:96) ≥ of the analysis of the LM algorithm, the following update rules are applied:1) Find the set of probabilities p ( (cid:96),R , N i,j , ≤ i ≤ d c , ≤ j ≤ d c − i , from: p ( (cid:96),R , N i,j = d c − i (cid:88) k = j p ( (cid:96) − ,R , N i,k p ( (cid:96),R N i,k ↓ j , where, p ( (cid:96),R N i,k ↓ j = (cid:18) kj (cid:19) (cid:0) A ( (cid:96) ) (cid:1) j (cid:0) − A ( (cid:96) ) (cid:1) k − j , A ( (cid:96) ) = p ( (cid:96) )∆ − D ( (cid:96) − , (17) D ( (cid:96) − = d c − (cid:88) j =1 j p ( (cid:96) − ,R , N ,j d c (cid:88) i =0 d c − (cid:88) j =1 jp ( (cid:96) − ,R , N i,j , (18)and p ( (cid:96) )∆ is given in (19).2) Find the set of probabilities p ( (cid:96),R , K i , ≤ i ≤ d v , from: p ( (cid:96),R , K i = (cid:18) d v i (cid:19) (cid:0) B ( (cid:96) ) (cid:1) i (cid:0) − B ( (cid:96) ) (cid:1) d v − i , where, B ( (cid:96) ) = p ( (cid:96),R , N , α ( (cid:96) ) d c .
3) Find the updated probability that a variable node is in the support set and is not verified from: α ( (cid:96) +1) = α ( (cid:96) ) (cid:32) − d v (cid:88) i =1 p ( (cid:96),R , K i (cid:33) .
4) Find the set of probabilities p ( (cid:96),R , N j,k , ≤ j ≤ d c , ≤ k ≤ d c − i , from: p ( (cid:96),R , N j,k = d c (cid:88) i = j p ( (cid:96),R , N i,k p ( (cid:96),R N i ↓ j,k , where, p ( (cid:96),R N i ↓ j,k = (cid:18) ij (cid:19) (cid:0) C ( (cid:96) ) (cid:1) j (cid:0) − C ( (cid:96) ) (cid:1) i − j , C ( (cid:96) ) = p ( (cid:96),R , K − B ( (cid:96) ) . Note that: p ( (cid:96),R N ↓ , = 1 , p ( (cid:96),R N ↓ , = 0 .
5) Find the set of probabilities p ( (cid:96),R , i , ≤ i ≤ d v , from: p ( (cid:96),R , i = (cid:18) d v i (cid:19) (cid:0) D ( (cid:96) ) (cid:1) i (cid:0) − D ( (cid:96) ) (cid:1) d v − i , where D ( (cid:96) ) is calculated according to (18).6) And lastly, find the probability that a variable node is zero-valued and unverified at iteration (cid:96) , from: p ( (cid:96) +1)∆ = p ( (cid:96) )∆ p ( (cid:96),R ,aHR . (19)We have the following initial probabilities, by letting α := α (0) denote the initial density factor. p (1)∆ = (1 − α ) (cid:16) − (1 − α ) d c − (cid:17) d v , D (0) = (1 − α ) d c − ⇒ A (1) = (1 − α ) (cid:16) − (1 − α ) d c − (cid:17) d v − . Since all check nodes have degree d c in the subgraph induced by the unverified variable nodes at iterationzero, we have: p (0 ,R , N i,dc − i = (cid:18) d c i (cid:19) ( α ) i (1 − α ) d c − i , i = 0 , · · · , d c , and thus p (1 ,R , N i,j = p (0 ,R , N i,dc − i p (1 ,R N i,dc − i ↓ j , i = 1 , · · · , d v , j = 0 , · · · , d c − i, where, p (1 ,R N i,dc − i ↓ j = (cid:18) d c − ij (cid:19) (cid:0) A (1) (cid:1) j (cid:0) − A (1) (cid:1) d c − i − j , i = 1 , · · · , d v , j = 0 , · · · , d c − i. Since no element of the support set is verified at iteration zero, we also have: α (1) = α (0) = α. E. Update Rules for the SBB Algorithm
In one generic iteration (cid:96) , (cid:96) ≥ , of the SBB algorithm, the following update rules are applied in theanalysis:
1) Find the set of probabilities p ( (cid:96),R , , +) N ,j , p ( (cid:96),R , ,C ) N ,j , and p ( (cid:96),R , N i,j , ≤ i ≤ d c , ≤ j ≤ d c − i , from: p ( (cid:96),R , , +) N ,j = d c − (cid:88) k = j p ( (cid:96) − ,R , , +) N ,k p ( (cid:96),R N ,k ↓ j ,p ( (cid:96),R , ,C ) N ,j = d c − (cid:88) k = j p ( (cid:96) − ,R , ,C ) N ,k p ( (cid:96),R N ,k ↓ j ,p ( (cid:96),R , N i,j = d c − i (cid:88) k = j p ( (cid:96) − ,R , N i,k p ( (cid:96),R N i,k ↓ j , where, p ( (cid:96),R N i,k ↓ j , and parameters A ( (cid:96) ) and D ( (cid:96) − therein, are obtained from equations (17) and (18).2) Find the set of probabilities p ( (cid:96),R , K , p ( (cid:96),R , , +) K , p ( (cid:96),R , ,C ) K , and p ( (cid:96),R , K j , ≤ j ≤ d v , from: p ( (cid:96),R , K = 1 N ( (cid:96),R p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ ,p ( (cid:96),R , , +) K = 1 N ( (cid:96),R p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ (cid:0) − f ( (cid:96),R , +) (cid:1) ,p ( (cid:96),R , ,C ) K = 1 N ( (cid:96),R p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ (cid:0) − f ( (cid:96),R ,C ) (cid:1) ,p ( (cid:96),R , K = p ( (cid:96),R , , +) K + p ( (cid:96),R , ,C ) K ,p ( (cid:96),R , K j = 0 , j = 2 , · · · , d v , where, p ( (cid:96),R K j ↑ i = (cid:18) d v − ji − j (cid:19) (cid:0) B ( (cid:96) ) (cid:1) i − j (cid:0) − B ( (cid:96) ) (cid:1) d v − i , B ( (cid:96) ) = d c − (cid:88) j =0 p ( (cid:96),R , , +) N ,j d c − (cid:88) j =0 p ( (cid:96),R , , +) N ,j + d c (cid:88) i =2 d c − i (cid:88) j =0 ip ( (cid:96),R , N i,j , and, f ( (cid:96),R , +) = p ( (cid:96),R , , +) N , d c − (cid:88) j =0 p ( (cid:96),R , , +) N ,j , f ( (cid:96),R ,C ) = p ( (cid:96),R , ,C ) N , d c − (cid:88) j =0 p ( (cid:96),R , ,C ) N ,j , and, N ( (cid:96),R = p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ + p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ (cid:0) − f ( (cid:96),R , +) (cid:1) + p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ (cid:0) − f ( (cid:96),R ,C ) (cid:1) .
3) The probability that a variable node belongs to the support set and remains unverified after iteration (cid:96) , α ( (cid:96) +1) is calculated as follows: α ( (cid:96) +1) = α ( (cid:96) ) N ( (cid:96),R .
4) Find the set of probabilities p ( (cid:96),R , N ,j , p ( (cid:96),R , , +) N ,j , p ( (cid:96),R , ,C ) N ,j , and p ( (cid:96),R , N k,j , ≤ k ≤ d c , ≤ j ≤ d c − k , from: p ( (cid:96),R , N ,j = p ( (cid:96),R , N ,j + p ( (cid:96),R ,C,O ) N ,j + p ( (cid:96),R , + ,O ) N ,j + d c (cid:88) i =2 p ( (cid:96),R , N i,j p ( (cid:96),R N i ↓ ,j ,p ( (cid:96),R , , +) N ,j = d c (cid:88) i =2 p ( (cid:96),R , N i,j p ( (cid:96),R N i ↓ ,j ,p ( (cid:96),R , ,C ) N ,j = p ( (cid:96),R ,C,F ) N ,j + p ( (cid:96),R , + ,F ) N ,j ,p ( (cid:96),R , N k,j = d c (cid:88) i = k p ( (cid:96),R , N i,j p ( (cid:96),R N i ↓ k,j , k = 2 , · · · , d c , where, p ( (cid:96),R , + ,F ) N ,i = p ( (cid:96),R , , +) N ,i p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ d v (cid:88) j =1 jp ( (cid:96) − ,R , K p ( (cid:96),R K ↑ j + d v (cid:88) j =2 ( j − p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ j ,p ( (cid:96),R , + ,O ) N ,i = p ( (cid:96),R , , +) N ,i − p ( (cid:96),R , + ,F ) N ,i ,p ( (cid:96),R ,C,F ) N ,i = p ( (cid:96),R , ,C ) N ,i p ( (cid:96),R K ↑ d v (cid:88) j =1 p ( (cid:96),R K ↑ j ,p ( (cid:96),R ,C,O ) N ,i = p ( (cid:96),R , ,C ) N ,i − p ( (cid:96),R ,C,F ) N ,i ,p ( (cid:96),R N i ↓ k,j = (cid:18) ik (cid:19) (cid:0) C ( (cid:96) ) (cid:1) k (cid:0) − C ( (cid:96) ) (cid:1) i − k , p ( (cid:96),R = d c − (cid:88) j =0 p ( (cid:96),R , , +) N ,j + d c − (cid:88) j =0 p ( (cid:96),R , ,C ) N ,j α ( (cid:96) ) d c ,C ( (cid:96) ) = p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ − p ( (cid:96),R + p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ (cid:18) d v − d v (cid:19) − p ( (cid:96),R (cid:0) − f ( (cid:96),R , +) (cid:1) + p ( (cid:96) − ,R , K p ( (cid:96),R K ↑ (cid:18) d v − d v (cid:19) − p ( (cid:96),R (cid:0) − f ( (cid:96),R ,C ) (cid:1) .
5) Find the set of probabilities p ( (cid:96),R , i , ≤ i ≤ d v , from: p ( (cid:96),R , i = (cid:18) d v i (cid:19) (cid:0) D ( (cid:96) ) (cid:1) i (cid:0) − D ( (cid:96) ) (cid:1) d v − i , where D ( (cid:96) ) is calculated according to (18).6) Find the probability that a variable node is zero-valued and remains unverified in iteration (cid:96) + 1 as follows: p ( (cid:96) +1)∆ = p ( (cid:96) )∆ p ( (cid:96),R , . The initial conditions in the SBB algorithm are for iterations 0 and 1, and are given below. The updaterules for iteration serve as the initial conditions for iteration . • HR1-R1: p (1 ,R , N i,j = (cid:18) d c i (cid:19) (cid:0) α (0) (cid:1) i (cid:0) − α (0) (cid:1) d c − i p (1 ,R N i,dc − i ↓ j , i = 1 , · · · , d c , j = 0 , · · · , d c − i, where, p (1 ,R N i,dc − i ↓ j = (cid:18) d c − ij (cid:19) (cid:0) A (1) (cid:1) j (cid:0) − A (1) (cid:1) d c − i − j , A (1) = (1 − α (0) ) (cid:16) − (cid:0) − α (0) (cid:1) d c − (cid:17) d v − . • HR2-R1 (Regrouping variable nodes after the recovery): p (1 ,R , K = 1 N (1 ,R (cid:0) − B (1) (cid:1) d v .p (1 ,R , K = d v N (1 ,R (cid:0) − f (1 ,R (cid:1) B (1) (cid:0) − B (1) (cid:1) d v − .p (1 ,R , K i = 0 , ≤ i ≤ d v , where, B (1) = d c − (cid:88) j =0 p (1 ,R , N ,j α (0) d c , f (1 ,R = p (1 ,R , N , d c − (cid:88) j =0 p (1 ,R , N ,j , and, N (1 ,R = (cid:0) − B (1) (cid:1) d v + d v (cid:0) − f (1 ,R (cid:1) B (1) (cid:0) − B (1) (cid:1) d v − . • HR2-R1 (Recovering variable nodes based on ECN): α (2) = α (0) N (1 ,R (cid:16) p (1 ,R , K + p (1 ,R , K (cid:17) , • HR1-R2: p (1 ,R , N k,j = d c (cid:88) i = k p (1 ,R , N i,j p (1 ,R N i ↓ k,j , k = 2 , · · · , d c , j = 0 , · · · , d c − i,p (1 ,R , , +) N ,j = d c (cid:88) i =2 p (1 ,R , N i,j p (1 ,R N i ↓ ,j ,p (1 ,R , ,C ) N ,j = p (1 ,R , N ,j p (1 ,R N ↓ ,j , where, p (1 ,R N ↓ ,j = 1 , ≤ j ≤ d c , p (1 ,R N ↓ , = 1 , p (1 ,R N ↓ , = 0 ,p (1 ,R N ↓ ,j = 1 − (cid:0) − B (1) (cid:1) d v − , p (1 ,R N ↓ ,j = (cid:0) − B (1) (cid:1) d v − , ≤ j ≤ d c − ,p (1 ,R N i ↓ k,j = (cid:18) ik (cid:19) (cid:0) C (1) (cid:1) k (cid:0) − C (1) (cid:1) i − k , ≤ i ≤ d c , ≤ k ≤ i, ≤ j ≤ d c − i,C (1) = (cid:0) − B (1) (cid:1) d v − + ( d v − B (1) (cid:0) − B (1) (cid:1) d v − (cid:0) − f (1 ,R (cid:1) . • HR2-R2: p (2)∆ = (1 − α (0) ) (cid:16) − (cid:0) − α (0) (cid:1) d c − (cid:17) d v (cid:0) − D (1) (cid:1) d v , where, D (1) is obtained by (18). VI. N OISY M EASUREMENTS
We adopt the following model for the case where the measurements are noisy [37]: c = Gv + n . In this new model, v and G are the original signal and the sensing matrix, respectively. The new term, n ,represents the noise vector added to the noiseless measurements Gv , resulting in the noisy measurementvector c . Elements of the noise vector are assumed to be i.i.d. Gaussian random variables with mean and variance σ . The addition of noise to the measurements results in the following two probabilities tobe zero: 1) the probability of having a zero measurement, and 2) the probability of having two equalmeasurements. This will disable the ZCN and ECN rules in recovering the signal elements. Without theZCN and ECN rules, zero-valued variable nodes are not verified, and consequently, no check node willhave a reduced degree in the subgraph induced by unverified variable nodes. Therefore, the D1CN rulewill also be ineffective.In the context of message-passing algorithms, there are generally two approaches to deal with noisymeasurements. In the first approach, the original formulation of the problem is changed so that the noiseis taken into consideration [25], [26]. In the other approach, the algorithms are changed in order to copewith the presence of noise in the measurements [3]. Although the first approach may result in lowerreconstruction noise, it requires unbounded message size and is susceptible to approximation errors. Theauthors in [3] instead equipped their VB algorithm with some thresholding techniques and proved that ifthe original signal is sparse enough, they are able to recover the location and the sign of the non-zerosignal elements successfully. In what follows, we propose a similar thresholding technique to deal withthe noisy measurements.Thresholding is a common technique in detection theory to deal with noisy measurements [38]. We applythis technique to VB algorithms by defining two thresholds (cid:15) and (cid:15) . We use (cid:15) to convert small noisy measurements to zero; i.e., any measurement c , such that | c | < (cid:15) , is set to zero. We use (cid:15) as theacceptable tolerance for the equality of two noisy measurements; i.e., we consider two measurements c and c equal if | c − c | < (cid:15) . In this case, we assign c and c a new common value equal to ( c + c ) / .While the scope of this work is not to optimize thresholds (cid:15) and (cid:15) , our goal is to demonstrate thepotential of thresholding in desensitizing the VB algorithms to the measurement noise. We explain thisthrough an example and by comparing the performance of the SBB algorithm equipped with thresholdingand two methods based on (cid:96) minimization in the case where the measurements are noisy.Consider a signal of length n = 1000 . We let the size of the support set, k , to increase from to in steps of . For each such support size k , we pick k out of the n elements randomly, and assign toeach element an even integer uniformly distributed in the range [ − , , independent of the valueof the other nonzero elements. In the case of the SBB algorithm, the signal is measured through a (3 , unweighted bigraph. In the case of (cid:96) -based algorithms, we use sensing matrices consisting of orthonormalcolumns with Standard Gaussian elements [37]. In all cases, the number of measurements, m , is fixedat . Each measurement is independently contaminated with a Gaussian noise of mean and varianceequal to σ = 0 . .For the SBB, we set both thresholds (cid:15) and (cid:15) equal to 1.99. Since the value of nonzero signal elementsare drawn from a finite alphabet and since the graph is unweighted, false alarm may occur with nonzeroprobability in the recovery process. In our simulations, we consider a recovery algorithm “successful” ifit can fully recover the support set.The first (cid:96) -based recovery algorithm is the (cid:96) regularization method introduced in [37]. For the secondalgorithm, we empower the (cid:96) regularization algorithm with the knowledge of the size of the support set.The algorithm thus keeps the k components that are the largest in magnitude and converts the rest to zero.To simulate the two (cid:96) -based decoders, we use the L1MAGIC package available in [39].As the measure of performance, we consider the mean square error (MSE) between the original and therecovered signals. For each value of k , we perform simulations until we obtain “successful” recoveryinstances. The results for the 3 algorithms are reported in Fig. 1, where for each algorithm, MSE averagedover all simulated cases for a given value of k is shown.As can be seen, the SBB recovery algorithm significantly (by about two orders of magnitude) outperformsboth (cid:96) -based algorithms. VII. S IMULATION R ESULTS
In this section, we present simulation results obtained by running the recovery algorithms over randombiregular graphs to recover sparse signals of finite length n . We also present analytical results obtainedthrough the mathematical analysis described in Section V for the asymptotic regime when n → ∞ . This k k x − ˆ x k Modified L1Original L1SBB
Fig. 1. Comparison between SBB, (cid:96) minimization, and modified (cid:96) minimization in terms of MS reconstruction error in the presence ofGaussian measurement noise with zero mean and variance σ = 0 . (n = 1000, m=500). includes the success threshold of different VB algorithms over different biregular graphs. The comparisonof asymptotic and finite-length results shows that there is a good agreement between the two for moderatelylarge block lengths ( n ≥ ).In all simulations, a signal element belongs to the support set with probability α (0) , unless otherwise spec-ified. Also, each nonzero signal element (variable) is drawn according to a standard Gaussian distribution.The biregular graphs are constructed randomly with no parallel edges and all the edge weights are equalto one. In each set of simulations, the sensing graph is fixed and each simulation point is generated byaveraging over 1000 random instances of the input signal, unless specified otherwise. We repeated eachsimulation with different randomly generated graphs (with the same variable and check node degrees),and observed that the results were almost identical for every graph. Each simulation is run until thealgorithm makes no further progress. In this case, if the signal is recovered perfectly, the recovery iscalled successful, otherwise a failure is declared.For the analytical results, based on the fact that α ( (cid:96) ) is a non-increasing function of iteration number (cid:96) ,we consider the following stopping criteria:1) α ( (cid:96) ) ≤ − ,2) α ( (cid:96) ) > − and | α ( (cid:96) ) − α ( (cid:96) − | < − .If the analysis stops based on the first stopping criterion, the algorithm is considered successful. If, on the other hand, it stops based on the second criterion, the algorithm is considered unsuccessful and a failureis reported. To calculate the success threshold, a binary search is performed until the separation betweenthe start and the end of the search region is less than − .To motivate the use of recovery algorithms over sparse graphs, as the first simulation result, we presentthe comparison between the SBB algorithm and two benchmark (cid:96) -based algorithms, (cid:96) minimization [2]and iterative weighted (cid:96) minimization [40]. The setup is as follows. For SBB, we choose a random (3 , biregular sensing graph with 1000 variable nodes and 500 check nodes. The sensing matrix used for thetwo (cid:96) -based algorithms consists of 500 rows and 1000 columns. The elements are initially i.i.d. standardGaussian random variables. Then the rows are made orthonormal.The cost functions used in (cid:96) and weighted (cid:96) minimization algorithms are (cid:107) v (cid:107) := (cid:80) i | v i | and (cid:107) W v (cid:107) := (cid:80) i w i | v i | , respectively, where v is the original signal of interest with elements v i , and W is a diagonalmatrix with positive diagonal elements w i representing the weights. Weighted (cid:96) minimization is aniterative algorithm in which the weights at iteration t ( w ( t ) i ) are updated according to w ( t ) i = 1 / ( | v ( t − i | + (cid:15) ) ,where v ( t − i is the estimate of the signal element v i at iteration t − . The weighted (cid:96) is not verysensitive to the parameter (cid:15) as noted in [40]. We found (cid:15) = 0 . is a good choice based on our simulations.Regarding the maximum number of iterations for the weighted (cid:96) minimization algorithm, authors in [40]show that as this parameter increases, better results are achieved, with the cost of longer running time.The improvement gained by increasing the number of iterations beyond however, is negligible [40].Therefore, in our simulations, we choose a conservative maximum number of iterations equal to . As (cid:96) and weighted (cid:96) minimization algorithms output an estimate which is very close to the original signal,but not exactly the same, we declare a success for these two algorithms if the difference between everyoriginal signal element and its corresponding estimate is less than − . Lastly, we use the L1MAGICpackage in [39] as the optimization engine for simulating (cid:96) and weighted (cid:96) minimization algorithms.To have a fair comparison, the same signal vectors are used for all the algorithms. We also choose thesize of the support set deterministically, and let the size range from 10 to 300. For each support size,100 instances of the signal vector are generated. Each signal vector is then measured according to thecorresponding sensing mechanism for each class of algorithms. The success or failure of the recoveryalgorithms over the resulting measurements are then averaged over the 100 instances, and plotted in Figure2. In Figure 3 the average running time, in seconds, is plotted for the three algorithms. The algorithmswere implemented in MATLAB and were run on a computer with an AMD Phenom 9650 Quad-CoreProcessor 2.3 GHz, 3 GB RAM and a Windows 7 operating system. As can be seen, the SBB algorithmrecovers signals with more nonzero elements at a speed which is about 2 orders of magnitude fastercompared to that of the (cid:96) algorithms.For the next experiment, we apply XH, SBB and LM algorithms to four randomly constructed (5 , regular graphs with n = { , , , } × . The success ratio of the algorithms vs. the initial density factor α = α (0) are shown in Figure 4. From the figure, we can see that, for all algorithms, byincreasing n , the transition part of the curves becomes sharper such that the curves for n = 10 practicallylook like a step function. In the figure, we have also shown the success threshold of the algorithms for (5 , graphs, obtained based on the proposed analysis, by arrows. As can be seen, the thresholds matchvery well with the waterfall region of the simulation curves.
0 50 100 150 170 200 220 250 30000.10.20.30.40.50.60.70.80.91 k S u cc e ss R a t i o L1SBBWeighted L1
Fig. 2. Comparison between success ratios of (cid:96) , weighted (cid:96) and SBB algorithms for n = 1000 , m = 500 . In Table III, we have listed the analytical success thresholds of the iterative recovery algorithms for graphswith different d v and d c values. The result for XH algorithm on (3 , graphs, and more generally forgraphs with d v = 3 , is missing as the algorithm performs poorly on such graphs. For every graph, theGenie algorithm has the best performance. This is followed by SBB, LM and XH algorithms, respectively.Careful inspection of the results in Table III indicates that the oversampling ratio r o = d v /αd c , improvesconsistently by decreasing both d v and d c values. In fact, among the results presented in Table III, theapplication of the Genie and SBB to (3 , graphs results in the lowest oversampling ratio of ≈ . and ≈ . , respectively.In Table IV, we have listed the analytical success thresholds of the iterative recovery algorithms forgraphs with compression ratio d v /d c = 0 . and different d v and d c values. In general, as we decrease d v ,algorithms perform better in terms of recovery capability. This also implies that for a fixed compression The reason is that for d v = 3 , a variable node is verified with the common value of (cid:100) d v / (cid:101) = 2 check nodes. However, if two nonzerovariable nodes share the same two check nodes (a cycle of length 4 exists in the graph), then a false verification may occur. These results are consistent with the results observed for the Belief Propagation (BP) decoding of binary LDPC codes based on biregulargraphs. −3 −2 −1 k E x e c u t i on T i m e ( s e c ) L1SBBWeighted L1
Fig. 3. Comparison between the average running times of (cid:96) , weighted (cid:96) and SBB algorithms for n = 1000 , m = 500 . α S u cc e ss R a t i o n = 3,000n = 15,000n = 100,000n = 1,000,000 SBBLMXH
Fig. 4. Success ratio of XH, LM and SBB algorithms vs. α = α (0) for (5 , graphs with n = { , , and } × . Analyticalthresholds are shown by arrows. ratio, the oversampling ratio improves by decreasing d v and d c .In Tables V and VI, we have listed the number of iterations required for different recovery algorithms torecover signals with density factor equal to the success thresholds reported in Tables III and IV minus . , respectively. These results, which are obtained by the asymptotic analysis are in close agreement with finite-length simulation results at block lengths of about . These results indicate that with a fewexceptions the better performance comes at the expense of a larger number of iterations. In particular,among the practical recovery algorithms, SBB requires the largest number of iterations for convergence. TABLE IIIS
UCCESS T HRESHOLDS FOR DIFFERENT GRAPHS AND ALGORITHMS ( d v , d c ) (3 ,
4) (5 ,
6) (5 ,
7) (5 ,
8) (7 , Genie 0.6474 0.5509 0.4786 0.4224 0.4708SBB 0.4488 0.3892 0.3266 0.2806 0.3335LM 0.3440 0.2871 0.2305 0.1907 0.2385XH - 0.1846 0.1552 0.1339 0.1435
TABLE IVS
UCCESS T HRESHOLDS FOR DIFFERENT GRAPHS AND ALGORITHMS FOR FIXED COMPRESSION RATIO r c = 0 . Graphs: ( d v , d c ) (3 ,
6) (4 ,
8) (5 ,
10) (6 ,
12) (7 ,
14) (8 , Genie 0.4294 0.3834 0.3415 0.3074 0.2797 0.2568SBB 0.2574 0.2394 0.2179 0.1992 0.1835 0.1703LM 0.1702 0.1555 0.1391 0.1253 0.1140 0.1048XH - 0.1875 0.1050 0.1170 0.0791 0.0834
TABLE VN
UMBER OF ITERATIONS REQUIRED FOR DIFFERENT RECOVERY ALGORITHMS OVER DIFFERENT GRAPHS TO RECOVER A SIGNAL WITHDENSITY RATIO EQUAL TO THE SUCCESS THRESHOLD MINUS . Graphs: ( d v , d c ) (3 ,
4) (5 ,
6) (5 ,
7) (5 ,
8) (7 , Genie 106 66 66 62 55SBB 655 178 165 200 344LM 258 139 103 126 108XH - 63 58 54 41
TABLE VIN
UMBER OF ITERATIONS REQUIRED FOR DIFFERENT RECOVERY ALGORITHMS OVER DIFFERENT GRAPHS WITH FIXED COMPRESSIONRATIO r c = 0 . , TO RECOVER A SIGNAL WITH DENSITY RATIO EQUAL TO THE SUCCESS THRESHOLD MINUS . Graphs: ( d v , d c ) (3 ,
6) (4 ,
8) (5 ,
10) (6 ,
12) (7 ,
14) (8 , Genie 93 69 57 50 46 41SBB 247 167 172 163 127 108LM 142 94 136 97 55 67XH - 64 48 38 32 28To further investigate the degree of agreement between our theoretical asymptotic analysis and finite-length simulation results, we have presented in Fig. 5 the evolution of α ( (cid:96) ) with iterations (cid:96) for the fouralgorithms Genie, LM, SBB, and XH over a (5 , graph. For each algorithm, two values of α (0) areselected: one above and one below the success threshold presented in Table III. The theoretical resultsare shown by solid lines while simulations for n = 10 are presented with dotted lines. As one can see, the two sets of results are in close agreement particularly for the cases where α (0) is above the thresholdand for smaller values of (cid:96) . Iteration ( ‘ ) α ( ‘ ) SBB
Iteration ( ‘ ) α ( ‘ ) LM Iteration ( ‘ ) α ( ‘ ) XH Iteration ( ‘ ) α ( ‘ ) Genie
Theoretical Above ThresholdTheoretical Below ThresholdSimulation Below ThresholdSimulation Above Threshold
Fig. 5. Evolution of α ( (cid:96) ) vs. iteration number (cid:96) for the four recovery algorithms over a (5 , graph (finite-length simulations are for n = 10 ). Next, for different values of α (0) , we estimate the average fraction of unverified nonzero variable nodes α ( (cid:96) ) using the analysis, and denote the value of α ( (cid:96) ) at the time that the analysis stops (because oneof the stopping criteria is met) as α ( stop ) . These values are plotted vs. the corresponding values of α (0) in Fig. 6 for the four VB recovery algorithms over the (5 , sensing graphs. In the same figure, wehave also given the corresponding simulation results for two randomly selected (5 , sensing graphs with n = 10 and . The simulation results for both lengths closely match the analytical results, with thoseof n = 10 being practically identical to the analytical results. We have indicated the success thresholdof the algorithms by arrows. From the figure, it can also be seen that as α (0) increases and tends to one,the curves tend to the asymptote α ( stop ) = α (0) .As the last experiment, we compare the running time and the accuracy of the proposed asymptotic analysisagainst those of the differential equation approach presented in [23]. For comparison, a biregular (3 , graph and the SBB algorithm were chosen. The binary search for the success threshold starts with theinterval [0 . , . and ends when the separation between the start and the end of the search region inless than − . The analysis is implemented in MATLAB and executed on the same computer describedbefore. Using the proposed analysis, we obtain the success threshold of 0.2574 in 23.1 seconds. Table VIIsummarizes the results of running the analysis of [23] on the same machine for different values of n . Thereported thresholds increase with the increase in n . For n = 100 , , the running time is roughly 100times that of our proposed method. Moreover, the obtained threshold of . is only in agreement with α (0) α ( s t o p ) Genie α (0) α ( s t o p ) XH α (0) α ( s t o p ) LM α (0) α ( s t o p ) SBB
TheorySimulation n = 100,000Simulation n = 1,000,000
Fig. 6. Fraction of unrecoverable variable nodes for different recovery algorithms and different starting density factors over random (5,6)regular bipartite graphs. Each arrow represents the theoretical success threshold. the threshold of . , obtained by the proposed method, up to two decimal points. In fact, experimentssimilar to those reported in Fig. 5 reveal that the accuracy of the threshold obtained by the method of [23]is lower than our results. In particular, our simulations show that the SBB algorithm over (3 , graphswith n = 10 fails for α (0) = 0 . , which would imply that the threshold . is only accurate up totwo decimal points. TABLE VIIS
UCCESS THRESHOLD AND RUNNING TIME OF THE ANALYSIS OF [23]
FOR
SBB
OVER A RANDOM (3 , REGULAR GRAPH . n
100 1 ,
000 10 ,
000 20 ,
000 50 ,
000 100 , Success Threshold 0.2465 0.2577 0.2589 0.2590 0.2590 0.2591Running Time (seconds) 1.1 9.9 103.9 220.6 647.4 2044.1VIII. A
CKNOWLEDGMENT
The authors wish to thank H. D. Pfister for providing them with the latest version of [23]. R EFERENCES [1] D. Donoho, “Compressed sensing,”
IEEE Trans. Inform. Theory , vol. 52 (4), pp. 1289–1306, April 2006.[2] E. Cand`es, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequencyinformation,”
IEEE Trans. Inform. Theory , pp. 489–509, February 2006.[3] W. Xu and B. Hassibi, “Efficient compressive sensing with deterministic guarantees using expander graphs,” in
Proc. InformationTheory Workshop (ITW) , September 2007, pp. 414–419.[4] Y. Wu and S. Verd´u, “Fundamental limits of almost lossless analog compression,” in
Proc. IEEE Int. Symp. Information Theory (ISIT) ,2009, pp. 359 – 363.[5] R. G. Baraniuk, “Compressive sensing,”
IEEE Signal Processing Magazine , vol. 24, pp. 118–124, July 2007.[6] J. Tropp, “Topics in sparse approximation,” Ph.D. dissertation, University of Texas at Austin, 2004.[7] R. Berinde, A. Gilbert, P. Indyk, and K. Strauss, “Combining geometry and combinatorics: A unified approach to sparse signal recovery,”in , September 2008, pp. 798–805.[8] J. Tropp and A. Gilbert, “Signal recovery from random measurements via orthogonal matching pursuit,”
IEEE Trans. Inform. Theory ,vol. 53 (12), pp. 4655–4666, December 2007.[9] D. Needell and R. Vershynin, “Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit,”
Foundations of Computational Mathematics , vol. 9 (3), pp. 317–334, June 2009.[10] S. Mendelson, A. Pajor, and N. Tomczak-Jaegermann, “Uniform uncertainty principle for bernoulli and sub-gaussian ensembles,”
Constructive Approximation , vol. 28 (3), pp. 277–289, December 2008.[11] R. Baraniuk, M. Davenport, R. DeVore, and M. Wakin, “A simple proof of the restricted isometry property for random matrices,”
Constructive Approximation, Springer New York , vol. 28, no. 3, pp. 253–263, December 2008.[12] E. Cand`es and T. Tao, “Decoding by linear programming,”
IEEE Trans. Inform. Theory , vol. 51, no. 12, pp. 4203–4215, December2005.[13] V. Chandar, “A negative result concerning explicit matrices with the restricted isometry property,”
Preprint , March 2008.[14] G. Cormode and M. Muthukrishnan, “Combinatorial algorithms for compressed sensing,” in
Proc. Structural Information andCommunication Complexity (SIROCCO) , 2006, pp. 280–294.[15] P. Indyk, “Explicit constructions for compressed sensing of sparse signals,” in
Proc. Symp. on Discrete Algorithms (SODA) , 2008.[16] A. C. Gilbert, M. J. Strauss, J. A. Tropp, and R. Vershynin, “Algorithmic linear dimension reduction in the (cid:96) norm for sparse vectors,”in , 2006.[17] ——, “One sketch for all: Fast algorithms for compressed sensing,” in Proc. 39th ACM Symposium on Theory of Computing (STOC) ,2007, pp. 237–246.[18] S. Sarvotham, D. Baron, and R. Baraniuk, “Sudocodes - fast measurement and reconstruction of sparse signals,” in
Proc. IEEE Int.Symp. Information Theory (ISIT) , July 2006, pp. 2804–2808.[19] V. Chandar, D. Shah, and G. W. Wornell, “A simple message-passing algorithm for compressed sensing,” in
Proc. IEEE Int. Symp.Information Theory (ISIT) , June 2010, pp. 1968–1972.[20] F. Zhang and H. D. Pfister, “On the iterative decoding of high rate ldpc codes with applications in compressed sensing.” [Online].Available: http://arxiv.org/abs/0903.2232[21] ——, “Compressed sensing and linear codes over real numbers,” in
Proc. Information Theory and Applications Workshop , February2008, pp. 558–561.[22] ——, “List-message passing achieves capacity on the q-ary symmetric channel for large q,” in
IEEE Global TelecommunicationsConference (GLOBECOM) , November 2007, pp. 283–287.[23] ——, “List-message passing achieves capacity on the q-ary symmetric channel for large q,”
Preprint Submitted to IEEE Trans. Inform.Theory .[24] Y. Lu, A. Montanari, B. Prabhakar, S. Dharmapurikar, and A. Kabbani, “Counter braids: A novel counter architecture for per-flowmeasurement,” in
Proc. International Conference on Measurement and Modeling of Computer Systems ACM SIGMETRICS , June 2008,pp. 121–132.[25] M. Akcakaya, J. Park, and V. Tarokh, “Low density frames for compressive sensing,” in
Proc. IEEE Int. Conf. Acoustics, Speech, andSignal Processing (ICASSP) , March 2010, pp. 3642–3645.[26] D. Baron, S. Savotham, and R. G. Baraniuk, “Bayesian compressive sensing via belief propagation,”
IEEE Transactions on SignalProcessing , vol. 58 (1), pp. 269–280, January 2010. [27] C. P. Robert, The Bayesian Choise: A Decision Theoretic Motivation . Springer-Verlag, 1994.[28] C. C. Paige and M. A. Saunders, “Lsqr: Sparse linear equations and least squares problems,”
ACM Transactions on MathematicalSoftware (TOMS) , vol. 8, pp. 195–209, June 1982.[29] M. Luby and M. Mitzenmacher, “Verification-based decoding for packet-based low-density parity-check codes,”
IEEE Trans. Inform.Theory , vol. 51 (1), pp. 120–127, January 2005.[30] Y. Lu, A. Montanari, and B. Prabhakar, “Counter braids: Asymptotic optimality of the message passing decoding algorithm,” in , September 2008, pp. 209 – 216.[31] M. G. Luby, M. Mitzenmacher, M. A. Shokrollahi, and D. A. Spielman, “Efficient erasure correcting codes,”
IEEE Trans. Inform.Theory , vol. 47, pp. 569–584, February 2001.[32] M. Luby, M. Mitzenmacher, M. A. Shokrollahi, and D. Spielman, “Improved low-density parity-check codes using irregular graphs,”
IEEE Trans. Infor. Theory , vol. 47, pp. 585–598, February 2001.[33] D. J. MacKay,
Information Theory, Inference, and Learning Algorithms . Cambridge University Press, 2003.[34] C. Berrou and A. Glavieux, “Near optimum error correcting coding and decoding: turbo-codes,”
IEEE Trans. Comm. , vol. 44 (10), pp.1261–1271, October 1996.[35] T. J. Richardson and R. L. Urbanke, “The capacity of low-density parity-check codes under message-passing decoding,”
IEEE Trans.Inform. Theory , vol. 47 (2), pp. 599–618, February 2001.[36] N. Alon and J. H. Spencer,
The Probabilistic Method . Wiley Series in Discrete Mathematics and Optimization, 2008.[37] E. Cand`es, J. Romberg, and T. Tao, “Stable signal recovery from incomplete and inaccurate measurements,”
Communications on PureMathematics , vol. 59, pp. 1207–1223, August 2006.[38] H. L. V. Trees,
Detection, Estimation, and Modulation Theory, Part I
The Journal of Fourier Analysis andApplications , vol. 14, no. 5-6, pp. 877–905, December 2008. A PPENDIX ID ETAILED D ESCRIPTION OF THE A NALYSIS FOR G ENIE
A. General Setup
To derive the update rules in the analysis of the Genie, assume that we are at the start of the first half-round of iteration (cid:96) . We thus have the probabilities α ( (cid:96) ) , p ( (cid:96) − , N i , and p ( (cid:96) − , K j , and are interested in derivingthe same probabilities for iteration (cid:96) + 1 . The update rules relate the two sets of probabilities; one at thebeginning of iteration (cid:96) and the other at the end of iteration (cid:96) . Hence, we first derive the update rules foriteration (cid:96) , then we discuss the initial conditions for iteration 0.In the following, we use the notation s e to refer to the status bit in the message transmitted over edge e from variable node to check node. B. Derivation of Formulas
When a variable node is recovered in the second half-round of iteration (cid:96) − , the d v edges adjacent tothe variable node carry the recovery message to the neighboring nodes. Therefore, check nodes neighborto the recovered variable node face a reduction in their degree. We denote by p ( (cid:96) ) N j ↓ i the probability thatthe degree of a check node in the induced subgraph is reduced from i to j ≤ i after the first half-roundof iteration (cid:96) . This happens if out of i edges emanating from the check node and incident to the set ofunresolved variable nodes K ( (cid:96) ) , i − j of them carry a message from their variable nodes indicating thatthey have been recovered.On the other side of the graph, when a variable node in K ( (cid:96) − , i ( ≤ i ≤ d v ) is recovered, by definition, outof d v check nodes receiving the recovery message, i have degree and d v − i have degree j ( ≤ j ≤ d c ).In the asymptotic case, as n grows large, we may assume that for each recovered variable node, the setof i check nodes of degree and the set of d v − i check nodes of degree more than are distributedindependently and uniformly with respect to the set of all check nodes of degree and the set of allcheck nodes of degree more than , respectively. As one set contains check nodes of only degree andthe other contains check nodes with a variety of degrees, we shall differentiate between p ( (cid:96) ) N j ↓ i for j = 1 and j > . Once these probabilities are found, the new distribution of check node degrees p ( (cid:96), N i can bederived using the total probability law: p ( (cid:96), N i = d c (cid:88) j = i p ( (cid:96) − , N j p ( (cid:96) ) N j ↓ i , ≤ i ≤ d c . To find the probability p ( (cid:96) ) N j ↓ i , j ≥ , we need the conditional probability that an edge connecting a checknode in the set N ( (cid:96) − , j and an unverified variable node, carries a recovered message to the check node in the first half-round of iteration (cid:96) . We denote this conditional probability by p ( (cid:96) ) d> . Assuming this probabilityis known, we have: p ( (cid:96) ) N j ↓ i = (cid:18) jj − i (cid:19) (cid:16) p ( (cid:96) ) d> (cid:17) j − i (cid:16) − p ( (cid:96) ) d> (cid:17) i , j = 2 , · · · , d c , i = 0 , · · · , j. The probability p ( (cid:96) ) d> can be computed as follows. p ( (cid:96) ) d> = Pr[ s e = 1 | c ∈ {N ( (cid:96) − , , · · · , N ( (cid:96) − , d c } , v ∈ K ( (cid:96) − ] , = d v (cid:88) i =1 Pr[ v ∈ K ( (cid:96) − , i | c ∈ {N ( (cid:96) − , , · · · , N ( (cid:96) − , d c } , v ∈ K ( (cid:96) − ] , = d v (cid:88) i =1 Pr[ c ∈ {N ( (cid:96) − , , · · · , N ( (cid:96) − , d c }| v ∈ K ( (cid:96) − , i , v ∈ K ( (cid:96) − ] Pr[ v ∈ K ( (cid:96) − , i | v ∈ K ( (cid:96) − ]Pr[ c ∈ {N ( (cid:96) − , , · · · , N ( (cid:96) − , d c }| v ∈ K ( (cid:96) − ] , = d v (cid:88) i =1 (cid:16) − Pr[ c ∈ N ( (cid:96) − , | v ∈ K ( (cid:96) − , i , v ∈ K ( (cid:96) − ] (cid:17) Pr[ v ∈ K ( (cid:96) − , i | v ∈ K ( (cid:96) − ] (cid:16) − Pr[ c ∈ N ( (cid:96) − , | v ∈ K ( (cid:96) − ] (cid:17) , = d v (cid:88) i =1 (cid:18) − id v (cid:19) p ( (cid:96) − , K i − p ( (cid:96) ) = d v (cid:88) i =1 p ( (cid:96) − , K i − d v (cid:88) i =1 id v p ( (cid:96) − , K i − p ( (cid:96) ) , where, p ( (cid:96) ) is defined as the probability of the edge e being adjacent to the check node c ∈ N ( (cid:96) − , conditioned on the fact that the variable node v ∈ K ( (cid:96) − . This probability can be calculated as: p ( (cid:96) ) = Pr[ c ∈ N ( (cid:96) − , | v ∈ K ( (cid:96) − ] , = d v (cid:88) i =0 Pr[ c ∈ N ( (cid:96) − , , v ∈ K ( (cid:96) − , i | v ∈ K ( (cid:96) − ] , = d v (cid:88) i =0 Pr[ c ∈ N ( (cid:96) − , | v ∈ K ( (cid:96) − , i , v ∈ K ( (cid:96) − ] Pr[ v ∈ K ( (cid:96) − , i | v ∈ K ( (cid:96) − ] , = d v (cid:88) i =0 id v p ( (cid:96) − , K i . Therefore, the probability p ( (cid:96) ) d> can be simplified as follows. p ( (cid:96) ) d> = d v (cid:88) i =1 p ( (cid:96) − , K i − d v (cid:88) i =1 id v p ( (cid:96) − , K i − p ( (cid:96) ) , = 1 − p ( (cid:96) − , K − p ( (cid:96) ) − p ( (cid:96) ) , = 1 − p ( (cid:96) − , K − p ( (cid:96) ) . By using Bayes’ rule, the probability p ( (cid:96) ) can be calculated more efficiently as: p ( (cid:96) ) = Pr[ c ∈ N ( (cid:96) − , | v ∈ K ( (cid:96) − ] , = Pr[ v ∈ K ( (cid:96) − | c ∈ N ( (cid:96) − , ] Pr[ c ∈ N ( (cid:96) − , ]Pr[ v ∈ K ( (cid:96) − ] , = 1 d c × p ( (cid:96) − , N α ( (cid:96) − , = p ( (cid:96) − , N α ( (cid:96) − d c . (20)In the Genie algorithm, all variable nodes neighbor to at least one check node of degree are recovered.Therefore, all check nodes of degree must be grouped as degree in the first half-round of the nextiteration. Hence, we have: p ( (cid:96) ) N ↓ = 1 , p ( (cid:96) ) N ↓ = 0 . So far, we found the update rules for the first half-round of iteration (cid:96) . In the second half-round, variablenodes receive messages from their neighboring check nodes. The degrees reflected in the messages mayre-group some variable nodes. According to the verification rule in the Genie algorithm, the only unverifiedset of variable nodes is the set K ( (cid:96) − , . Variable nodes in this set have no connection to check nodesin the set N ( (cid:96) − , but d v connections to the sets {N ( (cid:96) − , , · · · , N ( (cid:96) − , d c } . Suppose v ∈ K ( (cid:96) − , i . In thiscase, if one of the adjacent check nodes of v in {N ( (cid:96) − , , · · · , N ( (cid:96) − , d c } turns to a check node in N ( (cid:96), , v will move from K ( (cid:96) − , i to K ( (cid:96), i +1 . This is shown in Figure 7. Fig. 7. A variable node in K i turns to a variable node in K i +1 . Due to the random structure of the graph assumed in the asymptotic case, the single edges connected tothe set of check nodes N ( (cid:96), are uniformly and independently distributed with respect to the candidateedges adjacent to the set of check nodes {N ( (cid:96) − , , · · · , N ( (cid:96) − , d c } .In the Genie, the probability p ( (cid:96) ) K ↑ j is defined as the probability of a variable node v ∈ K ( (cid:96) − , turning to v ∈ K ( (cid:96), j . As the set K ( (cid:96) − , is also the unverified support set, we have: p ( (cid:96), K j = p ( (cid:96) ) K ↑ j = Pr[ v ∈ K ( (cid:96), j | v ∈ K ( (cid:96) − , , v ∈ K ( (cid:96) ) ] = (cid:18) d v j (cid:19) (cid:0) p ( (cid:96) ) x (cid:1) j (cid:0) − p ( (cid:96) ) x (cid:1) d v − j , j = 0 , · · · , d v , (21) where p ( (cid:96) ) x is defined as the probability that an edge adjacent to a variable node v ∈ K ( (cid:96) − , carries amessage indicating that the adjacent check node c has a degree equal to ; i.e., c ∈ N ( (cid:96), . The probability p ( (cid:96) ) x is calculated as follows: p ( (cid:96) ) x = Pr[ c ∈ N ( (cid:96), | v ∈ K ( (cid:96) − , ] , = Pr[ c ∈ N ( (cid:96), ] Pr[ v ∈ K ( (cid:96) − , | c ∈ N ( (cid:96), ] d c (cid:88) i =0 Pr[ c ∈ N ( (cid:96), i ] Pr[ v ∈ K ( (cid:96) − , | c ∈ N ( (cid:96), i ] , = p ( (cid:96), N × d cd c (cid:88) i =0 id c p ( (cid:96), N i = p ( (cid:96), N d c (cid:88) i =1 ip ( (cid:96), N i = p ( (cid:96), N α ( (cid:96) ) d c = p ( (cid:96) +1) . (22)In the Genie algorithm, a variable node is resolved if among all its neighboring check nodes, there exists atleast one with degree equal to . Hence, the probability of a variable node in the set K ( (cid:96) ) being recoveredis calculated as (cid:80) d v i =1 p ( (cid:96), K i . Therefore, according to the total probability theorem, the probability of avariable node v remaining unverified, i.e., v ∈ K ( (cid:96) +1) , is: α ( (cid:96) +1) = α ( (cid:96) ) (cid:32) − d v (cid:88) i =1 p ( (cid:96), K i (cid:33) = α ( (cid:96) ) p ( (cid:96), K . C. Initial Probabilities for the Genie Algorithm
In the first half-round of iteration , all check nodes have degree d c in the subgraph induced by theunverified variable nodes. This degree is reflected in their outgoing messages. These messages carry noinformation to the variable nodes, and no variable node can be resolved based on them. On the otherhand, variable nodes not in the support set are resolved with a value equal to . Assuming an initialdensity factor of α (0) , − α (0) fraction of the messages from variable nodes carry a recovery notice tocheck nodes. Since at iteration no variable node in the support set was verified we have α (1) = α (0) .Hence, p (1) d> = 1 − α (1) . Since p (0 , N dc = 1 , the set of probabilities p (1 , N i , is given by the following. p (1 , N i = p (0 , N dc p (1) N dc ↓ i = p (1) N dc ↓ i , ≤ i ≤ d c , = (cid:18) d c d c − i (cid:19) (cid:16) p (1) d> (cid:17) d c − i (cid:16) − p (1) d> (cid:17) i , ≤ i ≤ d c , = (cid:18) d c i (cid:19) (cid:0) α (1) (cid:1) i (cid:0) − α (1) (cid:1) d c − i , ≤ i ≤ d c . To find the probability p (1 , K i , we first find the probability p (1) x from (22) and then replace it in the (21). A PPENDIX
IID
ETAILED D ESCRIPTION OF THE A NALYSIS FOR
LMWe first prove Theorem 3 in which the verified variable nodes in the support set are characterized.
Proof of Theorem 3:
Suppose that we partition the set of unknown variable nodes in the support set K ( (cid:96) ) into subsets K ( (cid:96) ) i , where i represents the number of messages received by the variable node indicatingthat the transmitting check node has degree ; i.e., i neighboring check nodes are in the set N ( (cid:96) )1 , . Sincecheck nodes in N , have both degree and a non-zero value, each variable node v in the set (cid:83) d v i =1 K ( (cid:96) ) i has at least one non-zero check node of degree one as a neighbor. Recall that in each iteration of the LMalgorithm a non-zero variable node is verified if and only if it is connected to a check node of degreeone. Therefore, v ∈ (cid:83) d v i =1 K ( (cid:96) ) i means that v is recovered at iteration (cid:96) of the LM algorithm.To prove the converse, we assume that a variable node v is resolved at iteration (cid:96) of the LM algorithm.We show that v ∈ (cid:83) d v i =1 K ( (cid:96) ) i . The fact that v is resolved at iteration (cid:96) of the LM algorithm implies that itwas connected to at least one check node of degree one. As we dealt with all zero-valued check nodesin the previous round, the degree-one check nodes must have a non-zero value. Therefore, the variablenode v is, by definition, in the set (cid:83) d v i =1 K ( (cid:96) ) i . A. Recovering Variable Nodes • A variable node in K is recovered if it is connected to a check node in N , . • A variable node in ∆ is recovered if it is connected to a check node in (cid:83) d c i =1 N ,i ; i.e., a zero-valuedcheck node. B. Iteration Zero
The messages passed in the first round of iteration zero do not contribute to the verification of variablenodes and do not change the grouping of the check nodes. Iteration zero thus consists of only one roundand hence, two half-rounds. In the first half-round, check nodes pass their values along with their degrees( d c ) to their neighboring variable nodes. In the second half-round, variable nodes process the incomingmessages. A variable node is verified if it receives at least one message with a value equal to zero. Inthis case, the variable node is verified with a value equal to zero according to the ZCN rule. The set ofall variable nodes verified in this half-round make the set R (1) .Let N (0) i denote the set of check nodes with i neighboring variable nodes in the support set. The probability p (0) N i defined as the probability that a check node belongs to the set N (0) i is calculated as follows: p (0) N i (cid:44) Pr( c ∈ N (0) i ) = (cid:18) d c i (cid:19) (cid:0) α (0) (cid:1) i (cid:0) − α (0) (cid:1) d c − i , i = 0 , · · · , d c . Hence, the probability that a check node c has a value equal to zero ( c = 0 ) is: Pr[ c = 0] = p (0) N = (cid:0) − α (0) (cid:1) d c . Let ∆ (0 ,R , j denote the set of zero-valued variable nodes that receive j zero-valued messages. Theprobability p (0 ,R , j defined as the probability that a zero-valued variable node belongs to the set ∆ (0 ,R , j is calculated as follows: p (0 ,R , j = Pr[ v ∈ ∆ (0 ,R , j | v / ∈ K (0) ] = (cid:18) d v j (cid:19) (cid:16) p (0) δ (cid:17) j (cid:16) − p (0) δ (cid:17) d v − j , j = 0 , · · · , d v , where p (0) δ is the probability that an edge adjacent to a zero-valued variable node carries a message withvalue zero, and is calculated as follows: p (0) δ = (cid:0) − α (0) (cid:1) d c − . (23)Let p (1)∆ denote the probability that a variable node has a zero value and does not receive even one messagewith value equal to zero in the second half-round of iteration zero. We have: p (1)∆ (cid:44) Pr[ v ∈ ∆ (1) ] = Pr[ v / ∈ K (0) ] Pr[ v ∈ ∆ (0 ,R , | v / ∈ K (0) ] = (1 − α (0) ) (cid:16) − (cid:0) − α (0) (cid:1) d c − (cid:17) d v . (24)Since no element of the support set is verified at iteration zero, we have K (0) = K (1) , and hence, α (1) = Pr[ v ∈ K (1) ] = α (0) . At the end of iteration , all check nodes have degree d c in the subgraph induced by the unverifiedvariable nodes. Thus, the set of all check nodes can be partitioned into subsets N (0 ,R , i,d c − i , where i denotesthe number of neighboring variable nodes in the support set ( ≤ i ≤ d c ). The edges adjacent to a checknode are partitioned into two sets: K -edges and ∆ -edges. K -edges are connected to variable nodes in thesupport set, while ∆ -edges are connected to zero-valued variable nodes. Therefore, a check node in theset N (0 ,R , i,d c − i ( ≤ i ≤ d c ) has i , K -edges and d c − i , ∆ -edges. C. Iteration One and Beyond
Here we present the analysis for iteration one. Since the analysis of the second iteration and beyond issimilar to that of iteration one, they are omitted. The summary of the formulas can be found in SectionV-D.The verified messages sent from variable nodes to check nodes at the end of iteration zero, are processed atcheck nodes at iteration , HR1-R1. Based on the recovery process at iteration zero, all verified messagesare sent from variable nodes in the sets ∆ j , ≤ j ≤ d v . We partition the set of edges adjacent to avariable node in the set ∆ j , ≤ j ≤ d v , into N =0 -edges and N (cid:54) =0 -edges. Edges in the set N =0 -edges areconnected to zero-valued check nodes (check nodes in the set N ), while edges in the set N (cid:54) =0 -edges are connected to non-zero check nodes.Since check nodes in the set N (0 ,R , receive d c verified messages, we are interested in the set of checknodes that are regrouped from N (0 ,R , i,d c − i to N (1 ,R , i,j in HR1-R1 of iteration for ≤ i ≤ d c . Such aset of check nodes is denoted by N (1 ,R i,d c − i ↓ j . The messages responsible for such regrouping are carriedover N (cid:54) =0 -edges. To analyze the regrouping, we need to find the probability p E R of an edge in the setof ∆ -edges to carry a verified message. Such edges are not connected to the set ∆ . Before finding theprobability p E R , we introduce two notations v e and c e to denote the variable node and the check nodeconnected by means of the edge e . Now, we have: p (1 ,R E R = 1 − Pr[ v e ∈ ∆ (0 ,R , | v e / ∈ K (1) , c e / ∈ N (1)0 ] , = 1 − Pr[ v e ∈ ∆ (0 ,R , | v e / ∈ K (1) ] Pr[ c e / ∈ N (1)0 | v e / ∈ K (1) , v e ∈ ∆ (0 ,R , ]Pr[ c e / ∈ N (1)0 | v e / ∈ K (1) ] , = 1 − Pr[ v e ∈ ∆ (0 ,R , | v e / ∈ K (1) ] Pr[ c e / ∈ N (1)0 | v e / ∈ K (1) , v e ∈ ∆ (0 ,R , ] d v (cid:88) i =0 Pr[ c e / ∈ N (1)0 , v e ∈ ∆ (0 ,R , i | v e / ∈ K (1) ] , = 1 − p (0 ,R , × d v (cid:88) i =0 p (0 ,R , i (cid:18) d v − id v (cid:19) , = 1 − p (0 ,R , − p (0) δ = 1 − p (1)∆ − p (0) δ , where p (1)∆ and p (0) δ are given in (24) and (23), respectively. We thus have the following regrouping ofcheck nodes based on the second index: p (1 ,R N i,dc − i ↓ j = (cid:18) d c − ij (cid:19) (cid:16) − p (1 ,R E R (cid:17) j (cid:16) p (1 ,R E R (cid:17) d c − i − j , i = 1 , · · · , d v , j = 0 , · · · , d c − i. Hence, p (1 ,R , N i,j = p (0 ,R , N i,dc − i p (1 ,R N i,dc − i ↓ j , i = 1 , · · · , d v , j = 0 , · · · , d c − i. (25)In HR2-R1, messages are sent from check nodes to variable nodes. At this stage, check nodes in theset N (1 ,R , , transmit a message with its first coordinate equal to . Variable nodes in the support setthat receive at least one such message, are verified with the value contained in that message in HR2-R1.After processing the received messages, we divide the set of all variable nodes in the support set K (1) into subsets K (1 ,R , i , ≤ i ≤ d v , where i denotes the number of degree- messages a variable nodereceives. We denote the set of such variable nodes by K (1 ,R ↑ i . Let p (1 ,R denote the probability that anedge adjacent to a variable node in the support set carries a message with the first coordinate equal to .Using the same notations v e and c e defined above, we have: p (1 ,R = Pr[ c e ∈ N (1 ,R , , | v e ∈ K (1) ] , (26) = Pr[ c e ∈ N (1 ,R , , ] Pr[ v e ∈ K (1) | c e ∈ N (1 ,R , , ]Pr[ v e ∈ K (1) ] , (27) = p (1 ,R , N , × /d c α (1) = p (1 ,R , N , α (1) d c . (28)Hence, the probability p (1 ,R K ↑ i that a variable node v ∈ K (1) belongs to the set K (1 ,R , i is calculated asfollows: p (1 ,R , K i = p (1 ,R K ↑ i (cid:44) Pr( v ∈ K (1 ,R , i | v ∈ K (1) ) , (29) = (cid:18) d v i (cid:19) (cid:0) p (1 ,R (cid:1) i (cid:0) − p (1 ,R (cid:1) d v − i , i = 0 , · · · , d v . (30)Based on the D1CN rule in the LM algorithm, variable nodes in the set (cid:83) d v i =1 K (1 ,R , i are verified.Therefore, the probability that a variable node in the support set remains unverified for iteration is asfollows: α (2) = α (1) (cid:32) − d v (cid:88) i =1 p (1 ,R , K i (cid:33) . With this, the first round of iteration is over. In HR1-R2 of iteration , check nodes receive messagesfrom variable nodes. At this point, check nodes should be regrouped based on their first index, as somevariable nodes in the support set have been verified at HR2-R1. A check node in the set N (1 ,R , i,k isregrouped into the set N (1 ,R , j,k at HR1-R2, if from i edges in the set of K -edges adjacent to the checknode, i − j of them carry a verified message. The set of such check nodes are denoted by N (1 ,R i ↓ j,k .To analyze this regrouping, we need the probability p (1 ,R d that a K -edge carries a verified message atHR1-R2. To find this probability, we proceed as follows: p (1 ,R d = 1 − Pr[ v e ∈ K (1 ,R , | v e ∈ K (1) , c e / ∈ N (1 ,R , , ] , = 1 − Pr[ v e ∈ K (1 ,R , | v e ∈ K (1) ] Pr[ c e / ∈ N (1 ,R , , | v e ∈ K (1) , v e ∈ K (1 ,R , ]Pr[ c e / ∈ N (1 ,R , , | v e ∈ K (1) ] , = 1 − Pr[ v e ∈ K (1 ,R , | v e ∈ K (1) ] Pr[ c e / ∈ N (1 ,R , , | v e ∈ K (1) , v e ∈ K (1 ,R , ] d v (cid:88) i =0 Pr[ v e ∈ K (1 ,R , i | v e ∈ K (1) ] Pr[ c e / ∈ N (1 ,R , , | v e ∈ K (1) , v e ∈ K (1 ,R , i ] , = 1 − p (1 ,R , K × d v (cid:88) i =0 p (1 ,R , K i (cid:18) d v − id v (cid:19) , = 1 − d v p (1 ,R , K d v (cid:88) i =0 ( d v − i ) p (1 ,R , K i , = 1 − p (1 ,R , K − p (1 ,R , where p (1 ,R , K and p (1 ,R are given by (30) and (28), respectively. Hence, the probability p (1 ,R N i ↓ j,k that acheck node belongs to the set N (1 ,R i ↓ j,k is calculated as follows: p (1 ,R N i ↓ j,k = (cid:18) ij (cid:19) (cid:16) p (1 ,R d (cid:17) i − j (cid:16) − p (1 ,R d (cid:17) j , i = 1 , · · · , d c , j = 0 , · · · , i, k = 0 , · · · , d c − i. Note that by construction, we have p (1 ,R N ↓ , = 1 and p (1 ,R N ↓ , = 0 . After the regrouping, the probability p (1 ,R , N j,k that a check node belongs to the set N (1 ,R , j,k is calculated by: p (1 ,R , N j,k = d c (cid:88) i = j p (1 ,R , N i,k p (1 ,R N i ↓ j,k , j = 0 , · · · , d c , k = 0 , · · · , d c − i. The measurement corresponding to check nodes in the set N (1 ,R , ,k , ≤ k ≤ d c − , changes to zero,as the check nodes are no longer connected to an unverified variable node in the support set. Hence, themessages transmitted by such check nodes have a value equal to zero, which in turn verifies some variablenodes (in the set ∆ ) at HR2-R2 of iteration . This is indeed the last step in iteration .As explained before, the probability p (1 ,R δ is needed to find the probability that a zero-valued variablenode is verified at this stage. This probability is calculated as follows: p (1 ,R δ = d c − (cid:88) j =1 Pr[ c e ∈ N (1 ,R , ,j | v e ∈ ∆ (1) ] , = d c − (cid:88) j =1 Pr[ c e ∈ N (1 ,R , ,j ] Pr[ v e ∈ ∆ (1) | c e ∈ N (1 ,R , ,j ] d c (cid:88) i =0 d c − (cid:88) j =1 Pr[ c e ∈ N (1 ,R , i,j ] Pr[ v e ∈ ∆ (1) | c e ∈ N (1 ,R , i,j ] , = d c − (cid:88) j =1 p (1 ,R , N ,j (cid:18) jd c (cid:19) d c (cid:88) i =0 d c − (cid:88) j =1 p (1 ,R , N i,j (cid:18) jd c (cid:19) , = d c − (cid:88) j =1 j p (1 ,R , N ,j d c (cid:88) i =0 d c − (cid:88) j =1 jp (1 ,R , N i,j . Note that the denominator is indeed
Pr[ v e ∈ ∆ (1) ] (cid:44) p (1)∆ . Hence, the probability p (1 ,R , i , as definedbefore, is calculated as follows: p (1 ,R , i = (cid:18) d v i (cid:19) (cid:16) p (1 ,R δ (cid:17) i (cid:16) − p (1 ,R δ (cid:17) d v − i , i = 0 , · · · , d v . Lastly, the probability that a variable node is zero-valued and remains unverified for iteration is asfollows: p (2)∆ = p (1)∆ p (1 ,R , . A PPENDIX
IIID
ETAILED D ESCRIPTION OF THE A NALYSIS FOR
SBB
Proof of Theorem 4:
First we show that if a variable node v ∈ (cid:83) d v i =2 K ( (cid:96),R , i ∪ ˆ K ( (cid:96),R , , it will berecovered in the SBB algorithm at iteration (cid:96) , HR2-R1. A check node in the set N is only connected toone element of the support set. So, a variable node v in the support set with i neighbors in the set N receives i messages from the neighboring check nodes with the same value. Hence, variable nodes in theset (cid:83) d v i =2 K ( (cid:96),R , i are recovered. Variable nodes in the set K ( (cid:96),R , , however, can not be verified accordingto ECN rule because they are not connected to at least two check nodes with the same value. So, theonly verification rule applicable would be the D1CN. With the definition of the set ˆ K ( (cid:96),R , , a variablenode v ∈ ˆ K ( (cid:96),R , is verified based on D1CN at iteration (cid:96) of the SBB algorithm.To prove the converse, we have to show that when a variable node v is verified with a non-zero value atiteration (cid:96) of the SBB algorithm, then we have v ∈ (cid:83) d v i =2 K ( (cid:96),R , i ∪ ˆ K ( (cid:96),R , . The statement is true basedon the definition of the sets and the fact that false verification happens with probability zero. A. Verifying Variable Nodes • A variable node in K is verified if it is connected to a check node in N , . • A variable node in K i , ≤ i ≤ d v , is verified if it is connected to a check node in N (cid:44) (cid:83) d c − j =0 N ,j . • A variable node in ∆ is verified if it is connected to a check node in (cid:83) d c i =1 N ,i ; i.e., a zero-valuedcheck node. B. Iteration Zero
The process for iteration zero is the same as that of the LM algorithm discussed in II-B. The analysis isthus not repeated here.
C. Iteration One
In the first half-round of the first round (HR1-R1) of any iteration, check nodes process the receivedmessages from variable nodes and generate the outgoing messages accordingly. Note that HR1-R1 in theSBB and the LM algorithm are the same.In this section, we adopt the notation N i , with any superscript, to denote the set (cid:83) d c − ij =0 N i,j . The verifiedmessages sent from variable nodes to check nodes at the end of iteration zero, are processed at checknodes at iteration , HR1-R1. Based on the recovery process at iteration zero, all verified messages aresent from variable nodes in the sets ∆ j , ≤ j ≤ d v . We partition the set of edges adjacent to a variable node in the set ∆ j , ≤ j ≤ d v , into N =0 -edges and N (cid:54) =0 -edges. N =0 -edges are connected to zero-valuedcheck nodes (check nodes in the set N ), while N (cid:54) =0 -edges are connected to non-zero check nodes.We are interested in the set of check nodes that are regrouped from N (0 ,R , i,d c − i to N (1 ,R , i,j for ≤ i ≤ d c ,and eventually the probability p (1 ,R , N i,j in HR1-R1 of iteration . The calculation of this probability isexactly the same as the derivation of (25) in the analysis of the LM algorithm.In HR2-R1, the following 2 types of variable nodes in the support set are verified:1) variable nodes neighbor to at least one check node of degree ; i.e., variable nodes that receive atleast one message with the first coordinate equal to . These variable nodes are verified with thevalue contained in that message based on D1CN.2) variable nodes neighbor to at least two check nodes in the set N (1 ,R , ; i.e., variable nodes thatreceive at least two messages with the same value. These variable nodes are verified with thecommon value of the messages based on the ECN rule.Therefore, after processing the received messages, we divide the set of all unverified variable nodes in thesupport set K (1) into subsets K (1 ,R i , ≤ i ≤ d v , where i denotes the number of neighboring check nodesin the set N (1 ,R , . We denote the set of such variable nodes by K (1 ,R ↑ i . It is worth mentioning that somevariable nodes in the set K (1 ,R i are verified according to D1CN and ECN. The remaining unverifiedvariable nodes, after the recovery, make the sets K (1 ,R , j , ≤ j ≤ d v . The sets K (1 ,R i are removedfrom the summarized formulas in Section V-E to prevent any confusion. They appear here because theysimplify the notations and explanations.Let p (1 ,R denote the conditional probability that an edge is adjacent to a check node in the set N (1 ,R , given that it is adjacent to a variable node in the support set. Using the same notations v e and c e definedbefore, we have: p (1 ,R = Pr[ c e ∈ N (1 ,R , | v e ∈ K (1) ] , = Pr[ c e ∈ N (1 ,R , ] Pr[ v e ∈ K (1) | c e ∈ N (1 ,R , ]Pr[ v e ∈ K (1) ] , = d c − (cid:88) j =0 p (1 ,R , N ,j × /d c α (1) = d c − (cid:88) j =0 p (1 ,R , N ,j α (1) d c . (31)Hence, the probability p (1 ,R K ↑ i that a variable node v ∈ K (1) belongs to the set K (1 ,R i is calculated asfollows: p (1 ,R K i = p (1 ,R K ↑ i (cid:44) Pr( v ∈ K (1 ,R i | v ∈ K (1) ) , = (cid:18) d v i (cid:19) (cid:0) p (1 ,R (cid:1) i (cid:0) − p (1 ,R (cid:1) d v − i , i = 0 , · · · , d v . Based on the ECN rule, variable nodes in the set (cid:83) d v i =2 K (1 ,R i are verified. A fraction f (1 ,R of variablenodes in the set K (1 ,R that receive a message with the first coordinate equal to one are also verifiedbased on the D1CN rule. Using the Bayes’ rule, this fraction is calculated as follows: f (1 ,R = Pr[ c e ∈ N (1 ,R , , | v e ∈ K (1 ,R , c e ∈ N (0 ,R , ] . By omitting the superscripts, we obtain: f (1 ,R = Pr[ c e ∈ N , ] Pr[ v e ∈ K | c e ∈ N , ] Pr[ c e ∈ N | c e ∈ N , , v e ∈ K ]Pr[ c e ∈ N ] Pr[ v e ∈ K | c e ∈ N ] , = p (1 ,R , N , × X × p (1 ,R , N × X , = p (1 ,R , N , p (1 ,R , N , where we have used the fact that Pr[ v e ∈ K | c e ∈ N , ] = Pr[ v e ∈ K | c e ∈ N ] (cid:44) X . Therefore, theprobability that a variable node in the support set remains unverified for iteration is as follows: α (2) = α (1) (cid:32) − f (1 ,R p (1 ,R K − d v (cid:88) i =2 p (1 ,R K i (cid:33) , = α (1) (cid:16) p (1 ,R K + p (1 ,R K − f (1 ,R p (1 ,R K (cid:17) , = α (1) (cid:16) p (1 ,R K + (cid:0) − f (1 ,R (cid:1) p (1 ,R K (cid:17) . The final regrouping of variable nodes into sets K (1 ,R , i , ≤ i ≤ d v , is performed by taking into accountthe verification of some sets of variable nodes. We have: p (1 ,R , K = 1 N (1 ,R p (1 ,R K .p (1 ,R , K = 1 N (1 ,R (cid:0) − f (1 ,R (cid:1) p (1 ,R K .p (1 ,R , K i = 0 , ≤ i ≤ d v . (32)The normalization factor N (1 ,R is used to make the set of parameters p (1 ,R , K i a valid probability measure,and is calculated as follows: N (1 ,R = p (1 ,R K + (cid:0) − f (1 ,R (cid:1) p (1 ,R K = α (2) α (1) . With this, the analysis of the first round of iteration is over. In HR1-R2 of iteration , check nodesreceive messages from variable nodes. At this point, check nodes should be regrouped based on theirfirst index, as some variable nodes in the support set have been verified at HR2-R1. Since a fraction ofvariable nodes in the set K (1 ,R , are left unverified, unlike the LM algorithm, not all check nodes inthe set N (1 ,R , ,j ( ≤ j ≤ d c − ) are regrouped into the set N (1 ,R , ,j ; some will stay in the same set N (1 ,R , ,j . Hence, in addition to analyzing the set of check nodes N (1 ,R i ↓ k,j that are regrouped from N (1 ,R , i,j to N (1 ,R , k,j , we also have to analyze the set of check nodes N (1 ,R ↓ ,j that are regrouped from N (1 ,R , ,j to N (1 ,R , ,j .Suppose that edges adjacent to a check node are partitioned into two sets: K -edges and ∆ -edges. K -edgesare connected to variable nodes in the set K (1) , while ∆ -edges are connected to zero-valued variablenodes. To analyze the regrouping of check nodes in the sets N (1 ,R , and N (1 ,R , i , ≤ i ≤ d c , we needthe probabilities p (1 ,R d =1 and p (1 ,R d (cid:54) =1 defined as follows. The probability p (1 ,R d =1 is defined as the conditionalprobability of an edge carrying a verified message given that it is a K -edge adjacent to a check nodein the set N (1 ,R , . The probability p (1 ,R d (cid:54) =1 is defined similarly with respect to the set of check nodesin (cid:83) d c i =2 N (1 ,R , i . To find the probability p (1 ,R d =1 , we shall consider only the variable nodes in the set (cid:83) d v i =2 K (1 ,R , i . This is because, variable nodes in the set K (1 ,R , are connected to check nodes in theset N (1 ,R , , only. Let f e denote the status flag ( f e ∈ { , } ) of the message carried over the edge e . Weproceed as follows: p (1 ,R d =1 = Pr[ f e = 1 | v e ∈ K (1) , c e ∈ N (1 ,R , ] , = Pr[ v e ∈ d v (cid:91) i =2 K (1 ,R , i | v e ∈ K (1) , c e ∈ N (1 ,R , ] , = 1 − Pr[ v e ∈ K (1 ,R , | v e ∈ K (1) , c e ∈ N (1 ,R , ] , = 1 − Pr[ v e ∈ K (1 ,R , | v e ∈ K (1) ] Pr[ c e ∈ N (1 ,R , | v e ∈ K (1 ,R , , v e ∈ K (1) ]Pr[ c e ∈ N (1 ,R , | v e ∈ K (1) ] , = 1 − p (1 ,R , K /d v p (1 ,R , = 1 − p (1 ,R , K d v p (1 ,R , where p (1 ,R , K and p (1 ,R are given by (32) and (31), respectively. Using the same approach, the calcu-lation of the probability p (1 ,R d (cid:54) =1 follows. In each step, we omit some of the superscripts to simplify thepresentation. p (1 ,R d (cid:54) =1 = Pr[ f e = 1 | v e ∈ K (1) , c e ∈ d c (cid:91) j =2 N (1 ,R , j ] , = 1 − Pr[ v e ∈ K (1 ,R , | v e ∈ K (1) , c e ∈ d c (cid:91) j =2 N j ] − Pr[ v e ∈ K (1 ,R , , f e = 0 | v e ∈ K (1) , c e ∈ d c (cid:91) j =2 N j ] , = 1 − Pr[ v e ∈ K | v e ∈ K , c e ∈ d c (cid:91) j =2 N j ] − Pr[ v e ∈ K | v e ∈ K , c e ∈ d c (cid:91) j =2 N j ] Pr[ f e = 0 | v e ∈ K , v e ∈ K , c e ∈ d c (cid:91) j =2 N j ] , = 1 − Pr[ v e ∈ K | v e ∈ K ] Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K , v e ∈ K ]Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K ] − Pr[ v e ∈ K | v e ∈ K ] Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K , v e ∈ K ]Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K ] (cid:0) − f (1 ,R (cid:1) , = 1 − Pr[ v e ∈ K | v e ∈ K ] Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K , v e ∈ K ]1 − Pr[ c e ∈ N | v e ∈ K ] − Pr[ v e ∈ K | v e ∈ K ] Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K , v e ∈ K ]1 − Pr[ c e ∈ N | v e ∈ K ] (cid:0) − f (1 ,R (cid:1) , = 1 − p (1 ,R , K × − p (1 ,R − p (1 ,R , K (cid:18) d v − d v (cid:19) − p (1 ,R (cid:0) − f (1 ,R (cid:1) . Hence, the probabilities p (1 ,R N i ↓ k,j and p (1 ,R N ↓ ,j that a check node belongs respectively to the set of check nodes N (1 ,R i ↓ k,j and N (1 ,R ↓ ,j are calculated as follows: p (1 ,R N i ↓ k,j = (cid:18) ik (cid:19) (cid:16) p (1 ,R d (cid:54) =1 (cid:17) i − k (cid:16) − p (1 ,R d (cid:54) =1 (cid:17) k , i = 2 , · · · , d c , k = 0 , · · · , i, j = 0 , · · · , d c − i.p (1 ,R N ↓ ,j = (cid:18) (cid:19) (cid:16) p (1 ,R d =1 (cid:17) (cid:16) − p (1 ,R d =1 (cid:17) = p (1 ,R d =1 , p (1 ,R N ↓ ,j = 1 − p (1 ,R d =1 , j = 1 , · · · , d c − i.p (1 ,R N ↓ , = 1 , p (1 ,R N ↓ , = 0 ,p (1 ,R N ↓ ,j = 1 , j = 1 , · · · , d c − i. After the regrouping, the probability p (1 ,R , N k,j that a check node belongs to the set N (1 ,R , k,j is calculatedas follows: p (1 ,R , N k,j = d c (cid:88) i = k p (1 ,R , N i,j p (1 ,R N i ↓ k,j , k = 0 , · · · , d c , j = 0 , · · · , d c − i. The measurement corresponding to check nodes in the set N (1 ,R , ,k , ≤ k ≤ d c − , changes to zero,as the check nodes are no longer connected to an unverified variable node in the support set. Hence, themessages transmitted by such check nodes have a value equal to zero, which in turn verifies some variablenodes (in the set ∆ ) at HR2-R2 of iteration . This is indeed the last step in iteration .As defined before, the probability p (1 ,R δ , is needed to find the probability that a zero-valued variablenode is verified at this stage. This probability is calculated as follows: p (1 ,R δ = d c − (cid:88) j =1 Pr[ c e ∈ N (1 ,R , ,j | v e ∈ ∆ (1) ] , = d c − (cid:88) j =1 Pr[ c e ∈ N (1 ,R , ,j ] Pr[ v e ∈ ∆ (1) | c e ∈ N (1 ,R , ,j ] d c (cid:88) i =0 d c − (cid:88) j =1 Pr[ c e ∈ N (1 ,R , i,j ] Pr[ v e ∈ ∆ (1) | c e ∈ N (1 ,R , i,j ] , = d c − (cid:88) j =1 p (1 ,R , N ,j (cid:18) jd c (cid:19) d c (cid:88) i =0 d c − (cid:88) j =1 p (1 ,R , N i,j (cid:18) jd c (cid:19) , = d c − (cid:88) j =1 j p (1 ,R , N ,j d c (cid:88) i =0 d c − (cid:88) j =1 jp (1 ,R , N i,j . Note that the denominator is in fact
Pr[ v e ∈ ∆ (1) ] (cid:44) p (1)∆ . Hence, the probability p (1 ,R , i , defined asthe probability that an unverified zero-valued variable node belongs to the set ∆ (1 ,R , i , is calculated asfollows: p (1 ,R , i = (cid:18) d v i (cid:19) (cid:16) p (1 ,R δ (cid:17) i (cid:16) − p (1 ,R δ (cid:17) d v − i , i = 0 , · · · , d v . Lastly, the probability that a variable node is zero-valued and remains unverified for iteration is givenby: p (2)∆ = p (1)∆ p (1 ,R , . D. Iterations Two and Beyond
At iteration , HR1-R1, the regrouping of check nodes based on their second index is similar to theprocess in HR1-R1 at iteration . We have: p (2 ,R E R = 1 − p (2)∆ − p (1 ,R δ . For the regrouping of check nodes based on the second index we thus have: p (2 ,R N i,j ↓ k = (cid:18) jk (cid:19) (cid:16) − p (2 ,R E R (cid:17) j (cid:16) p (2 ,R E R (cid:17) j − k , i = 1 , · · · , d c , j = 0 , · · · , d c − i, k = 0 , · · · , j. Hence, p (2 ,R , N i,k = d c − i (cid:88) j = k p (1 ,R , N i,j p (2 ,R N i,j ↓ k , i = 1 , · · · , d c , k = 0 , · · · , d c − i. At iteration , HR2-R1, variable nodes in the support set K (2) should be regrouped. Note that the set K (2) consists of two partitions: K (1 ,R , and K (1 ,R , . Since variable nodes in the set K (1 ,R , are connectedto check nodes in the set N (2 ,R , , and since the regrouping of the variable nodes is based on the numberof their neighbors in the set N (2 ,R , , we shall partition the set of check nodes in N (2 ,R , into two sets: N (2 ,R , +)1 and N (2 ,R ,C )1 . Check nodes in the set N (2 ,R , +)1 were regrouped into the set N (1 ,R , from allthe other sets N (1 ,R , i , ≤ i ≤ d c , at iteration , HR1-R2. Check nodes in the set N (2 ,R ,C )1 , however,were regrouped into the set N (1 ,R , from the set N (1 ,R , . Edges in the set N (2 ,R , +)1 are responsible forthe regrouping of variable nodes at iteration , HR2-R1. We denote by p (2 ,R k the conditional probability that an edge is adjacent to a check node in N (2 ,R , +)1 given that 1) it emanates from an unverified variablenode 2) it is not adjacent to a check node in the set N (2 ,R ,C )1 . This is indeed the probability that a variablenode has an edge that increases its index. This probability is calculated as follows: p (2 ,R k = Pr[ c e ∈ N (2 ,R , +)1 | v e ∈ K (2) , c e / ∈ N (2 ,R ,C )1 ] , = Pr[ c e ∈ N (2 ,R , +)1 ] Pr[ v e ∈ K (2) | c e ∈ N (2 ,R , +)1 ] Pr[ c e / ∈ N (2 ,R ,C )1 | v e ∈ K (2) , c e ∈ N (2 ,R , +)1 ]Pr[ v e ∈ K (2) , c e / ∈ N (2 ,R ,C )1 ] , = Pr[ c e ∈ N (2 ,R , +)1 ] Pr[ v e ∈ K (2) | c e ∈ N (2 ,R , +)1 ] Pr[ c e / ∈ N (2 ,R ,C )1 | v e ∈ K (2) , c e ∈ N (2 ,R , +)1 ]Pr[ v e ∈ K (2) , c e ∈ (cid:83) d c i =2 N (2 ,R , i ∪ N (2 ,R , +)1 ] , = Pr[ c e ∈ N (2 ,R , +)1 ] Pr[ v e ∈ K (2) | c e ∈ N (2 ,R , +)1 ] Pr[ c e / ∈ N (2 ,R ,C )1 | v e ∈ K (2) , c e ∈ N (2 ,R , +)1 ]Pr[ c e ∈ N (2 ,R , +)1 ] Pr[ v e ∈ K (2) | c e ∈ N (2 ,R , +)1 ] + d c (cid:88) i =2 Pr[ N (2 ,R , i ] Pr[ v e ∈ K (2) |N (2 ,R , i ] , = p (2 ,R , +) N × /d c p (2 ,R , +) N × /d c + d c (cid:88) i =2 p (2 ,R , N i (cid:18) id c (cid:19) , = p (2 ,R , +) N p (2 ,R , +) N + d c (cid:88) i =2 ip (2 ,R , N i , where p (2 ,R , +) N denotes the probability that a check node belongs to the set N (2 ,R , +)1 . Since the two sets N (2 ,R , +)1 and N (2 ,R ,C )1 are disjoint and their union is the set N (2 ,R , , we have: p (2 ,R , +) N = d c (cid:88) j =2 d c − j (cid:88) i =0 p (1 ,R , N j,i p (1 ,R N j ↓ ,i , p (2 ,R ,C ) N = p (2 ,R , N − p (2 ,R , +) N . Hence, the probability p (2 ,R K i ↑ j , i ∈ { , } , that a variable node from K (1 ,R , i is regrouped into K (2 ,R , j , i ≤ j ≤ d v , is calculated as follows: p (2 ,R K i ↑ j = (cid:18) d v − ij − i (cid:19) (cid:16) p (2 ,R k (cid:17) j − i (cid:16) − p (2 ,R k (cid:17) d v − j , i = 0 , , j = i, · · · , d v . Finally, the probability p (2 ,R K j that a variable node in the support set belongs to the set K (2 ,R j is calculatedby: p (2 ,R K j = (cid:88) i =0 p (1 ,R , K i p (2 ,R K i ↑ j , j = 0 , · · · , d v . The probability p (2 ,R , K j that a variable node in the support set belongs to the set K (2 ,R , j , is calculatedbased on the set of verified variable nodes at this stage. Variable nodes in the set K (2 ,R j , ≤ j ≤ d v , areall verified. Variable nodes in the set K (2 ,R are left intact, and a fraction of the variable nodes in the set K (2 ,R are verified. The procedure to find this fraction is as follows. The set K (2 ,R consists of two sets of variable nodes: K (2 ,R ↑ and K (2 ,R ↑ . Variable nodes in the set K (2 ,R ↑ are neighbor to check nodes in the set N (2 ,R , +)1 , while variable nodes in the set K (2 ,R ↑ areneighbor to check nodes in the set N (2 ,R ,C )1 . Since the structure and evolution of the two sets N (2 ,R , +)1 and N (2 ,R ,C )1 are different, the sets K (2 ,R ↑ and K (2 ,R ↑ also evolve differently. The sets N (2 ,R , +)1 and N (2 ,R ,C )1 are formed at iteration , HR1-R2. We shall partition the two sets further into subsets N (2 ,R , +)1 ,j and N (2 ,R ,C )1 ,j , ≤ j ≤ d c − . At iteration , HR1-R1, the subsets are to be regrouped based on theirsecond index, just like any other set of check nodes. Hence, we have: p (2 ,R , , +) N ,k = d c − (cid:88) j = k p (1 ,R , , +) N ,j p (2 ,R N ,j ↓ k , k = 0 , · · · , d c − ,p (2 ,R , ,C ) N ,k = d c − (cid:88) j = k p (1 ,R , ,C ) N ,j p (2 ,R N ,j ↓ k , k = 0 , · · · , d c − , where, p (2 ,R N ,j ↓ k = (cid:18) jk (cid:19) (cid:16) − p (2 ,R E R (cid:17) j (cid:16) p (2 ,R E R (cid:17) j − k , p (2 ,R E R = 1 − p (2)∆ − p (1 ,R δ , and, p (1 ,R , , +) N ,j = d c (cid:88) k =2 p (1 ,R , N k,j p (1 ,R N k ↓ ,j ,p (1 ,R , ,C ) N ,j = p (1 ,R , N ,j p (1 ,R N ↓ ,j . Variable nodes in the two sets K (2 ,R ↑ and K (2 ,R ↑ are verified at iteration , HR2-R1, if and only if, theyare neighbor to check nodes in the sets N (2 ,R , , +)1 , and N (2 ,R , ,C )1 , , respectively. Therefore, the parameters f (2 ,R , +) and f (2 ,R ,C ) , defined as the respective fraction of variable nodes in K (2 ,R ↑ and K (2 ,R ↑ that areverified at iteration , HR2-R1, are calculated as follows: f (2 ,R , +) = p (2 ,R , , +) N , d c − (cid:88) k =0 p (2 ,R , , +) N ,k .f (2 ,R ,C ) = p (2 ,R , ,C ) N , d c − (cid:88) k =0 p (2 ,R , ,C ) N ,k . Finally, for the set of probabilities p (2 ,R , K j , we have: p (2 ,R , K = 1 N (2 ,R p (2 ,R K = 1 N (2 ,R p (1 ,R , K p (2 ,R K ↑ ,p (2 ,R , , +) K = 1 N (2 ,R p (1 ,R , K p (2 ,R K ↑ (cid:0) − f (2 ,R , +) (cid:1) ,p (2 ,R , ,C ) K = 1 N (2 ,R p (1 ,R , K p (2 ,R K ↑ (cid:0) − f (2 ,R ,C ) (cid:1) , p (2 ,R , K j = 0 , j = 2 , · · · , d v . The normalization factor N (2 ,R is used to make the set of parameters p (2 ,R , K i a valid probability measure,and is calculated by: N (2 ,R = p (1 ,R , K p (2 ,R K ↑ + p (1 ,R , K p (2 ,R K ↑ (cid:0) − f (2 ,R , +) (cid:1) + p (1 ,R , K p (2 ,R K ↑ (cid:0) − f (2 ,R ,C ) (cid:1) . The probability that a variable node belongs to the support set and remains unverified after iteration , α (3) , is calculated as follows: α (3) = α (2) (cid:16) p (1 ,R , K p (2 ,R K ↑ + p (1 ,R , K p (2 ,R K ↑ (cid:0) − f (2 ,R , +) (cid:1) + p (1 ,R , K p (2 ,R K ↑ (cid:0) − f (2 ,R ,C ) (cid:1)(cid:17) , = α (2) N (2 ,R . For iteration , HR1-R2, we find the probability p (1 ,R d (cid:54) =1 in the following. (For simplicity, some superscriptsare omitted. They appear when there is a risk of ambiguity.) p (2 ,R d (cid:54) =1 = Pr[ f e = 1 | v e ∈ K (2) , c e ∈ d c (cid:91) j =2 N (2 ,R , j ] , = 1 − Pr[ v e ∈ K (2 ,R , f e = 0 | v e ∈ K (2) , c e ∈ d c (cid:91) j =2 N j ] − Pr[ v e ∈ K (2 ,R , f e = 0 | v e ∈ K (2) , c e ∈ d c (cid:91) j =2 N j ] , = 1 − Pr[ v e ∈ K | v e ∈ K , c e ∈ d c (cid:91) j =2 N j ] − Pr[ v e ∈ K +1 | v e ∈ K , c e ∈ d c (cid:91) j =2 N j ] Pr[ f e = 0 | v e ∈ K +1 , v e ∈ K , c e ∈ d c (cid:91) j =2 N j ] − Pr[ v e ∈ K C | v e ∈ K , c e ∈ d c (cid:91) j =2 N j ] Pr[ f e = 0 | v e ∈ K C , v e ∈ K , c e ∈ d c (cid:91) j =2 N j ] , = 1 − Pr[ v e ∈ K | v e ∈ K ] Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K , v e ∈ K ]Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K ] − Pr[ v e ∈ K +1 | v e ∈ K ] Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K +1 , v e ∈ K ]Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K ] (cid:0) − f (2 ,R , +) (cid:1) − Pr[ v e ∈ K C | v e ∈ K ] Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K C , v e ∈ K ]Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K ] (cid:0) − f (2 ,R ,C ) (cid:1) , = 1 − Pr[ v e ∈ K | v e ∈ K ] Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K , v e ∈ K ]1 − Pr[ c e ∈ N | v e ∈ K ] − Pr[ v e ∈ K +1 | v e ∈ K ] Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K +1 , v e ∈ K ]1 − Pr[ c e ∈ N | v e ∈ K ] (cid:0) − f (2 ,R , +) (cid:1) − Pr[ v e ∈ K C | v e ∈ K ] Pr[ c e ∈ (cid:83) d c j =2 N j | v e ∈ K C , v e ∈ K ]1 − Pr[ c e ∈ N | v e ∈ K ] (cid:0) − f (2 ,R ,C ) (cid:1) , = 1 − p (1 ,R , K p (2 ,R K ↑ × − p (2 ,R − p (1 ,R , K p (2 ,R K ↑ (cid:18) d v − d v (cid:19) − p (2 ,R (cid:0) − f (2 ,R , +) (cid:1) − p (1 ,R , K p (2 ,R K ↑ (cid:18) d v − d v (cid:19) − p (2 ,R (cid:0) − f (2 ,R ,C ) (cid:1) . Hence, the probability p (2 ,R N i ↓ k,j that a check node belongs to the set of check nodes N (2 ,R i ↓ k,j is calculatedas follows: p (2 ,R N i ↓ k,j = (cid:18) ik (cid:19) (cid:16) p (2 ,R d (cid:54) =1 (cid:17) i − k (cid:16) − p (2 ,R d (cid:54) =1 (cid:17) k , i = 2 , · · · , d c , k = 0 , · · · , i, j = 0 , · · · , d c − i. Note that the probability p (2 ,R is calculated by: p (2 ,R = p (2 ,R , , +) N + p (2 ,R , ,C ) N α (2) d c . As we explain in the following, the evolution of the sets N (2 ,R , , +)1 and N (2 ,R , ,C )1 is a bit more involved.A variable node in the set K (2 ,R ↑ i , ≤ i ≤ d v , has i neighboring check nodes in the set N (2 ,R , , +)1 .On the other hand, a variable node in the set K (2 ,R ↑ i , ≤ i ≤ d v , has neighboring check node in theset N (2 ,R , ,C )1 and i − neighboring check nodes in the set N (2 ,R , , +)1 . Now let us consider a checknode c ∈ N (2 ,R , , +)1 . Suppose, c is neighbor to a variable node v ∈ K (2 ,R ↑ . Variable node v is verifiedif and only if c belongs to the subset N (2 ,R , , +)1 , . Hence, c is regrouped as a zero-valued check nodeif it belongs to the set N (2 ,R , , +)1 , . Now, suppose c is neighbor to a variable node v (cid:48) ∈ (cid:83) d v i =2 K (2 ,R ↑ i , or v (cid:48) ∈ (cid:83) d v i =2 K (2 ,R ↑ i . Since the variable node v (cid:48) is verified with probability , check node c is regrouped intothe set of zero-valued check nodes with probability as well. A similar argument holds true for the setof check nodes in N (2 ,R , ,C )1 and variable nodes in K (2 ,R ↑ . Therefore, to regroup check nodes in the sets N (2 ,R , , +)1 and N (2 ,R , ,C )1 , we need to divide them further based on their neighbors; i.e., whether or notthey are neighbor to variable nodes in the sets K (2 ,R ↑ and K (2 ,R ↑ , respectively.We partition the set N (2 ,R , , +)1 into subsets N (2 ,R , + ,O )1 and N (2 ,R , + ,F )1 . We also partition the set N (2 ,R , ,C )1 into subsets N (2 ,R ,C,O )1 and N (2 ,R ,C,F )1 . Check nodes in sets N (2 ,R , + ,F )1 and N (2 ,R ,C,F )1 are neighbor tovariable nodes in sets K (2 ,R ↑ and K (2 ,R ↑ , respectively. Any other check node in the set N (2 ,R , , +)1 , notbeing part of the set N (2 ,R , + ,F )1 is grouped into the set N (2 ,R , + ,O )1 . Similarly, any other check node inthe set N (2 ,R , ,C )1 , not being part of the set N (2 ,R ,C,F )1 is grouped into the set N (2 ,R ,C,O )1 .Let p (2 ,R , + ,O ) N and p (2 ,R , + ,F ) N denote the probabilities that a check node belongs to sets N (2 ,R , + ,O )1 and N (2 ,R , + ,F )1 , respectively. Probabilities p (2 ,R ,C,O ) N and p (2 ,R ,C,F ) N are defined similarly. The calculation of the probability p (2 ,R , + ,F ) N ,i , ≤ i ≤ d c − , follows: p (2 ,R , + ,F ) N ,i = Pr[ c ∈ N (2 ,R , + ,F )1 ,i ] , = Pr[ c ∈ N (2 ,R , , +)1 ,i ] Pr[ c ∈ N (2 ,R , + ,F )1 ,i | c ∈ N (2 ,R , , +)1 ,i ] , = Pr[ c ∈ N (2 ,R , , +)1 ,i ] Pr[ v e ∈ K (2 ,R ↑ | c e ∈ N (2 ,R , , +)1 ,i , v e ∈ K (2) ] , = Pr[ c ∈ N (2 ,R , , +)1 ,i ] Pr[ v e ∈ K (2 ,R ↑ | v e ∈ K (2) ] Pr[ c e ∈ N (2 ,R , , +)1 ,i | v e ∈ K (2 ,R ↑ , v e ∈ K (2) ]Pr[ c e ∈ N (2 ,R , , +)1 ,i | v e ∈ K (2) ] , = Pr[ c ∈ N (2 ,R , , +)1 ,i ] Pr[ v e ∈ K (2 ,R ↑ | v e ∈ K (2) ] Pr[ c e ∈ N (2 ,R , , +)1 ,i | v e ∈ K (2 ,R ↑ , v e ∈ K (2) ] A + B , where, A = d v (cid:88) j =1 Pr[ v e ∈ K (2 ,R ↑ j , c e ∈ N (2 ,R , , +)1 ,i | v e ∈ K (2) ] , = d v (cid:88) j =1 jp (1 ,R , K p (2 ,R K ↑ j ,B = d v (cid:88) j =2 Pr[ v e ∈ K (2 ,R ↑ j , c e ∈ N (2 ,R , , +)1 ,i | v e ∈ K (2) ] , = d v (cid:88) j =2 ( j − p (1 ,R , K p (2 ,R K ↑ j . Hence, p (2 ,R , + ,F ) N ,i = p (2 ,R , , +) N ,i p (1 ,R , K p (2 ,R K ↑ d v (cid:88) j =1 jp (1 ,R , K p (2 ,R K ↑ j + d v (cid:88) j =2 ( j − p (1 ,R , K p (2 ,R K ↑ j ,p (2 ,R , + ,O ) N ,i = p (2 ,R , , +) N ,i − p (2 ,R , + ,F ) N ,i . Following a similar approach, we have: p (2 ,R ,C,F ) N ,i = p (2 ,R , ,C ) N ,i p (2 ,R K ↑ d v (cid:88) j =1 p (2 ,R K ↑ j ,p (2 ,R ,C,O ) N ,i = p (2 ,R , ,C ) N ,i − p (2 ,R ,C,F ) N ,i . Since check nodes in the sets N (2 ,R , + ,O )1 , N (2 ,R ,C,O )1 , N (2 ,R , + ,F )1 , , and N (2 ,R ,C,F )1 , receive d c verifiedmessages at iteration , HR1-R2, all such check nodes are grouped into the set N (2 ,R , . On the other hand, check nodes in the sets N (2 ,R , + ,F )1 ,i , and N (2 ,R ,C,F )1 ,i , ≤ i ≤ d c − , are grouped into the sets N (2 ,R , ,C )1 ,i . After the regrouping, the probability p (2 ,R , N k,j that a check node belongs to the set N (2 ,R , k,j iscalculated as follows: p (2 ,R , N k,j = d c (cid:88) i = k p (2 ,R , N i,j p (2 ,R N i ↓ k,j , k = 2 , · · · , d c , j = 0 , · · · , d c − i,p (2 ,R , , +) N ,j = d c (cid:88) i =2 p (2 ,R , N i,j p (2 ,R N i ↓ ,j , j = 0 , · · · , d c − ,p (2 ,R , ,C ) N ,j = p (2 ,R ,C,F ) N ,j + p (2 ,R , + ,F ) N ,j , j = 1 , · · · , d c − ,p (2 ,R , N ,j = p (2 ,R , N ,j + p (2 ,R ,C,O ) N ,j + p (2 ,R , + ,O ) N ,j + d c (cid:88) i =2 p (2 ,R , N i,j p (2 ,R N i ↓ ,j , j = 0 , · · · , d c − . As previously defined, the probability p (2 ,R δ , is needed to find the probability that a zero-valued variablenode is verified at this stage. This probability is calculated as follows: p (2 ,R δ = d c − (cid:88) j =1 Pr[ c e ∈ N (2 ,R , ,j | v e ∈ ∆ (2) ] , = d c − (cid:88) j =1 Pr[ c e ∈ N (2 ,R , ,j ] Pr[ v e ∈ ∆ (2) | c e ∈ N (2 ,R , ,j ] d c (cid:88) i =0 d c − (cid:88) j =1 Pr[ c e ∈ N (2 ,R , i,j ] Pr[ v e ∈ ∆ (2) | c e ∈ N (2 ,R , i,j ] , = d c − (cid:88) j =1 p (2 ,R , N ,j (cid:18) jd c (cid:19) d c (cid:88) i =0 d c − (cid:88) j =1 p (2 ,R , N i,j (cid:18) jd c (cid:19) , = d c − (cid:88) j =1 j p (2 ,R , N ,j d c (cid:88) i =0 d c − (cid:88) j =1 jp (2 ,R , N i,j . Note that the denominator is indeed
Pr[ v e ∈ ∆ (2) ] (cid:44) p (2)∆ . Hence, the probability p (2 ,R , i , defined asthe probability that an unverified zero-valued variable node belongs to the set ∆ (2 ,R , i , is calculated asfollows: p (2 ,R , i = (cid:18) d v i (cid:19) (cid:16) p (2 ,R δ (cid:17) i (cid:16) − p (2 ,R δ (cid:17) d v − i , i = 0 , · · · , d v . Lastly, the probability that a variable node is zero-valued and remains unverified for iteration is givenby: p (3)∆ = p (2)∆ p (2 ,R , . The analysis of an iteration (cid:96) , (cid:96) ≥ , is similar to that of iteration 2. The update rules for a genericiteration (cid:96) , (cid:96) ≥ is given in subsection V-E. A PPENDIX
IVD
ETAILS OF C ONCENTRATION R ESULTS
A. Probability of Tree-Like Neighborhood
Consider a particular variable node v . We enumerate M (cid:96) and C (cid:96) , respectively, the number of variablenodes and check nodes in N (cid:96)v under the assumption that N (cid:96)v is tree-like, as follows: M (cid:96) = 1 + d v ( d c − (cid:96) (cid:88) i =1 ( d c − i − ( d v − i − , and C (cid:96) = d v (cid:96) (cid:88) i =1 ( d c − i − ( d v − i − . Recall that m and n are the number of check nodes and the variable nodes, respectively. Fix (cid:96) ∗ , and let (cid:96) < (cid:96) ∗ . Assuming that N (cid:96)v is tree-like, the probability that N (cid:96) +1 v is tree-like is lower bounded by (cid:18) − C (cid:96) ∗ m (cid:19) C (cid:96) +1 − C (cid:96) , and assuming that N (cid:96) +1 v is tree-like, the probability that N (cid:96) +2 v is tree-like is lower bounded by (cid:18) − M (cid:96) ∗ n (cid:19) M (cid:96) +1 − M (cid:96) , for sufficiently large n (see Appendix A in [35]). Thus, the probability that N (cid:96) ∗ v is tree-like is boundedfrom below by (lower bounding is done by using the chain rule) (cid:18) − M (cid:96) ∗ n (cid:19) M ∗ (cid:96) (cid:18) − C (cid:96) ∗ m (cid:19) C ∗ (cid:96) , for sufficiently large n , and since (1 − M (cid:96) ∗ /n ) M (cid:96) ∗ ≥ (1 − M (cid:96) ∗ /n ) , and (1 − C (cid:96) ∗ /m ) C (cid:96) ∗ ≥ (1 − C (cid:96) ∗ /m ) for large n , then Pr[ N (cid:96) ∗ v is not tree-like ] ≤ − (cid:18) − M (cid:96) ∗ n (cid:19) (cid:18) − C (cid:96) ∗ m (cid:19) ≤ M (cid:96) ∗ n + C (cid:96) ∗ m = M (cid:96) ∗ + C (cid:96) ∗ ( d c /d v ) n . Taking γ = M (cid:96) ∗ + C (cid:96) ∗ ( d c /d v ) , we see that γ is a constant not depending on n , and thus, Pr[ N (cid:96) ∗ v is not tree-like ] ≤ γn . B. Derivation of Bounds on the Difference between Two Consecutive Elements in Martingale Sequences
Consider two realizations T (cid:48) and T (cid:48)(cid:48) from the ensemble of all graphs, inputs, and weights. We considertwo cases: • Case I: Realizations T (cid:48) and T (cid:48)(cid:48) are the same except for the value of a variable node. • Case II: Realizations T (cid:48) and T (cid:48)(cid:48) are the same except for the connection of two variable nodes v and v to two check nodes c and c ; i.e., T (cid:48) : c ∈ M ( v ) , c ∈ M ( v ) and T (cid:48)(cid:48) : c ∈ M ( v ) , c ∈ M ( v ) . • Case III: Realizations T (cid:48) and T (cid:48)(cid:48) are the same except for one weight of an edge in the graph.Suppose the difference between the two realizations results in the difference of N ( (cid:96) ) in verified variablenodes at iteration (cid:96) . The goal in this appendix is to find an upper bound on N ( (cid:96) ) for Genie, LM and SBBalgorithms. Without loss of generality, suppose that realization T (cid:48) verifies less number of variable nodescompared to T (cid:48)(cid:48) . As we are seeking an upper bound, we assume the worst configuration for realization T (cid:48) and the best configuration for T (cid:48)(cid:48) . The realization T (cid:48) being the worst configuration implies that thevariable node/edge/weight that is the difference between the two realizations T (cid:48) and T (cid:48)(cid:48) results in noverification of variable nodes up to iteration (cid:96) under consideration for T (cid:48) . On the other hand, we assumethat the configuration in T (cid:48)(cid:48) is so that the verification of a variable node at an iteration, results in themaximum number of verifications in the next iteration. With these configurations for T (cid:48) and T (cid:48)(cid:48) , in orderto maximize the difference between the verified variable nodes between the two realizations at iteration (cid:96) , we need to maximize the number of variable nodes that can be verified at iteration .Let the parameters E ( (cid:96) ) , D ( (cid:96) ) , Z ( (cid:96) ) denote the difference in the number of variable nodes verified atiteration (cid:96) between the two realizations due to the ECN, D1CN, and ZCN verification rules, respectively.Therefore, N ( (cid:96) ) = E ( (cid:96) ) + D ( (cid:96) ) + Z ( (cid:96) ) .In what fallows we find an upper bound for N ( (cid:96) ) in the case of the Genie algorithm. A similar reasoningcan be used for the other algorithms.
1) Genie:
The only verification rule applied to the Genie is D1CN. We find the maximum N ( (cid:96) ) for eachof the three cases discussed above. Focusing on case I, three possibilities exist:1) A variable node in T (cid:48) is non-zero, while the same variable node in T (cid:48)(cid:48) is zero.2) A variable node in T (cid:48) is zero, while the same variable node in T (cid:48)(cid:48) is non-zero.3) A variable node in both T (cid:48) and T (cid:48)(cid:48) is non-zero, but with two different values.In the first scenario, the worst configuration is such that the variable node under consideration remainsunverified up to iteration (cid:96) . As the corresponding variable node is zero in realization T (cid:48)(cid:48) , this means thatthe d v neighboring check nodes have degrees smaller by one compared to their counterparts in realization T (cid:48) . The best configuration T (cid:48)(cid:48) is then formed if each one of these d v check nodes has degree 1. Each such check node results in the verification of one variable node with the D1CN rule. So, N (0) ≤ d v .A variable node verified based on D1CN at iteration i − ( ≤ i ≤ (cid:96) ) can reduce the degree of at most d v − check nodes. In the best case, each such check node has degree which results in the verificationof another variable node at iteration i . Therefore, we have: D ( i ) ≤ D ( i − d v − , which results in N ( (cid:96) ) ≤ d v ( d v − (cid:96) , (cid:96) ≥ . In the second and third scenarios, considering the fact that T (cid:48) is the worst configuration and that therealizations T (cid:48) and T (cid:48)(cid:48) have the same weighted graph and the same input vector (except for the variablenode under consideration), the realization T (cid:48)(cid:48) can not verify more variable nodes than T (cid:48) . Therefore, inthis case, N ( (cid:96) ) = 0 . This is assuming that no false verification happens in either realizations.Now we consider Case II. Possible scenarios for the values of v and v are as follows:1) v = 0 , v (cid:54) = 0 (due to the symmetry, this is the same as v (cid:54) = 0 , v = 0 ),2) v = 0 , v = 0 ,3) v (cid:54) = 0 , v (cid:54) = 0 .To make the worst realization, we assume that all the neighbors of c are zero-valued variable nodes, andwe let c to be neighbor to only one other non-zero variable node, say v . In this realization, as c hasdegree 2, then v and v can not be verified in iteration zero. However, when we switch the connections, c and c will both have degree 1. Therefore, both variable nodes v and v are verified based on theD1CN rule. So, in this scenario N (0) = 2 . Again, to find the maximum number of variable nodes thatcan be verified in further iterations, we assume that the verification of each variable node at iteration i − results in the verification of d v − other variable nodes at iteration i . We thus have: D ( i ) ≤ D ( i − d v − , which results in N ( (cid:96) ) ≤ d v − (cid:96) , (cid:96) ≥ . Due to the symmetry of the problem, the other two scenarios result in no difference in the number ofverified variable nodes.For Case III, the weights are, by definition, non-zero. Thus, the change in the weight of an edge in thegraph, has no effect on the recovery of variable nodes. Based on the above discussions, we have the following upper bound for the Genie algorithm: N ( (cid:96) ) ≤ d v ( d v − ( (cid:96) ) , (cid:96) ≥ .
2) LM:
This algorithm applies D1CN and ZCN verification rules in the first and second half-rounds ofeach iteration. Following the same steps as those for the Genie algorithm, one can show that the maximum N ( (cid:96) ) between all possible configurations for cases I, II, and III is achieved when a variable node changesits value from a non-zero value to a zero value. With the same logic as in the Genie, d v variable nodescan be verified with the D1CN rule at iteration in T (cid:48)(cid:48) that can not be verified in T (cid:48) . We thus have N (0) = d v . When a variable node is verified based on D1CN at the first half-round of iteration i − , at most d v − check nodes can have a value equal to zero, each of which results in the verification of d c − variablenodes with the ZCN rule in the second half-round of the same iteration.On the other hand, when a variable node is verified based on ZCN at the second half-round of iteration i − , at most d v − check nodes can have a degree one for the first half-round of the next iteration. So, d v − variable nodes can be verified using the D1CN rule at iteration i .Putting these two steps together, we form the following recursive formulas: D ( i ) ≤ Z ( i − d v − ,Z ( i ) ≤ D ( i )( d v − d c − . Solving the recursions with the initial condition Z (0) = d v , we have: Z ( (cid:96) ) ≤ d v ( d v − (cid:96) ( d c − (cid:96) ,D ( (cid:96) ) ≤ d v ( d v − (cid:96) − ( d c − (cid:96) − ,N ( (cid:96) ) = Z ( (cid:96) ) + D ( (cid:96) ) ≤ Z ( (cid:96) ) = 2 d v ( d v − (cid:96) ( d c − (cid:96) .
3) SBB:
This algorithm applies all ZCN, D1CN, and ECN verification rules. In order to find the upperbound on N ( (cid:96) ) , we first find the number of variable nodes verified at a generic iteration i due to theverification of a variable node at iteration i − based on each verification rule, separately. Then we findthe maximum number of variable nodes that can possibly be verified at iteration zero based on each ofthe three Cases I, II, or III and use them as initial conditions to solve recursive formulas like the oneswe saw for LM.Assume two check nodes with the same value result in a variable node being verified according to the ECN rule. Therefore, the two check nodes result in the verification of d c − variable nodes in total withZCN verification rule in the next round of the same iteration. Also, the other d v − adjacent check nodesface a reduction in the degree as well as change in the value. Therefore, they can result in the verificationof ( d v − d c − variable nodes with ZCN in the next round of the same iteration, or the verificationof d v − variable nodes based on the D1CN or ECN in the next iteration. One may be able to find outthe best combination of rules that results in the maximum number of verified variable nodes. However,as we are interested in finding an upper bound, we assume that all the events above can happen at thesame time.If a variable node is verified according to the ZCN rule, the d v − adjacent check nodes do not face achange in their values, and thus can not contribute to the verification of other variable nodes based onECN or ZCN. Therefore, each such check node verifies variable node based on the D1CN rule.When a variable node is verified according to the D1CN rule, d v − check nodes face a reduction indegree as well as change in value. With the same reasoning as that used in the ECN case, each suchcheck node results in the verification of d c − , , and variable nodes based on the ZCN, D1CN, andECN rules, respectively.Putting everything together, we have: E ( i ) ≤ E ( i − d v −
2) + D ( i − d v − ,D ( i ) ≤ Z ( i − d v −
1) + D ( i − d v −
1) + E ( i − d v − ,Z ( i ) ≤ E ( i )( d v )( d c −
1) + D ( i )( d v − d c − . Using the third inequality, we can rewrite the first two inequalities as follows: E ( i ) ≤ E ( i − d v −
2) + D ( i − d v − ≤ d v E ( i −
1) + d v D ( i − ,D ( i ) ≤ E ( i −
1) ( d v ( d v − d c −
1) + ( d v − D ( i − d v −
1) (( d v − d c − , ≤ ( d v d c + d v ) E ( i −
1) + d v d c D ( i − . Looking for an upper bound, we assume that the inequalities are satisfied with equalities. Therefore, wehave: E ( i ) = d v E ( i −
1) + d v D ( i − ,D ( i ) = ( d v d c + d v ) E ( i −
1) + d v d c D ( i −
1) = d v d c E ( i ) + d v E ( i − . Replacing D ( i − of the first equality by its value from the second one, we get: E ( i ) = (cid:0) d v + d v d c (cid:1) E ( i −
1) + d v E ( i − . This is a second order linear homogeneous recurrence relation. Using the characteristic polynomialtechnique, a closed form solution can be found. To find a more elegant solution, however, we derivethe following upper bound on E ( i ) . E ( i ) = (cid:0) d v + d v d c (cid:1) E ( i −
1) + d v E ( i − ≤ (cid:0) d v + d v d c (cid:1) E ( i −
1) + (3 d v + 2 d v d c )4 E ( i − . Considering the inequality as equality, and using the characteristic polynomial technique, we then obtain E ( i ) = K (cid:18) d v d c + 32 d v (cid:19) i + K (cid:18) − d v (cid:19) i , for some K and K that depend on the initial conditions. We can simplify this even further as follows: E ( i ) ≤ K (cid:0) d v d c + 2 d v (cid:1) i . By replacing this upper bound in D ( i ) and Z ( i ) , we get: D ( i ) ≤ d v d c K (cid:0) d v d c + 2 d v (cid:1) i + d v K (cid:0) d v d c + 2 d v (cid:1) i − ≤ d v d c K (cid:0) d v d c + 2 d v (cid:1) i ,Z ( i ) ≤ d v d c ( E ( i ) + D ( i )) = d v d c K (cid:0) d v d c + 2 d v (cid:1) i (1 + 2 d v d c ) , and thus N ( i ) = E ( i ) + D ( i ) + Z ( i ) ≤ ( d v d c + 1) K (cid:0) d v d c + 2 d v (cid:1) i (1 + 2 d v d c ) , ≤ K (cid:0) d v d c + 2 d v (cid:1) i +2 , where K is the maximum number of variable nodes that can be verified at iteration for the Cases I,II, and III. Once again, one can show that the maximum is achieved when a variable node changes valuefrom a non-zero value to zero, and where d v variable nodes are verified using the ZCN rule. Therefore,we have K = d v . Hence, N ( (cid:96) ) ≤ d v (cid:0) d v d c + 2 d v (cid:1) (cid:96) +2+2