On Syndrome Decoding for Slepian-Wolf Coding Based on Convolutional and Turbo Codes
aa r X i v : . [ c s . I T ] A p r On Syndrome Decoding for Slepian-Wolf CodingBased on Convolutional and Turbo Codes
Lorenzo Cappellari,
Member, IEEE
Abstract —In source coding, either with or without side infor-mation at the decoder, the ultimate performance can be achievedby means of random binning. Structured binning into cosetsof performing channel codes has been successfully employed inpractical applications. In this letter it is formally shown thatvarious convolutional- and turbo-syndrome decoding algorithmsproposed in literature lead in fact to the same estimate. Anequivalent implementation is also delineated by directly tacklingsyndrome decoding as a maximum a posteriori probabilityproblem and solving it by means of iterative message-passing.This solution takes advantage of the exact same structures andalgorithms used by the conventional channel decoder for the codeaccording to which the syndrome is formed.
Index Terms —Slepian-Wolf coding, source coding, syndrome-based binning, turbo codes, message-passing.
I. I
NTRODUCTION
Several practical schemes for lossless source coding withside information at the decoder directly embody the optimalbinning scheme given by Wyner in [1]. In practice, the syn-drome with respect to a turbo (or an LDPC) code that is a good channel code for the fictitious correlation channel betweenthe source and the side information is used as compressedrepresentation of a given realization. At the decoder the bestestimate is found into the coset signalled by this syndrome.In this letter, the focus is on the schemes based on convo-lutional and (parallel) turbo codes [2], [3], [4], [5]. All theseapproaches use in fact a systematic syndrome former as sourceencoder, since this is a computationally efficient solution forsyndrome formation w.r.t. these codes, in particular w.r.t. turbocodes. Decoding is instead performed with ad-hoc algorithms.Despite this, here it is shown that the resulting estimate is thesame in all cases and could be found by simply performing a maximum a posteriori probability (MAP) search.II. C
HANNEL C ODING WITH C ONVOLUTIONAL AND T URBO C ODES
In channel coding based on an ( n, k ) convolutional codeover GF ( q ) , encoding is often realized by the systematic generator G ( D ) = [ I k | P ( D )] . The actual structure of the trellis section that realizes the k -in ( n − k ) -out system P ( D ) is described by the function χ σ i σ i − ( x si , x pi ) , which indicatesif at any time i the transition from state σ i − to state σ i ispossible with the systematic k -tuple x si as input and the parity ( n − k ) -tuple x pi as output. Manuscript received February 17, 2010. The associate editor coordinatingthe review of this letter and approving it for publication was Prof. Z. Yan.L. Cappellari is with the Dept. of Information Engineering, University ofPadova, 35131 Padova, Italy (email: [email protected]).
PSfrag replacements σ i − σ i α i − α i β i − β i x pi x si µ i µ i λ i λ i χ (a) PSfrag replacements σ i − σ i x pi x si ˆ µ i ˆ λ i χζ { s i = x pi − p i } p i s i p ( x pi | y pi ) p ( r i | s i ) (b)Fig. 1. A portion of the factor-graph used for decoding (a) convolutionalcodes and (b) systematic syndromes. The symbol-by-symbol MAP decoder evaluates arg max x si p ( x si | y s ( D ) , y p ( D )) , where y s ( D ) and y p ( D ) are the received sequences corresponding to the systematicand to the parity sequence, respectively, sent by the encoder( formal vector power series are defined as x ( D ) , P i x i D i ,where x i are row vectors). If the transmission channel is memoryless , i.e. is defined by p ( y si | x si ) (and by p ( y pi | x pi ) ),and the incoming symbols into the encoder are i.i.d. , p ( x si | y s ( D ) , y p ( D )) can be efficiently found using theBCJR algorithm, that is by means of message-passing marginalization over the factor-graph shown (accordingto the convention of [6]) in Fig. 1(a). For example, atthe i -th step of the forward phase the input messages µ i ( x si ) = p ( x si ) p ( y si | x si ) ∝ p ( x si | y si ) and µ i ( x pi ) = p ( y pi | x pi ) are used to evaluate the forward metric α i ( σ i ) = X ∼ σ i χ σ i σ i − ( x si , x pi ) α i − ( σ i − ) µ i ( x si ) µ i ( x pi ) ( P ∼ x indicates the sum w.r.t. all involved variables except x ).Once also the backward metric β i ( σ i ) is available, the optimalchoice at time i is found maximizing µ i ( x si ) λ i ( x si ) , where λ i ( x si ) = X ∼ x si χ σ i σ i − ( x si , x pi ) α i − ( σ i − ) β i ( σ i ) µ i ( x pi ) . In case of turbo coding, two ( n j , k ) systematic convolu-tional encoders are used ( j = 0 , ). MAP decoding is approx-imated with an iterative procedure that alternatively decodesthe two received parity sequences. The input messages fordecoding one code are taken from the output messages relativeto the other one; in particular, µ ( j ) i ( x si ) ∝ p ( x si | y si ) λ (1 − j ) i ( x si ) .III. S YNDROME -B ASED S OURCE C ODING
Given a linear code, any corresponding parity-check matrix defines a source encoder as it bins any source sequence according to its syndrome . In correspondence of an ( n, k ) con-volutional code, the systematic parity-check matrix H ( D ) =[ − P T ( D ) | I n − k ] represents the most straightforward choicefor this purpose. In fact, if the source sequence (of i.i.d. n -tuples over GF ( q ) ) x ( D ) is broken into the sequences x s ( D ) and x p ( D ) composed by k - and ( n − k ) -tuples such that x i = [ x si | x pi ] , then the output of the n -in ( n − k ) -out encoder H T ( D ) is s ( D ) = x ( D ) H T ( D ) = x p ( D ) − p ( D ) , where p ( D ) = x s ( D ) P ( D ) is the output from P ( D ) with x s ( D ) as input; with this choice, the source encoder can re-use thesame algorithm used in a systematic channel encoder.In case of turbo codes, up to symbol reordering, thechannel encoder realizes a block-based transformation G =[ I Nk | P | P ] where P j has N ( n j − k ) outputs. If the sourcesequence (of N ( n + n − k ) symbols) x is suitably broken intothe sequences x s , x , and x composed by N k , N ( n − k ) , and N ( n − k ) symbols, the systematic turbo-syndrome s = [ s | s ] can be evaluated as the difference between [ x | x ] and the turbo-parity corresponding to x s , obtained with a conventionalturbo encoder. Each syndrome component s j acts as a system-atic syndrome for the corresponding constituent code.Several algorithms have been proposed in literature forsystematic syndrome decoding; if a side information sequence y ( D ) is present, they usually assume that there exists a mem-oryless correlation channel between x ( D ) and y ( D ) definedby p ( y i | x i ) . As shown below, despite they seem different,they actually obtain the exact same source reconstruction; thisholds for both convolutional and turbo codes. A. Principal and Complementary Trellises
In syndrome decoding, differently from channel decoding,the side information sequence must be decoded over thecoset of the original code signalled by the syndrome. In [2]each other trellis section structure is modified for taking intoaccount this fact. In particular, at time i , the actual structureis described by χ σ i σ i − ( x si , x pi ; i ) , χ σ i σ i − ( x si , x pi − s i ) . In practice, if s i = 0 the BCJR algorithm uses the principal (original) trellis, otherwise it uses one over q n − k − comple-mentary trellises that share the same state transitions of theoriginal one but have different branch labels. Each input mes-sage µ i ( x pi ) is modified too in order to take into account theknown prior probability. More precisely, the input messagesare µ ′ i ( x si ) = µ i ( x si ) and µ ′ i ( x pi ) = p ( x pi ) µ i ( x pi ) ∝ p ( x pi | y pi ) .The forward metrics and output messages are now α ′ i ( σ i ) = X ∼ σ i χ σ i σ i − ( x si , x pi ; i ) α ′ i − ( σ i − ) µ ′ i ( x si ) µ ′ i ( x pi ) λ ′ i ( x si ) = X ∼ x si χ σ i σ i − ( x si , x pi ; i ) α ′ i − ( σ i − ) β ′ i ( σ i ) µ ′ i ( x pi ) . The output messages λ ′ i ( x pi ) are also computed to estimate themost probable x pi as the one maximizing µ ′ i ( x pi ) λ ′ i ( x pi ) . B. Inverse Syndrome Formation
Instead of modifying the trellis structure, in [3] a codeword c ( D ) belonging to the signalled coset is formed and subtracted from the side information prior to decoding over the originalcode. The source sequence equals x ( D ) = c ( D ) + c ( D ) forsome codeword c ( D ) belonging to the original code. Since y ( D ) − c ( D ) = c ( D ) + [ y ( D ) − x ( D )] and the code istailored for the correlation channel (assumed linear), c ( D ) can be in principle found decoding this difference; eventually,the sum with c ( D ) permits to reconstruct the source sequence.In practice, in correspondence of the (systematic) syndrome s ( D ) a suitable c ( D ) is the one such that c i = [ | s i ] . Conse-quently, the actual input messages to the BCJR algorithm are µ ′′ i ( x si ) ∝ p ( x si | y si − ) ∝ µ ′ i ( x si ) and µ ′′ i ( x pi ) ∝ p ( x pi | y pi − s i ) ;if the backward correlation channel (i.e. the one between theside information and the source) is linear, then µ ′′ i ( x pi ) ∝ p ( x pi + s i | y pi ) ∝ µ ′ i ( x pi + s i ) .Despite the different strategy, the forward and the backwardmetrics are proportional to the ones evaluated in [2] . Forexample, in fact, assuming α ′′ i − ( σ i − ) ∝ α ′ i − ( σ i − ) , α ′′ i ( σ i ) = X ∼ σ i χ σ i σ i − ( x si , x pi ) α ′′ i − ( σ i − ) µ ′′ i ( x si ) µ ′′ i ( x pi ) == X ∼ ( σi s i ) χ σ i σ i − ( x si , x pi − s i ) α ′′ i − ( σ i − ) µ ′′ i ( x si ) µ ′′ i ( x pi − s i ) ∝∝ X ∼ σ i χ σ i σ i − ( x si , x pi ; i ) α ′ i − ( σ i − ) µ ′ i ( x si ) µ ′ i ( x pi ) = α ′ i ( σ i ) . Similarly, the translated output messages λ ′′ i ( x si − ) and λ ′′ i ( x pi − s i ) are proportional to λ ′ i ( x si ) and λ ′ i ( x pi ) , respec-tively, so that the sum of the MAP estimate that the algorithmfinds for c ( D ) with c ( D ) coincides exactly with the MAPestimate evaluated in [2] . C. Decoding Under Parity Perspective
In [4], it is correctly observed that the syndrome re-ceived by the decoder is also a parity message. In particular,the systematic syndrome corresponds to the parity obtainedwith the systematic generator G ′ ( D ) = [ I n | P ′ ( D )] , where P ′ ( D ) = H ( D ) T . The source is then recovered by jointlydecoding the side information and the received parity messagewith the corresponding channel decoder.A realization of the system P ′ ( D ) can be easily derivedfrom a realization of P ( D ) . More precisely, its trellis sectionstructure can be described by ξ σ i σ i − ( x i , s i ) , χ σ i σ i − ( x si , x pi − s i ) , where x si and x pi are a k -tuple and an ( n − k ) -tupleobtained from x i as above; ξ σ i σ i − ( x i , s i ) indicates if at anytime i the transition from state σ i − to state σ i is possiblewith the source n -tuple x i as input and the syndrome ( n − k ) -tuple s i as output. As a remark, in correspondence of eachtransition of the original trellis, this trellis has q n − k paralleltransitions that share the same sub-label x si ; in practice, thistrellis coincides with the source coding trellis of [2]. Source and side information must be defined over the same alphabet. In this treatment, it is implied that all initial metrics α × ( σ ) and β × N ( σ N ) ( N is the number of vectors) are the same, apart from some scaling factors. As x pi takes on a finite group, P ∼ x f ( x, x pi ) = P ∼ x, s i f ( x, x pi − s i ) . In [3], it is not clear if λ ′′ i ( x pi ) is actually maximized or rather a sub-optimal estimate of the parity portion x p ( D ) is found as the output of P ( D ) with the MAP estimate of the systematic portion x s ( D ) as input, in whichcase the resulting estimate of c ( D ) would be a real codeword. The BCJR algorithm takes now µ ′′′ i ( x i ) = p ( x i ) p ( y i | x i ) ∝ p ( x i | y i ) ∝ µ ′ i ( x si ) µ ′ i ( x pi ) and µ ′′′ i ( s ′ i ) = p ( r i | s ′ i ) as inputs,where p ( r i | s ′ i ) describes the memoryless channel between thesyndrome sequence sent and the one received r ( D ) . Again, ifthis channel is error free, i.e. µ ′′′ i ( s ′ i ) is non zero if and only if s ′ i equals the actual syndrome s i , the forward and the backwardmetrics involved in decoding equal the ones evaluated in [2].For example, in fact, assuming α ′′′ i − ( σ i − ) ∝ α ′ i − ( σ i − ) , α ′′′ i ( σ i ) = X ∼ σ i ξ σ i σ i − ( x i , s ′ i ) α ′′′ i − ( σ i − ) µ ′′′ i ( x i ) µ ′′′ i ( s ′ i ) ∝∝ X ∼ ( σi s i ) χ σ i σ i − ( x si , x pi − s i ) α ′′′ i − ( σ i − ) µ ′′′ i ( x i ) ∝∝ X ∼ σ i χ σ i σ i − ( x si , x pi ; i ) α ′ i − ( σ i − ) µ ′ i ( x si ) µ ′ i ( x pi ) = α ′ i ( σ i ) . In addition, the MAP estimate for x si is found maximizing P ∼ x si µ ′′′ i ( x i ) λ ′′′ i ( x i ) ∝ µ ′ i ( x si ) P ∼ x si µ ′ i ( x pi ) λ ′′′ i ( x i ) , wherethe output message λ ′′′ i ( x i ) satisfies λ ′′′ i ( x i ) = X ∼ x i ξ σ i σ i − ( x i , s ′ i ) α ′′′ i − ( σ i − ) β ′′′ i ( σ i ) µ ′′′ i ( s ′ i ) ∝ X ∼ ( [ x si | x pi ] s i ) χ σ i σ i − ( x si , x pi − s i ) α ′′′ i − ( σ i − ) β ′′′ i ( σ i ) ∝ X ∼ [ x si | x pi ] χ σ i σ i − ( x si , x pi ; i ) α ′ i − ( σ i − ) β ′ i ( σ i ) . Hence, this MAP estimate maximizes also µ ′ i ( x si ) λ ′ i ( x si ) , i.e. itis the exact same estimate evaluated in [2]; the same can beshown for x pi . D. Syndrome Trellis
In [5] the source-to-syndrome realization implemented bythe source encoder is expanded in order to construct a col-lection of q n − k syndrome trellises . In particular, each one ofthem is constructed using only the encoder state transitions thatcorrespond to the release of a specific syndrome symbol. Then,during decoding, the trellis to be used at time i is specifiedby the corresponding syndrome symbol received s i .If the source encoder was a systematic syndrome generator,the source-to-syndrome realization could be derived as in theprevious section, so that each syndrome trellis would be madeof the same state transitions. At time i , with the input message µ ′′′′ i ( x i ) = µ ′′′ i ( x i ) ∝ µ ′ i ( x si ) µ ′ i ( x pi ) , the BCJR algorithmwould use the trellis described by ξ σ i σ i − ( x i ; i ) , ξ σ i σ i − ( x i , s i ) .Consequently, the approach would be equal to the one de-scribed in the previous section.However, the fact that the source encoder considered in [5]is not a systematic syndrome generator implies a substantialmodification to the syndrome decoding process discussed here.Decoding based on the parity perspective as described abovecan as well handle non-systematic syndromes. A discussionabout the advantages offered by this choice is out of the scopeof this letter. The first consequence is that, in general, the trellis The slight performance increase shown in [4] w.r.t. the algorithm in [3]can be justified only assuming that a sub-optimal estimate for x p ( D ) is foundin the latter (see note 4). over which the BCJR algorithm is eventually run no longershares the trellis transitions used by the systematic channelencoder for the code w.r.t. which the syndrome is formed.IV. M AXIMUM A P OSTERIORI P ROBABILITY D ECODING
The scheme based on inverse syndrome formation isthe only one that exactly re-uses the trellis described by χ σ i σ i − ( x si , x pi ) , but requires that source and side informationare defined over the same alphabet. Also, syndromes could bein general corrupted during transmission, but this is not alwaysproperly handled by the algorithms described above.If the systematic syndrome is received as r ( D ) ,the optimal MAP estimate can be simply found as arg max x i p ( x i | y ( D ) r ( D )) , i.e. marginalizing the function p ( x ( D ) p ( D ) s ( D ) | y ( D ) r ( D )) . Apart from some scaling fac-tors, this function is factorable into four terms, given by (i) thebackward correlation channel p ( x ( D ) | y ( D )) , (ii) the response χ ( p ( D ) | x s ( D )) of the parity encoder P ( D ) , (iii) the response ζ { s ( D ) = x p ( D ) − p ( D ) } of the systematic syndrome former( ζ {·} indicates the condition in brackets), and (iv) the trans-mission channel p ( r ( D ) | s ( D )) . If (i) and (iv) are memoryless,once (ii) is exploded into P ∼ ( x s ( D ) p ( D ) ) Q i χ σ i σ i − ( x si , p i ) thecorresponding factor graph is the one shown in Fig. 1(b).Since the resulting factor graph is a super-graph of the onein Fig. 1(a) (and no additional cycles are added), in place ofusing an ad-hoc solution, syndrome decoding can re-use thesame exact algorithm used in channel decoding. In particular,before running the BCJR algorithm, it is sufficient to form thecorrect input messages ˆ µ ( p i ) and, after, to properly processthe output messages ˆ λ ( p i ) in order to form the MAP estimateof x pi . Under this perspective, it does not matter if r ( D ) and y ( D ) are defined over the same alphabet of x ( D ) or not.V. C ONCLUSION
It has been shown that several implementations of sys-tematic syndrome decoding lead actually to the same sourcereconstruction. The most straightforward and general way toachieve this estimate is to tackle the problem as a MAPone and use iterative message-passing in order to find (orapproximate) the optimal solution. This permits to handle any correlation and transmission channels without the needof designing a new decoding algorithm.R
EFERENCES[1] A. D. Wyner, “Recent results in the Shannon theory,”
IEEE Trans. Inf.Theory , vol. 20, no. 1, pp. 2–10, Jan. 1974.[2] A. D. Liveris, Z. Xiong, and C. N. Georghiades, “Distributed compressionof binary sources using conventional parallel and serial concatenatedconvolutional codes,” in
Proc. of IEEE Data Compression Conf. , Mar.2003, pp. 193–202.[3] Z. Tu, J. Li, and R. S. Blum, “An efficient SF-ISF approach for theSlepian-Wolf source coding problem,”
EURASIP J. Appl. Signal Process. ,vol. 2005, no. 6, pp. 961–971, May 2005.[4] P. Tan and J. Li, “Enhancing the robustness of distributed compressionusing ideas from channel coding,” in
Proc. of IEEE Global Telecom.Conf. , vol. 4, Nov. 2005, pp. 2385–2389.[5] A. Roumy, K. Lajnef, and C. Guillemot, “Rate-adaptive turbo-syndromescheme for slepian-wolf coding,” in
Conf. Rec. of 41 st IEEE AsilomarConf. on Signals, Syst. and Comput. , Nov. 2007, pp. 545–549.[6] F. R. Kschischang, B. J. Frey, and H.-A. Loeliger, “Factor graphs andthe sum-product algorithm,”