Linear code-based vector quantization for independent random variables
aa r X i v : . [ c s . I T ] M a y Linear code-based vector quantization forindependent random variables
Boris D. KudryashovDepartment of Information Systems,St. Petersburg Univ. of Information Technologies, Mechanics and OpticsSt. Petersburg 197101, RussiaEmail: [email protected] V. YurkovDepartment of Information Systems,St. Petersburg University on Aerospace Instrumentation,St. Petersburg, 190000, RussiaEmail: [email protected]
Abstract
The computationally efficient vector quantization for a discrete time source can be performed using lattices overblock linear codes or convolutional codes. For high rates (low distortion) and uniform distribution the performanceof a multidimensional lattice depends mainly on the normalized second moment (NSM) of the lattice. For relativelylow rates (high distortions) and non-uniform distributions the lattice-based quantization can be non-optimal in termsof achieving the Shannon rate-distortion function H ( D ) .In this paper we analyze the rate-distortion function R ( D ) achievable using linear codes over GF ( q ) , where q is a prime number. We show that even for q = 2 the NSM of code-based quantizers is close to the minimumachievable limit. If q → ∞ , then NSM → / (2 πe ) which is the NSM of the infinite dimension sphere. Byexhaustive search over q -ary time-invariant convolutional codes with memory ν ≤ , the NSM-optimum codes for q = 2 , , were found.For low rates (high distortions) we use a non-lattice quantizer obtained from a lattice quantizer by extendingthe “zero zone”. Furthermore, we modify the encoding by a properly assigned metric to approximation values. Dueto such modifications for a wide class of probability distributions and a wide range of bit rates we obtained up to1 dB signal-to-noise ratio improvement compared to currently known results. Index terms —Arithmetic coding, convolutional codes, lattice, lattice quantization, normalized secondmoment, random coding, rate-distortion function, scalar quantization, vector quantizationI. I
NTRODUCTION
For an arbitrary discrete time memoryless continuous stationary source X = { x } , approximationalphabet Y = { y } , and a given distortion measure d ( x, y ) the theoretical limit of the code rate R underthe condition that the average distortion does not exceed D is determined by the Shannon rate-distortionfunction H ( D ) [1], H ( D ) = min { f ( y | x ) } :E[ d ( x,y )] ≤ D { I ( X ; Y ) } where I ( X ; Y ) denotes the mutual information between X and Y .An encoder of a discrete time continuous source is often referred to as a vector quantizer, and the codeas a codebook [2]. Coding theorems about achievability of H ( D ) are proved by random coding techniqueswithout imposing any restrictions on the codebook structure. The encoding complexity for such a codebookin general is proportional to the codebook size; i.e., the encoding complexity for sequences of length n with rate R bits per sample is proportional to Rn . A large step towards reducing the quantization complexity is due to using quantizers based on mul-tidimensional lattices [3]. In particular, Conway and Sloane in [4] and [5] proposed a simple encodingalgorithm, which, in essence, requires a proportional to dimension n of a codebook number of scalarquantization operations and subsequent exhaustive search over the lattice points in vicinity of the sourcesequence. The complexity of this quantization still grows exponentially in the dimension n of the vectors,but much slower than Rn .Important results related to the vector quantization efficiency are obtained by Zador [6], who hasshown that, in the case of small quantization errors, the characteristics of the quantizer depend on aparameter G n which is determined by the shape of the Voronoi polyhedrons of the codebook entries. Forlattice quantizers, i.e., when all Voronoi regions are congruent to each other, G n is equal to the so-called normalized second moment (NSM) of the lattice (see (6) in Section 2 below).Constructions of lattices with good values of NSMs are presented in [3]. For example, for the Leechlattice Λ NSM = 0.0658. Since for lattice Z (the set of integers) NSM = 1/12, the so called “ granulargain ” of a lattice quantizer in dB is computed as
10 log (12NSM) . Therefore the granular gain of Λ withrespect to Z is equal to 1.029 dB which is 0.513 dB away from the theoretical limit
10 log (12 / πe ) =1 . dB corresponding to covering the Euclidean space by n -dimensional spheres when n → ∞ .Granular gains achieved by different code constructions were reported in [7]. In particular, the Marcellin-Fischer [8] trellis-based quantizers are presented in [7] as an NSM record-holder. For example, by a trelliswith 256 states at each level MSN=1.36 dB is achieved (only ≈ α = 0 . is presented in [13].Despite numerous attempts to construct good quantization schemes, the gap between asymptoticallyachievable results and the best existing quantizers is still large. For example, for 32-state trellises and bitrate 3 bits/sample, this gap is equal to 1.10, 1.53, and 1.78 dB for the Gaussian distribution, Laplaciandistribution, and generalized Gaussian distribution with α = 0 . , respectively [13].Since the best codes and coding algorithms for non-uniform distributions are not the same as for theuniform distribution, the following question arises: could it be that codes which deliberately are not goodin sense of granular gain be very good for a non-uniform distribution?The positive answer to this question is one of the results of this paper. Below we present the codingscheme which with a 32-state trellis at 3 bits/sample is not more than 0.32, 0.35, and 0.44 dB away fromthe Shannon limit for the three distributions mentioned above. Therefore, the achieved gain over existingtrellis quantizers exceeds 1 dB at high rates for a wide class of probability distributions.We have searched good quantizers among lattice quantizers over linear codes. The motivation is that theconstructions of some good lattices are closely related to the constructions of good block error-correctingcodes (e.g., the Hamming code and Golay codes). One more argument in favor of such trellises appears in[14], [15] and [16], where it was shown that there exist asymptotically optimal lattices over q -ary linearcodes as q tends to infinity.Although the NSM decreases (the granular gain increases) with alphabet size q , it was found by simu-lations that the quantization performance behavior for non-uniform quantization is different: binary codesappear to be better than codes over larger fields. Replacing the quaternary codes, typically considered thein trellis-based quantization literature, by binary codes is the first step towards improving the quantizationperformance.Another step is based on the important observation (see, e.g., [17], [18]) that the entropy-coded uniformscalar quantization with properly chosen reconstruction values provides near the same coding efficiency asthe optimum entropy-constrained scalar quantization (ECSQ) does. Moreover, choosing zero quant largerthan others, we obtain a simple scalar quantizer with performance extremely close to the ECSQ for allrates. In [18] this quantizer is called Extending Zero Zone (EZZ) quantizer. Notice that EZZ quantization does not follow the minimum Euclidean distance criterion. Therefore, it would be reasonable not to usethe Euclidean distance for vector quantization (or trellis quantization as a special case) as well.We begin in Section 2 with reviewing the random coding bounds [14], [15] on the NSM value for linearcode based lattices. The asymptotic behavior of the NSM as a function of the code alphabet size is studiedand we show that code-based lattices are asymptotically optimal when the alphabet size tends to infinity.In Section 3 the code search results are presented. In Section 4 we present entropy-constrained code-based quantization. By generalizing the EZZ approach to lattice quantization, we obtained near optimumSNR values for all code rates for the classes of non-uniform probability distributions mentioned above.Concluding remarks are given in Section 5.II. C ODE - BASED QUANTIZATION An n -dimensional lattice can be defined by its generator matrix G ∈ R n × n as [3] Λ( G ) = { λ ∈ R n |∃ z ∈ Z n : λ = G z } . (1)Let q be a prime number and C denote q -ary linear ( n, k ) code. Then a lattice over the code C isdefined as Λ( C ) = { λ ∈ Z n |∃ c ∈ C : λ ≡ c ( mod q ) } . (2)In other words, the lattice Λ( C ) consists of integer-valued sequences which are equal to codewords from C by modulo q . In the case of binary codes the least significant bits of the sequences λ ∈ Λ( C ) arecodewords of C .The lattice defined by (2) is known as Construction A [3].When a n -dimensional lattice is used as a codebook, the source sequence x ∈ R n is approximated bysome lattice point λ ∈ Λ( C ) . The whole set Λ( C ) is assumed to be ordered and a number representing theoptimal approximation point λ is transmitted instead of x . Therefore, we need some rule which assigns anumber to each point of Λ( C ) . To suggest such a rule we consider the following representation of Λ( C ) : Λ( C ) = q k [ m =1 { q Z n + c m } . (3)In other words, Λ( C ) is a union of q k cosets which leaders are coderwords of C . This representationallows a one-to-one mapping of all λ ∈ Λ( C ) onto the set of integer vectors ( m, b ) , where m is acodeword number and b = ( b , ..., b n ) ∈ Z n defines the point number in q Z n . The lattice point λ can bereconstructed from ( m, b ) by the formula λ = c m + q b . (4)The lattice Λ( C ) or its scaled version a Λ( C ) , a > can be used as a codebook for vector quantization.Below we always assume a = 1 since instead of quantizing the random vector x with a Λ( C ) we canquantize x /a with Λ( C ) . We assume also that • The stationary memoryless source sequences are drawn from X n = { x | x = ( x , . . . , x n ) } ⊆ R n witha known probability density function (pdf) f ( x ) = Q ni =1 f ( x i ) . • The squared Euclidean distance d ( x , y ) = 1 n k x − y k = 1 n n X i =1 ( x i − y i ) is used as a distortion measure. The quantization procedure can be described as a mapping y = Q ( x ) , where x ∈ X n , y ∈ Λ( C ) . Themean square error (MSE) of the quantizer D = 1 n M (cid:2) || x − Q ( x ) || (cid:3) = 1 n X i Z R i || x − λ i || f ( x ) d x (5)is minimized by the mapping Q ( x ) = λ i if x ∈ R i where R i are Voronoi regions [3] of lattice points, i.e., R i , R ( λ i ) = { x ∈ R n : || x − λ i || ≤ || x − λ j || ∀ j = i } . All Voronoi regions of lattice points are congruent to each other. We denote the Voronoi region whichcontains the origin as R . We use the conventional notations V ( R ) = Z R d x G n ( R ) = 1 n R R k x k d x V ( R ) /n (6)for the volume and the normalized second moment (NSM) of R , respectively.By entropy coding the rate R of the lattice quantizer can be arbitrary close to the entropy of the latticepoints R = − n X i p i log p i , where the probabilities p i of the lattice points are equal to p i , p ( λ i ) = Z R i f ( x ) d x , i = 1 , , . . . . The rate-distortion function R ( D ) of the lattice quantizer can be easily estimated under the assumptionthat the quantization errors are so small that the pdf f ( x ) can be approximated by a constant value insideeach region R ( λ i ) . In this case, the rate can be expressed via the differential entropy of the source h ( X ) = − Z f ( x ) log f ( x ) dx and the volume V ( R ) of the Voronoi region R , R ( D ) ≤ h ( X ) − n log V ( R ) . (7)For the average distortion value D from (5) and (6) we have D = G n ( R ) V ( R ) /n . (8)From the last two equations and the Shannon lower bound, H ( D ) ≥ h ( X ) −
12 log(2 πeD ) we obtain the inequalities H ( D ) ≤ R ( D ) ≤ H ( D ) + 12 log (2 πeG n ( R )) . (9)Therefore to prove that the optimal quantization would be asymptotically achievable by lattice quanti-zation (i.e., R ( D ) → H ( D ) , if n → ∞ ) we need to prove that G n ( R ) → / (2 πe ) when n → ∞ . TABLE IR
ANDOM CODING UPPER BOUNDS ON
NSM
FOR OPTIMUM CODE RATE AND FOR CODE RATE R C = 1 / q R G ∞ ( q ) R C = R R C = 1 / ∞ Consider a lattice Λ( C ) over a q -ary linear ( n, k ) code, and denote the code rate R C = k/n . It can beshown (see [14]) that the overall volume of the q k Voronoi regions of the codewords of C coincides withthe volume of the n -dimensional cube with edge length equal to q . It means that V ( R ) = q n /q k = q n − k and from (8) follows D = G n ( R ) q − R C ) . (10)Formally we cannot use equation (10) for finding NSM for a lattice over a code by estimating thequantization error for a random variable uniformly distributed over the cube. The reason is that the n -dimensional cube with length of the edge q does not contain precisely q k complete Voronoi regions. Someregions near the boundaries of the cube protrude from the cube. To solve this problem the “cyclic” metric ρ ( x, y ) = min { ( x − y ) , ( | x − y | − q ) } (11)was introduced in [14]. It was shown that the NSM of the lattice satisfies (10) if the lattice points of thecube are used as codebook entries and the distortion D is computed with respect to the metric ρ ( · , · ) .Below we will use (10) to estimate the NSM of convolutional codes and in [14] this equation was usedto obtain the random coding bound on the achievable NSM value. The reformulated result of [14] (seecomments in Appendix) is presented here without a proof. Theorem 1:
For any prime number q and any ε > , there exists a sequence C , C , C . . . , of q -arylinear codes, where C n is a code of length n , and such a number N , that for all n ≥ N | G n ( q ) − d q R ( q ) − | < ε (12)where G n ( q ) is the second normalized moment of the lattice based on the code C n , d is a solution ofthe equation d = 2 Z / ∂∂s g ( s, x ) g ( s, x ) dx (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) s = − d (13)and R ( q ) is defined as R ( q ) = 1ln q (cid:16) − − Z / ln g (1 / (2 d ) , x ) dx (cid:17) (14)where g ( s, x ) = 1 q q − X k =0 e sρ ( x,k ) (15)and ρ ( · , · ) is defined by (11).The values G ∞ ( q ) = lim n →∞ G n ( q ) for q = 2 , , , are given in Table II. Also we show in the table thecode rate value R C = R ( q ) which minimizes the random coding estimate on the NSM value G ∞ ( q ) andthe NSM value achievable when we consider codes with nonoptimal rate R C = 1 / . Presented estimates give the achievable NSM values if long enough codes are used for quantization. In particular, the searchresults presented in Section III show that the lattices obtained from convolutional codes with a numberof encoder states above 512 are roughly 0.2 dB away from the asymptotically achievable NSM value fora given alphabet size q . On the other hand, rate R C = 1 / binary codes with 64 states have even betterNSM than the NSM computed by averaging over randomly chosen infinitely long rate R C = 1 / codes.An analysis of the asymptotic behavior of G ∞ ( q ) and R ( q ) with q → ∞ leads to the following result: Corollary 1: lim q →∞ G ∞ ( q ) = 12 πe (16) lim q →∞ R ( q ) = 12 . (17)The proof is given in Appendix.III. NSM- OPTIMUM CONVOLUTIONAL CODES
It follows from Corollary 1 that codes with rate R C = 1 / provide asymptotically optimum NSM valuefor large code alphabet size q . It follows from the data in Table II that codes with rate R C = 1 / givenear-optimum NSM except for q = 2 . Below we consider only rate R C = 1 / codes since they are mostconvenient for practical applications.Moreover, we will search for good linear codes for quantization among truncated convolutional codes,since for such codes the quantization (search for a codeword closest to the source sequence) can be easilyperformed using the Viterbi algorithm.A q -ary, rate R C = 1 / convolutional encoder over GF ( q ) with memory m can be described by thepolynomial encoding matrix G ( D ) = ( g ( D ) g ( D )) where g i ( D ) = g i + g i D + · · · + g im D m , i = 1 , are q -ary polynomials such that max i { deg g i ( D ) } = m .In polynomial notations the infinite length input information sequence u ( D ) = u + u D + · · · + u i D i + · · · and the corresponding infinite length codeword v ( D ) = v + v D + · · · + v i D i + · · · , where v i = ( v i , v i ) satisfy the matrix equation v ( D ) = u ( D ) G ( D ) . (18)The transformation u ( D ) to v ( D ) is invertible if at least one of two polynomials is delay-free ( g = 1 or g = 1 ). The convolutional encoder (18) is non-catastrophic if g ( D ) and g ( D ) are relatively prime([19]). Since for any convolutional code both catastrophic and non-catastrophic encoders do exist, we canconsider only non-catastrophic encoders without loss of optimality.For any polynomial a ( D ) = a + a D + · · · + a i D i + · · · let ⌈ a ( D ) ⌉ n = a + a D + · · · + a n D n denotethe finite degree polynomial obtained from a ( D ) by truncating it to degree n .Furthermore we consider q -ary linear ( n, k = n/ codes which codewords are coefficients of truncatedvector polynomials ⌈ v ( D ) ⌉ n/ obtained from degree k = n/ information polynomial u ( d ) by encodingaccording to (18).The trellis representation of ( n, n/ binary linear code obtained from (1+ D + D , D ) convolutionalencoder is shown in Fig. 1. For q -ary codes, q > , similar trellises can be constructed but with q branchesleaving each node and merging in each node of the trellis diagram.
11 11 11 11 00 00 00 10 01 01 10 01 10 11 00 00 01 01 10 11 11 11 00 00 10 01 10 … … …
01 10 11 = t - n n Fig. 1. Trellis representation of ( n, n/ binary linear code obtained from the (1 + D , D + D ) convolutional encoderTABLE IIO PTIMUM BINARY CONVOLUTIONAL CODES AND THEIR
NSM S . T HE GENERATORS ARE GIVEN IN OCTAL NOTATION .states num. generator G n En. gain, dB2 [3;1] 0.0733 0.55714 [7;5] 0.0665 0.98008 [17;13] 0.0652 1.065716 [31;23] 0.0643 1.126132 [61;57] 0.0634 1.187364 [165;127] 0.0628 1.2286128 [357;251] 0.0623 1.2633256 [625;467] 0.0620 1.2843512 [1207;1171] 0.0618 1.2983 ∞ — 0.0598 1.4389 To find the NSM for a ( g ( D ) , g ( D )) convolutional encoder we first estimate the quantization errorfor source sequences uniformly distributed in the hypercube [0 , q ] n , when the “cyclic metric” ρ ( · , · ) (11)is used as the distortion measure (see [14]). Then this error is recalculated into the NSM value using (10).To speed up the exhaustive search for the optimum encoders over the set of all memory m encoderswith generators ( g ( D ) , g ( D )) we used the following rules to reduce the search space without loss ofoptimality, • Only delay-free polynomials are considered g = g = 1 . • Since the codes with the generators ( g ( D ) , g ( D )) and ( g ( D ) , g ( D )) are equivalent, we con-sider only encoders with deg( g ( D )) ≥ deg( g ( D )) . If deg( g ( D )) = deg( g ( D )) then we require | g ( D ) | ≥ | g ( D ) | , where we denote | a ( D ) | = deg( a ( D )) X i =0 a i q i . In other words, we interpret the polynomial coefficients as positions of a number written in q -ary form.Thereby we we assign an integer number (value) to each polynomial and only pairs of generatorssorted in descending order of the corresponding values are used as candidates. • Only non-catastrophic encoders are considered, i.e., gcd( g ( D ) , g ( D )) = 1 .Our search results are presented in Tables II, III, and IV for q = q grows, but the improvement provided by TABLE IIIO
PTIMUM TERNARY CONVOLUTIONAL CODES AND THEIR
NSM S .states num. generator G n En. gain, dB3 [12;11] 0.0720 0.63499 [121;111] 0.0663 0.993127 [1211;1112] 0.0641 1.139681 [11222;10121] 0.0626 1.2424243 [110221;101211] 0.0617 1.3053729 [1000112;112122] 0.0614 1.3265 ∞ — 0.0586 1.5231TABLE IVO PTIMUM QUINARY CONVOLUTIONAL CODES AND THEIR
NSM S .states num. generator G n En. gain, dB5 [14;13] 0.0716 0.659125 [131;102] 0.0642 1.1328125 [1323;1031] 0.0622 1.2703625 [10314;10133] 0.0613 1.3336 ∞ — 0.0585 1.5229 ternary or quinary codes over binary codes is small and does not worth the complication of the encoding.Notice also, that the determined NSM values correspond to less shaping gain than 1.36 dB reported in[8] for the 256-states quaternary codes. To achieve the same shaping gain we have to have NSM=0.0609whereas the best found 625-state 5-ary code provides NSM=0.0613 which is approximately 0.02 dB worsethan the value for the Marcellin-Fischer code.Notice, however, that the Marcellin-Fischer quantizer [8] is not a lattice quantizer. Therefore, the NSMvalues reported here are the best among known multidimensional lattices .IV. A RITHMETIC CODED LATTICE QUANTIZATION
Let us consider the lattice over a linear ( n, k ) code C (not necessarily a truncated convolutional code).The entropy quantization of the input sequence x can be performed in two steps: • Approximation, i.e., finding the vector of indices b and the codeword number m ∈ { , ..., q k − } that minimizes the distortion value d ( x , λ ) = n − k x − λ k = n − k x − ( c m + q b ) k (see (4)). • Entropy lossless coding of the pair (m, b ). First, we consider the approximation step. From (4) the following reformulation of the encoding problemfollows, min λ (cid:8) k x − λ k (cid:9) = min m min b (cid:8) k x − ( c m + q b ) k (cid:9) = min m min b ( n X t =1 ( x t − c mt − qb t ) ) = min m ( n X t =1 min b t (cid:8) ( x t − c mt − qb t ) (cid:9)) = min m ( n X t =1 µ ( x t , c mt ) ) = min m { µ ( x , c m ) } . (19)Here we introduced the additive metric µ ( x , c ) = n X t =1 µ ( x t , c t ) ,µ ( x, c ) = min b (cid:8) ( x − c − qb ) (cid:9) = ( x − c − qb ) (20)where the optimum index value b can be determined by “scalar quantization” with step q applied to ( x − c ) , b = (cid:28) x − cq (cid:29) . (21)By h·i we denote rounding to the nearest integer.It follows from (19), (20), and (21) that lattice quantization can be split into two steps. First, for all t = 1 , ..., n and c = 0 , ..., q − , the values x t − c should be scalar quantized and the n × q indices b tc andthe corresponding metrics µ tc = ( x t − c − qb tc ) are to be computed. In the second step, the codeword c m yielding the minimum of the metrics µ ( x , c m ) among all codewords c m ∈ C , m ∈ { , ..., q k − } shouldbe found. The codeword number m and vector of indices b = ( b tc m , ..., b tc mn ) describe the trellis pointclosest to x . The formal description of the algorithm is presented in Fig. 2.The problem of searching for the optimum entry over the infinite cardinality multidimensional latticecodebook is now divided into two steps: qn scalar quantization operations followed by searching for thebest codeword among the q k codewords of the q -ary linear ( n, k ) code. The latter step is similar to softdecision maximum-likelihood decoding of error correcting linear block codes and there are many suchdecoding algorithms known; in particular for truncated convolutional codes the Viterbi algorithm is areasonable choice.Now let us consider an algorithm for a variable-length lossless coding of the pair ( m, b ) obtained onthe approximation step.For a rate R C = 1 / truncated convolutional ( n, k = n/ -code, the codeword is determined by length k information sequence u = ( u , ..., u k ) and the codeword can be represented as a sequence of k tuplesof length 2, i.e., c = ( c , ..., c k ) , c t = ( c t , c t ) . We write the corresponding vector of indices in tuples oflength 2 too, i.e., b = ( b , ..., b k ) , b t = ( b t , b t ) .Denote by σ t = ( u t − , ..., u t − m ) the encoder state at time t , where we assume u t = 0 for t ≤ .To describe arithmetic coding (for details see [20]) it is enough to define a probability distribution onthe pairs ( u , b ) . Thus we introduce the following two probability distributions, { ϑ ( u t | σ t ) = ϑ ( u | σ ) , σ ∈ GF mq } Input : Source sequence x = ( x , x , . . . , x n ) . Output : Number m of codeword c m ∈ C , index sequence b = ( b , b , . . . , b n ) . Metrics and indices computation: for t = 1 to n dofor c = 0 to q − do b t ( c ) = D x t − cq E µ t ( c ) = ( x t − c − qb t ( c )) endend Codeword search:
Find the codeword c m = ( c m , c m , . . . , c mn ) that minimizes metric µ ( x , c m ) = P nt =1 µ t ( c mt ) . Result:
The algorithm outputs the codeword number m and the index vector b = ( b ( c m ) , b ( c m ) , . . . , b n ( c mn )) . Fig. 2. Approximation step of lattice quantization. { ϕ ( b it | c it ) = ϕ ( b | c ) , b ∈ Z , c ∈ GF q , i = 1 , } which we estimate for a given source pdf by training using a long simulated data sequence. The requiredprobability distribution p ( u , b ) is defined as p ( u , b ) = k Y t =1 ϑ ( u t | σ t ) ϕ ( b t | c t ) ϕ ( b t | c t ) where code symbols c it , i = 1 , , are uniquely defined by u t and σ t .Notice that similar entropy coding for trellis quantization was described in [21], [8], and [13]. Thedifference is that in these papers the approximation alphabet is finite, whereas in our algorithm theapproximation alphabet is not necessary finite since we allow any lattice point to be an approximationvector.Simulation results of the described arithmetic-coded quantization will be presented in next section.Below we present a modification which is efficient for low quantization rates for non-uniform probabilitydistributions.As it was mentioned in the Introduction, the codes which are good in terms of granular gain (NSM value)are not necessarily optimal for non-uniform distributions. The reason is that spheres are not necessarilygood Voronoi regions in this case. Moreover, Voronoi regions are not optimal quantization cells in thiscase. To understand this better, we consider properties of optimal entropy-constrained scalar quantization.First of all, notice that for a wide class of probability distributions the simulation results for optimalnon-uniform scalar quantization are presented in [17]. It is easy to verify that, for example, for GGDrandom variables with parameter α ≤ . , for all bit rates the best trellis quantizers loose with respect tooptimum scalar quantizers [13]. The analysis of optimum and near-optimum scalar quantizers from [17],[18] show that: • For GGD random variables the optimum ECSQ quantizer has the larger quantization cells near zeroand smaller cells for large values. • The quantization thresholds are not equal to half of the sum of the neighboring approximation values.Therefore, the minimum MSE (or minimum Euclidean distance) quantizers are not optimum ECSQquantizers. Input : Source sequence x = ( x , x , . . . , x n ) . Output : Number m of codeword c m ∈ C , index sequence b = ( b , b , . . . , b n ) . Metrics and indices computation: for t = 1 to n do ξ = 0 ; if x t < − ∆ then ξ = x t + ∆ endif x t > ∆ then ξ = x t − ∆ endfor c = 0 to q − do b t ( c ) = D ξ − cq E µ t ( c ) = ( ξ − c − qb t ( c )) endend Codeword search:
Find the codeword c m = ( c m , c m , . . . , c mn ) that minimizes metric µ ( x , c m ) = P nt =1 µ t ( c mt ) . Result:
The algorithm outputs the codeword number m and the index vector b = ( b ( c ) , b ( c ) , . . . , b n ( c n )) . Fig. 3. Approximation with an extended zero zone • The quantizers which have all quantization cells of equal size except the one symmetrically locatedaround zero (so called “extended zero zone” (EZZ) quantizer), has performances very close to thatof the optimum ECSQ quantizer .It follows from these observations that good trellis quantizers should be obtained as a generalization ofthe EZZ quantizer to the multidimensional case. (Another method of extending the zero zone for trellisquantization, so called New Trellis Source Code (NTSC), have been presented in [13]).To describe the multidimensional EZZ quantizer we need to generalize (20) and (19) to the case whenthe input data from some interval [ − ∆ , ∆] , ∆ > are artificially interpreted as zeroes. Therefor we replaceeach input r.v. x by another r.v. defined as ξ = x + ∆ x < − ∆0 , − ∆ ≤ x ≤ ∆ x − ∆ x > ∆ . (22)Then we apply (20) and (19) to the r.v. ξ . The generalized version of the algorithm describing theapproximation step of lattice quantization is shown in Fig. 3. The parameter ∆ is different for differentpdf’s and quantization rates. For each particular pdf and rate we minimize average distortion value byselecting the best value of ∆ from the set − s , s = 0 , , , . Simulation results will be presented in nextsection.Now we consider the reconstruction of the approximation vector y = ( y , ..., y n ) by the decoder fromthe received pair ( m, b = ( b , ..., b n )) . First, we introduce an additional integer parameter L which definesthe amount of approximation values which are estimated using a long training sequence (in our simulationswe used L = 10 ). For the training source sequence x = ( x , ..., x N ) we performed the lattice quantization described above and found the optimum codeword c = ( c , ..., c N ) and the corresponding sequence ofindices b = ( b , ..., b N ) . Then for all b ∈ {− L, ..., − , , ...L } and c ∈ , . . . , q − we computed “refined”approximation values β cb = , b = 0 , c = 0 P i ∈ J cb x i | J cb | + ∆ , b ∈ { , ...L } , c ∈ { , . . . , q − } or b = 0 , c ∈ { , . . . , q − } P i ∈ J cb x i | J cb | − ∆ , b ∈ {− L, ..., − } , c ∈ { , . . . , q − } where J cb = { t | c t = c, b t = b } . The set of values β cb is assumed to be known to decoder.When the pair ( m, b = ( b , ..., b n )) is received, the decoder first finds the codeword c m = ( c m ..., c mn ) and then reconstructs the approximation vector y = ( y , ..., y n ) as y t = (cid:26) β c mt b t b t {− L, ..., − , , ...L } c mt + qb t b t / ∈ {− L, ..., − , , ...L } . V. S
IMULATION RESULTS
This section contains the results of the arithmetic coded lattice quantization for different types ofsources. We consider the parametric class of generalized Gaussian distributions. The probability densityfunction is parametrized by the parameter α and expressed as p ( x ) = (cid:20) αη ( α, σ )2Γ(1 /α ) (cid:21) exp {− [ η ( α, σ ) | x | ] α } where η ( α, σ ) = σ − (cid:20) Γ(3 /α )Γ(1 /α ) (cid:21) / , α > z ) = Z ∞ t z − e − t dt and σ denotes the standard deviation. The Laplacian and Gaussian distributions are members of the familyof generalized Gaussian distributions with parameters α = 1 . and α = 2 . , respectively.Simulation results for the Gaussian distribution are presented in Table V. The SNR achieved by the“New Trellis Source Code” (NTSC) [13] and by the “Entropy Constrained Scalar Quantization” (ECSQ)[17] are also shown in the table. We can see that the estimated SNR value for the encoder with 512states is only about 0.2 dB below Shannon limit. For encoders with 4 and 32 states we obtain significantimprovements over the NTSC-based quantization.Simulation results for the Laplacian distribution and for the GGD with α = 0 . are presented in TablesVI and VII, respectively. We show the quantization efficiency both without EZZ ( ∆ = 0 ) and with EZZ( ∆ > ). Also data from [13] are given in parenthesis. It follows from the simulation results that the EZZis efficient for low quantization rates (below 2 bit/sample) and provides an SNR gain of about 0.1 dBfor the Laplacian source and about 0.4 dB for the GGD with α = 0 . . The gap between the achievedSNR and the theoretical limit H ( D ) for these two probability distributions is nearly the same: 0.2 – 0.3dB. Notice also, that for the GGD with α = 0 . , low-complexity (less than 32 encoder states) low ratequantizers do not perform better than the ECSQ. Therefore, constructing efficient quantization algorithmsfor low rate coding for the GGD with small α remains to be an open problem. TABLE VG
AUSSIAN SOURCE . D
ATA FOR
NTSC [13]
IN PARENTHESIS .Rate Trellis states H ( D ) ECSQ2 4 8 16 32 64 128 256 5120.5 2.29 2.50 2.55 2.61 2.64 2.69 2.69 2.76 2.76 3.01 2.101 5.06 5.53 5.61 5.66 5.71 5.74 5.74 5.85 5.85 6.02 4.64(5.33) (5.64)2 11.08 11.50 11.63 11.69 11.72 11.77 11.77 11.82 11.84 12.04 10.55(11.15) (11.37)3 17.11 17.55 17.64 17.70 17.74 17.80 17.80 17.85 17.87 18.06 16.56(16.71) (16.96)TABLE VIL
APLACIAN SOURCE . D
ATA FOR
NTSC [13]
IN PARENTHESIS . Rate ∆ Trellis states H ( D ) ECSQ2 4 8 16 32 64 128 256 5120.5 0 2.92 3.03 3.06 3.09 3.11 3.14 3.17 3.18 3.21 3.54 3.110.25 3.06 3.15 3.17 3.18 3.20 3.22 3.231 0 5.69 6.05 6.14 6.18 6.23 6.29 6.30 6.33 6.33 6.62 5.760.25 5.87 6.14 6.20 6.27 6.33 6.40 6.42(5.93) (6.07)2 0 11.68 12.15 12.22 12.29 12.32 12.38 12.41 12.43 12.44 12.66 11.310.125 11.74 12.15 12.24(11.49) (11.72)3 0 17.74 18.15 18.23 18.29 18.33 18.39 18.41 18.44 18.44 18.68 17.20(17.00) (17.15)
TABLE VIIG
ENERALIZED G AUSSIAN SOURCE WITH α = 0 . . D ATA FOR
NTSC [13]
IN PARENTHESIS . Rate ∆ Trellis states H ( D ) ECSQ2 4 8 16 320.5 0 4.74 4.81 4.83 4.83 4.84 5.62 5.370.5 5.19 5.22 5.23 5.23 5.241 0 8.00 8.13 8.18 8.22 8.22 9.21 8.610.5 8.53 8.53 8.62 8.64 8.64(8.11) (8.29)2 0 14.31 14.71 14.80 14.90 14.94 15.60 14.580.25 14.56 14.90 15.03 15.10 15.14(14.13) (14.47)3 0 20.54 20.96 21.04 21.10 21.16 21.70 20.490.25 20.62 21.06 21.14 21.20 21.26(19.61) (19.92) A PPENDIX
Comments to formulation of Theorem 1.
In [14] expressions (13) and (14) were given in the following form d = 1 q Z q ∂∂s g ( s, x ) g ( s, x ) dx (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) s = − d (23) R ( q ) = 1ln q (cid:16) − − q Z q ln g (1 / (2 d ) , x ) dx (cid:17) . (24)The simplification is due to the observation that g ( s, x ) is a periodical function of x with period 1 andinside the interval [0 , the function g ( s, x ) is symmetric with respect to the middle point x = 1 / , i.e., g ( s, x ) = g ( s, − x ) . Proof of Corollary 1.
Here we analyze the asymptotic behavior of the estimate of G n ( q ) given by Theorem 1 when q → ∞ .From definitions (11) and (15) for x ∈ [0 , . we obtain ρ ( x, k ) = (cid:26) ( x − k ) , k = 0 , ..., q − ( k − x − q ) , k = q +12 , ..., q − (25) g ( s, x ) = 1 q q − X k =0 e s ( x − k ) + 1 q q − X k = q +12 e s ( k − x − q ) (26) = 1 q q − X k = − q − e s ( x − k ) , x ∈ [0 , . . (27)To prove the Corollary, we choose s = − πeq (28)The first step is to verify that for large q this choice makes (13) valid, i.e., d = − s = q πe . (29)The second step is to substitute this expression into (14) and show that for large q we have R ( q ) → / .After these two steps we immediately obtain the main result, lim q →∞ G ∞ ( q ) = lim q →∞ d q R ( q ) − = 12 πe . We start the first step of derivations by demonstrating that asymptotically for large q the generatingfunction g ( s, x ) does not depend on x . By straightforward computations it is easy to verify that max x { g ( s, x ) } = g ( s,
0) = 1 q q − X k = − q − e sk (30) min x { g ( s, x ) } = g ( s, /
2) = 1 q q − X k = − q − e s (2 k − / . (31) In order to estimate these sums we will use [22], formula 552.6, that is, e − x + e − x + e − x + ... = −
12 + √ πx (cid:20)
12 + e − π /x + e − π /x + e − π /x + ... (cid:21) . (32)For large q we obtain from (30) that g ( s, ≤ q + 2 q ∞ X k =1 e sk = 1 q + 2 q (cid:18) −
12 + r − πs (cid:20)
12 + e π /s + e π /s + e π /s + ... (cid:21)(cid:19) . Substituting (28) and upper-bounding the sum by the geometric progression for q ≥ , we obtain theupper bound g ( s, x ) ≤ max x g ( s, x ) ≤ √ qe + e − q . (33)To obtain a lower bound on the sum in (31) we rewrite it in the form g ( s, /
2) = 2 q q − X k =1 e s (2 k − / + 1 q e sq / . Notice that the integer (2 k − in the exponent of summands runs over odd values. If we add termscorresponding to intermediate even values the sum increases less than twice. Therefore, g ( s, / ≥ q q X k =1 e sk / = 1 q ∞ X k =1 e sk / − q ∞ X k = q +1 e sk / . Next we estimate second sum by the geometric progression and estimate first sum using (32). Afterstraightforward computations we obtain g ( s, x ) ≥ min x g ( s, x ) ≥ √ qe − q . (34)We can substitute (33) and (34) into (13) to obtain lower and upper bounds on d ( q ) . In both cases wehave to estimate the following integral Z / ∂∂s g ( s, x ) dx = Z / − / q q − X k = − q − ( x − k ) e s ( x − k ) dx =1 q q − X k = − q − Z − k +1 / − k − / x e sx dx =1 q Z q/ − q/ x e sx dx = √ q πe / erf (cid:18) √ πeq (cid:19) − q πe e − πeq . From inequalities ([22], formula 592) − e − x x √ π ≤ erf( x ) ≤ follow the bounds √ q πe / − q πe e − πeq ≤ Z / ∂∂s g ( s, x ) dx ≤ √ q πe / . Substituting them together with (33) and (34) into (13) we obtain q πe (cid:0) − O (cid:0) q e − q (cid:1)(cid:1) ≤ d ≤ q πe (cid:18) O (cid:18) √ q (cid:19)(cid:19) . Therefore (29) is proven and the first step completed.To fulfil the second step and the proof of Corollary, we have to substitute the bounds (33) and (34)into expression (14). After simple derivations we immediately obtain (17)R
EFERENCES [1] T. M. Cover and J. A. Thomas,
Elments of Information Theory . New York: John Wiley & Sons, 1991.[2] R. M. Gray and D. L. Neuhoff., “Quantization,”
IEEE Trans. on Inf. Theory , vol. 44, pp. 2325–2383, Oct. 1998.[3] J. H. Conway and N. J. A. Sloane,
Sphere Packings, Lattices, and Groups . New York: Springer-Verlag, 2nd edition ed., 1993.[4] J. H. Conway and N. J. A. Sloane, “Fast quantizing and decoding and algorithms for lattice quantizers and codes,”
IEEE Trans. onInf. Theory , vol. 28, pp. 227–232, Mar 1982.[5] J. H. Conway and N. J. A. Sloane., “A fast encoding method for lattice codes and quantizers,”
IEEE Trans. on Inf. Theory , vol. 29,pp. 820–824, Nov 1982.[6] P. L. Zador,
Development and Evaluation of Procedures for Quantizing Multivariate Distributions . PhD thesis, Stanford Univ., 1963.[7] M. V. Eyuboglu and G. D. Forney, “Lattice and trellis quantization with lattice- and trellis-bounded codebooks—high-rate theory formemoryless sources,”
IEEE Trans. Inform. Theory , vol. 39, pp. 46–59, January 1993.[8] M. Marcellin and T. Fischer, “Trellis coded quantization of memoryless and gauss-markov sources,”
IEEE Trans. Commun. , vol. COM-38, pp. 82–93, January 1990.[9] T. R. Fischer and M. Wang, “Entropy-constrained trellis-coded quantization,”
IEEE Trans. Info. Th. , vol. IT-38, pp. 415 – 426, Jan.1992.[10] R. Laroia and N. Farvardin, “Trellis-based scalar-vector quantizer for memoryless sources,”
IEEE Trans. Inform. Theory , vol. IT-40,pp. 860 – 870, May 1994.[11] M. W. Marcellin, “On entropy-constrained trellis coded quantization,”
IEEE Trans. Comm. , vol. COM-42, pp. 14 – 16, Jun. 1994.[12] R. J. van der Vleuten and J. H. Weber, “Construction and evaluation of trellis-coded quantizers for memoryless sources,”
IEEE Trans.Inform. Theory , vol. IT-41, pp. 853 – 858, May 1995.[13] L. Yang and T. R. Fischer, “A new trellis source code for memoryless sources,”
IEEE Trans. on Inf. Theory , vol. 44, pp. 3056–3063,November 1998.[14] B. D. Kudryashov and K. V. Yurkov, “Random coding bound for the second moment of multidimensional lattices,”
Problems ofInformation Transmission , vol. 43, pp. 57 – 68, March 2007.[15] K. V. Yurkov and B. D. Kudryashov, “Random quantization bounds for lattices over q -ary linear codes,” in ISIT 2007 , pp. 236–240,June 2007.[16] U. Erez, S. Litsyn, and R. Zamir., “Lattices which are good for (almost) everything,”
IEEE Trans. on Inf. Theory , vol. 51, pp. 3401–3416, Oct. 2005.[17] N. Farvardin and J. W. Modestino, “Optimum quantizer performance for a class of non-gaussian memoryless sources,”
IEEE Trans.on Inf. Theory , vol. IT-30, pp. 485–497, May 1984.[18] B. D. Kudryashov, E. Oh, and A. V. Porov, “Scalar quantization for audio data coding,”
IEEE Trans. Audio, Speech, and LanguageProcessing , 2007. submitted for publication.[19] R. Johannesson and K. S. Zigangirov,
Fundamentals of Convolutional Coding . New York: IEEE Press, Piscataway, 1999.[20] I. H. Witten, R. M. Neal, and J. G. Cleary, “Arithmetic coding for data compression,”
Commun. ACM , vol. 30, pp. 520–540, June1987.[21] R. L. Joshi, V. J. Cramp, and T. R. Fischer, “Image subband coding using arithmetic coded trellis coded quantization,”
IEEE Trans.on Ciruits and Systems for Video Technology , vol. 5, pp. 515–523, Dec. 1995.[22] H. B. Dwight,