Some Relations between Divergence Derivatives and Estimation in Gaussian channels
CCLN: 10-xxx 1
Abstract — The minimum mean-square error of the estimation of a non-Gaussian signal where observed from an additive white Gaussian noise (WGN) channel's output, is analyzed. First, a quite general time-continuous channel model is assumed for which the behavior of the non-Gaussianess of the channel's output for small signal to noise ratio q, is proved. Then, It is assumed that the channel input's signal is composed of a (normalized) sum of N narrowband, mutually independent waves. It is shown that if N goes to infinity, then for any fixed q (no mater how big) both CMMSE and MMSE converge to the signal energy at a rate which is proportional to 1/N. Finally, a known result for the MMSE in the one-dimensional case, for small q, is used to show that all the first four terms in the Taylor expansion of the non-Gaussianess of the channel's output equal to zero.
Index Terms —non-Gaussianess, divergence, CMMSE, MMSE. I. I NTRODUCTION
In section II we show, under quite general time-continuous channel model, that the derivative at zero snr of the non-Gaussianess of the output equals zero. Then (section III) it is proved that a signal that is composed of the sum of N independent CW waves, is hard to be estimated where observed from its sum with Gaussian noise, for large N. In this case both MMSE (Minimum Mean-Square Error) and CMMSE (Causal Minimum Mean-Square Error) of the appropriate optimal estimations tend to the signal energy in a rate that is proportional to 1/N. This is true for any snr, no matter how big but fixed. In section IV we move in opposite direction. Based on a known result for the Taylor expansion of MMSE of a random variable X for small snr, it is shown that the values of the first three derivatives at zero snr, of the non-Gaussianess of the sum of a random variable X and a normal Gaussian variable W, equal to zero. An expression for the fourth derivative as a function of the third and the fourth moments of X is also given. The author is with the New Elective–Engineering Services Ltd, 34970 Haifa, Israel (e-mail: [email protected]). II. N ON -G AUSSIANESS OF THE OUTPUT OF TIME - CONTINUOUS G AUSSIAN CHANNEL WITH SMALL SNR
Let the process ξ be observed from the sum η t0 (t) w(t) q (s)ds, 0 t T η = + ξ ≤ ≤ ∫ , (1) where w is a standard Brownian motion, ξ is any process with zero mean, independent of W, with T 20
E (t)dt ξ = ξ < ∞ ∫ and q is a real constant. Without loss of generality we assume that the signal energy satisfies E 1 ξ = . Relations that connect the divergence between the measure in function space of the channel output process and the measure of the Gaussian process with the same covariance, to both CMMSE MMSE, achievable by optimal estimation of the input given the output are well known (e. g. [1]-[3]). We denote the non-Gaussianess of the channel output η by (cid:1) D(q) D( ) = η η , where (cid:1)
D( ) η η is the divergence between the measure induced by η in function space and that of η (cid:1) , where (cid:1) η is a Gaussian process with the same covariance function as that of η . Based on relations between the divergence and the minimum mean-square errors we show that D(q) is mainly determined for small q by its second derivative at q=0 . Lemma 1: As q 0 → , for any process ξ that fulfils (1) (2) 2 3 O = + , (2) where n(n) n q 0 dD (0) D(q)dq = = . Proof
In order to prove the lemma we recall the following relations between the divergence and the minimum mean-square errors [3]:
Some Relations between Divergence Derivatives and Estimation in Gaussian channels
Jacob Binia, Member, IEEE
LN: 10-xxx 2 ξ − ξ = (cid:1) , (3) dMMSE( ) MMSE( ) 2 D(q)dq ξ − ξ = (cid:1) . (4) In (3), (4) ξ (cid:1) is a Gaussian process with same covariance function as that of ξ . It is clear that as q 0 + → the values of CMMSE( ) ξ (cid:1) , CMMSE( ) ξ , MMSE( ) ξ (cid:1) and MMSE( ) ξ exceed one. Therefore, from (3), (4) q 0 dD(q)D(0) 0dq = = = . (5) By Taylor's theorem [4, p. 95] equation (5) implies (2). III. CMMSE
AND
MMSE
OF A SIGNAL WHICH IS COMPOSED
OF THE S UM OF CW I NTERFERERS IN A DDITIVE G AUSSIAN N OISE C HANNEL
Let us consider channel (1) with the following non-Gaussian interference signal i NN i k ii 1 = ξ = ξ = ω + θ ≤ ≤ ∑ . (6) In (6) all i a are mutually independent random variables, independent of w, identically distributed with i Ea =0, Ea 1 = . All phases i θ are uniformly distributed, mutually independent and independent of all i a and w. i Nik 1 πω = , are circular frequencies. In this case the energy of ξ fulfils for each N E 1 ξ = . Then, the following asymptotic mean square errors for large N hold: Proposition 1:
For each fixed q, as N → ∞ (2)N 2 O ξ = − + + , (7) (2)N 2 O ξ = − + + . (8) In (7), (8) (2) D (0) is as in (2), for the signal (6) where N=1.
The meaning of (7), (8) is that even for unlimited (but fixed) q, both CMMSE and MMSE tend to the signal energy as N goes to infinity with a rate that is proportional to 1/N. .
Proof
Let us replace the signal ξ in (1) with the signal N ξ (6), where i a 1, 1 i N = ≤ ≤ . Then the following results for the estimation errors N CMMSE( ) ξ and N MMSE( ) ξ were proved in [3]: (cid:1) N N
2N 1 q 2CMMSE( ) ln(1 ) D ( )q 2 N q ξ = + − η η , (9) (cid:1)
N N ξ = − η η+ . (10) In (9), (10) the divergence (cid:1) N D ( ) η η , is the sum of individual divergences (which are the same for each i, see also [3, section III]). Therefore (cid:1) N qD ( ) ND( )N η η = , (11) where D( ) • is the diminution of the divergence (cid:1) N D ( ) η η to the case N=1. The generalization of (9)-(11) to the case of signal with mutually independent, identical distributed random amplitudes i {a } is straightforward. By (11) and (2), as N → ∞ (cid:1) (2) 2N 32(2) 2 q N q 1D ( ) N D( ) D (0) ( ) N ( )N 2 N N1 q 1D (0) ( ), N .2 N N OO η η = = += + → ∞ (12) From (9), (10) and (12) we get (7), (8). Consider now the following signal that is composed from the sum of Gaussian narrowband waves i i
NN ci k si ki 1
1g (t) (a cos t a sin t)T = = ω + ω ∑ , (13) where { } ci s i a , a , N1 i are mutually independent, zero mean Gaussian random variables with variance 1/N, and i Nk 1 , i ω are as in (6). Since N ξ and N g possess same covariance we can consider N g as a realization of N ξ (cid:1) . In order to evaluate the mean-square errors of N g we can use the fact that both CMMSE and MMSE seek their maximum for the Gaussian signal. Hence, from (9), (10) N
2N qCMMSE(g ) ln(1 )q 2N = + , (14) N = + . (15) As we can see the estimation errors of the above Gaussian interference also converge to the interference energy with rate that is proportional to 1/N. Note that the inability to estimate the signal of the above examples for infinite N follows from two consequences. First, LN: 10-xxx 3 by (12) the measure of the normalized sum of the non-Gaussian narrowband waves becomes "more Gaussian" (in the sense of divergence) as N goes to infinity. Hence, the negative parts in (9), (10) vanish. Second, Gaussian processes are impossible to be estimated if all their spectral components are mutually independent and deeply immersed in the noise. In other words, since all spectral components of the signal are independent (such that there is no mutual information between any disjoint sets of components), they have to be estimated separately. However, in this case each individual snr is low although the total snr is high. IV. N ON -G AUSSIANESS AND
MMSE F OR L OW SNR
In this section we use a known result for MMSE in order to show non-Gaussianess behavior for small snr. Let
Y W q X = + (16) Where X is a random variable with EX < ∞ , independent of W = N (0,1) and q expresses snr. Without loss of generality we assume EX 0, EX 1 = = . The Taylor series expansion of the MMSE around q 0 + = to the third order has been obtained in [1] as = − + − −− + + O (17) For more properties of MMSE in the case (16) see [5]. Using lemma 1 we can express
MMSE(X,q) as a Taylor series that is dependent on the derivatives of the non-Gaussianess of Y , for small snr. Comparing the resulting expression to (17) above will allow us to find derivatives of D(q) at q 0 = as a function of the moments of X . First, let us note that the special case of (1), where (t) 2X cos t, 0 t 2 ξ = ≤ ≤ π is equivalent to (16) for same X . This could be seen immediately by representing signals in function space with the Fourier family basis (one dimensional case). Therefore MMSE and divergence calculations, as well as lemma 1 bring same results in both spaces. Expanding lemma 1 to forth order yields (2) 2 (3) 3(4) 4 5 O = + ++ → (18) From (4), (18) and well known result for MMSE of Gaussian random variable where X is replaced in (16) by X (cid:1) = N (0,1), we get (2) (3) 2(4) 3 4 (2)(3) 2 (4) 3 4 = − + +++ = − + +− − + + O O (19) Comparing (19) with (17) leads to the following proposition
Proposition 2:
The non-Gaussianess D(q) of the sum (16) of any random variable with zero mean and unit variance, and a Gaussian variable N (0,1), satisfies: (1) (2) (3) D(0) D (0) D (0) D (0) 0 = = = = (20) (4) 4 2 4 3 2
1D (0) [(EX ) 6EX 2(EX ) 9]2 = − − + (21) Hence, in the special case of random variables (16) , no matter how large is the non-Gaussianess of X, the first four terms in the Taylor expansion of the non-Gaussianess of the sum Y equal zero. R
EFERENCES [1]
D. Guo, S. Shamai and S. Verdú, "Mutual information and minimum mean-square error in Gaussian channels", IEEE Trans. Inform. Theory, vol. 51 Issue 4, pp. 1261-1282, Apr. 2005. [2]
M. Zakai, "On likelihood-ratios, mutual information and estimation error for the additive Gaussian channel", IEEE Trans. Inform. Theory, vol. 51 Issue 9, pp. 3017-3024, Sept. 2005. [3]