Joint Rate Distortion Function of a Tuple of Correlated Multivariate Gaussian Sources with Individual Fidelity Criteria
Evagoras Stylianou, Charalambos D. Charalambous, Themistoklis Charalambous
JJoint Rate Distortion Function of a Tuple ofCorrelated Multivariate Gaussian Sources withIndividual Fidelity Criteria
Evagoras Stylianou ∗ , Charalambos D. Charalambous † , and Themistoklis Charalambous ‡ ∗ Department of Electrical and Computer Engineering, ∗ Technical University of Munich, † Department of Electrical and Computer Engineering, University of Cyprus ‡ Department of Electrical Engineering and Automation, School of Electrical Engineering, Aalto UniversityEmails: [email protected], [email protected], themistoklis.charalambous@aalto.fi
Abstract —In this paper we analyze the joint rate distortionfunction (RDF), for a tuple of correlated sources taking values inabstract alphabet spaces (i.e., continuous) subject to two individ-ual distortion criteria. First, we derive structural properties of therealizations of the reproduction Random Variables (RVs), whichinduce the corresponding optimal test channel distributionsof the joint RDF. Second, we consider a tuple of correlatedmultivariate jointly Gaussian RVs, X : Ω → R p , X : Ω → R p with two square-error fidelity criteria, and we derive additionalstructural properties of the optimal realizations, and use theseto characterize the RDF as a convex optimization problem withrespect to the parameters of the realizations. We show that thecomputation of the joint RDF can be performed by semidefiniteprogramming. Further, we derive closed-form expressions of thejoint RDF, such that Gray’s [1] lower bounds hold with equality,and verify their consistency with the semidefinite programmingcomputations. We also verify our expressions reproduce theclosed-form formula of the joint RDF of scalar-valued RVs (i.e., p = p = ) derived by Xiao and Luo [2]. I. I
NTRODUCTION , P
ROBLEM F ORMULATION ,L ITERATURE , M
AIN R ESULTS
A. Literature Review
Gray [1, Theorem 3.1, Corollary 3.1] derived lower boundson the joint rate distortion functions (RDFs), of a tuple ofRandom Variables (RVs) taking values in arbitrary, abstractspaces, X : Ω → X , X : Ω → X , with a weighted distortion,expressed in terms of conditional RDFs, and marginal RDFs.Gray and Wyner in the seminal paper, Source Coding for aSimple Network [3], characterized the rate distortion regionof a tuple of correlated RVs, using the joint, conditionaland marginal RDFs. Xiao and Luo [2, Theorem 6] derivedthe closed-form expression of the joint RDF for a tuple ofscalar-valued correlated Gaussian RVs, with two square-errordistortion criteria, while Lapidoth and Tinguely [4] re-derivedXiao’s and Luo’s joint RDF using an alternative method.Xu, Liu and Chen [5] and Viswanatha, Akyol and Rose [6],generalized Wyner’s common information [7] to its lossycounter part, as the minimum common message rate on theGray and Wyner rate region with sum rate the joint RDF withtwo individual distortion functions. The analysis in [5], [6],includes the application of a tuple of scalar-valued, jointly Gaussian RVs. More recent work on rates that lie on the Grayand Wyner rate region are found in [8].
B. Main Problem of this Paper1) The Joint RDF with Individual Distortion Functions:
This paper is concerned with the joint RDF of a tuple of RVstaking values in abstract spaces (i.e., continuous-valued RVs), X : Ω → X , X : Ω → X of reconstructing X i by (cid:98) X i : Ω → (cid:98) X i ,for i = ,
2, , subject to two distortion functions d X i : X i × (cid:98) X i → [ , ∞ ) , i = ,
2, defined by R X , X ( ∆ , ∆ ) = inf M ( ∆ , ∆ ) I ( X , X ; (cid:98) X , (cid:98) X ) (I.1)where I ( X , X ; (cid:98) X , (cid:98) X ) is the mutual information of RVs ( X , X ) and ( (cid:98) X , (cid:98) X ) , the set M ( ∆ , ∆ ) is specified by M ( ∆ , ∆ ) = (cid:110) (cid:98) X : Ω → (cid:98) X , (cid:98) X : Ω → (cid:98) X (cid:12)(cid:12)(cid:12) P X , X , (cid:98) X , (cid:98) X has ( X , X ) -marginal P X , X , E (cid:8) d X i ( X i , (cid:98) X i ) (cid:9) ≤ ∆ i , i = , (cid:111) (I.2)and the level of distortions are ∆ i ∈ [ , ∞ ) , i = ,
2. The jointRDF characterizes the infimum of all achievable rates ofa sequence of rate distortion codes, ( f E , g D ) , as depictedin Figure I.1, of reconstructing ( X n , X n ) (cid:52) = { ( X , t , X , t ) : t = , , . . . , n } , by ( (cid:98) X n , (cid:98) X n ) (cid:52) = { ( (cid:98) X , t , (cid:98) X , t ) : t = , , . . . , n } , where (cid:98) X i , t : Ω → (cid:98) X i i = , , t = , , . . . , n and P X , t , X , t = P X , X , ∀ t ,with distortion n E { d X i ( X ni , (cid:98) X ni ) } ≤ ∆ i , i = ,
2, for sufficientlylarge n . The computation of R X , X ( ∆ , ∆ ) is indispensable inthe characterization of the Gray and Wyner rate region, andin the above mentioned applications.Our first objective is to identify structural properties ofrealizations of the tuple of RVs ( (cid:98) X , (cid:98) X ) in the set M ( ∆ , ∆ ) ,and structural properties of corresponding induced forward testchannel distributions P (cid:98) X , (cid:98) X | X , X or backward test channel dis-tributions P X , X | (cid:98) X , (cid:98) X , such that E (cid:8) d X i ( X i , (cid:98) X i ) (cid:9) ≤ ∆ i , i = , R X , X ( ∆ , ∆ ) .
2) The Joint RDF of a Tuple of Multivariate GaussianSources:
Our second objective is to compute the joint RDF R X , X ( ∆ , ∆ ) , of a tuple of jointly independent and identicallydistributed multivariate Gaussian RVs, ( X n , X n ) (cid:52) = { ( X , t , X , t ) : a r X i v : . [ c s . I T ] F e b ext TextText
Fig. I.1. Lossy Compression of correlated sources with individual distortioncriteria. t = , , . . . , n } , where X i , t : Ω → R p i , i = , , t = , , . . . , n ,i.e., P X , t , X , t = P X , X , ∀ t is a multivariate jointly Gaussiandistribution and denoted by ( X , X ) ∈ G ( , Q ( X , X ) ) , subjectto two square-error distortion functions, all defined by Q ( X , t , X , t ) = E (cid:26) (cid:18) X , t X , t (cid:19) (cid:18) X , t X , t (cid:19) T (cid:27) = (cid:32) Q X Q X , X Q T X , X Q X (cid:33) (I.3) X , t ∈ G ( , Q X ) , X , t ∈ G ( , Q X ) , ∀ t , (I.4) (cid:98) X , t : Ω → (cid:98) X (cid:52) = R p , (cid:98) X , t : Ω → (cid:98) X (cid:52) = R p ∀ t , (I.5) d X i ( x ni , (cid:98) x ni ) = n n ∑ t = || x i , t − (cid:98) x i , t || R pi , i = , . (I.6)Here X ∈ G ( , Q X ) means X is a Gaussian RV, with zero meanand symmetric nonnegative definite covariance matrix Q X (cid:23) C. Main Results of the Paper
The main contributions of this paper are:1) The derivation of structural properties of test channeldistributions P (cid:98) X , (cid:98) X | X , X , and corresponding realizationsof the reproduction RVs ( (cid:98) X , (cid:98) X ) which induce thesedistributions, and characterize R X , X ( ∆ , ∆ ) .2) The characterization of R X , X ( ∆ , ∆ ) for jointly Gaussianmultivariate sources, X : Ω → R p , X : Ω → R p , withsquare-error distortion criteria, (I.3)-(I.6), parametrizationof reproduction RVs ( (cid:98) X , (cid:98) X ) and corresponding test chan-nels, and calculation of R X , X ( ∆ , ∆ ) using convex nu-merical algorithms. Further, to derive closed-form expres-sions for R X , X ( ∆ , ∆ ) , to verify the numerical algo-rithms. This includes the distortion region D ( X , X ) , suchthat Gray’s lower bound [1] holds with equality, R X , X ( ∆ , ∆ ) = R X ( ∆ ) + R X ( ∆ ) − I ( X ; X ) . (I.7)For case p = p =
1, we verify that our results reproduce thevalue of the RDF derived by Xiao and Luo [2].II. P
ROPERTIES OF R EALIZATIONS OF T EST C HANNELS
Let Z and Z + be the set of integers and positive integers,respectively. Let R be the set of real numbers and R + = [ , ∞ ) .The expression R n × m denotes the set of n by m matriceswith elements the real numbers, for n , m ∈ Z + . For thesymmetric matrix Q ∈ R n × n , inequality Q (cid:31) Q (cid:23) Q (cid:23) Q means that Q − Q (cid:23)
0. For any matrix A ∈ R p × m , ( p , m ) ∈ Z + × Z + , we denote its transpose by A T ,and for m = p , we denote its trace and its determinant bytr ( A ) and det (cid:0) A (cid:1) , respectively. The n by n identity (zero)matrix is represented by I n (0 n ). For matrix A ∈ R p × p , diag ( A ) is a matrix with the diagonal entries of matrix A and zeroelsewhere. blkdiag ( A , B ) is a square diagonal matrix in whichthe diagonal elements are square matrices A ∈ R p × p and B ∈ R p × p , and the off-diagonal elements are zero. Givena triple of real-valued RVs X i : Ω → X i , i = , ,
3, we saythat RVs ( X , X ) are conditional independent given RV X if P X , X | X = P X | X P X | X − a.s (almost surely); the specificationa.s is often omitted. The mutual information between RV X and RV Y is denoted by I ( X ; Y ) .The conditional covariance of the two-component vector RV X = ( X T , X T ) T , X i : Ω → R p i , i = , (cid:98) X = ( (cid:98) X T , (cid:98) X T ) T , (cid:98) X i : Ω → R p i , i = , Q ( X , X ) | (cid:98) X (cid:52) = cov (cid:16) X , X (cid:12)(cid:12)(cid:12) (cid:98) X (cid:17) (cid:23)
0, where Q ( X , X ) | (cid:98) X = (cid:32) Q X | (cid:98) X Q X , X | (cid:98) X Q T X , X | (cid:98) X Q X | (cid:98) X (cid:33) ∈ R ( p + p ) × ( p + p ) , Q X , X | (cid:98) X (cid:52) = cov (cid:16) X , X (cid:12)(cid:12)(cid:12) (cid:98) X (cid:17) . ( ) = E (cid:110)(cid:16) X − E (cid:110) X (cid:12)(cid:12)(cid:12) (cid:98) X (cid:111)(cid:17)(cid:16) X − E (cid:110) X (cid:12)(cid:12)(cid:12) (cid:98) X (cid:111)(cid:17) T (cid:111) = E (cid:110) E E T (cid:111) , E i (cid:52) = X i − E (cid:110) X i (cid:12)(cid:12)(cid:12) (cid:98) X (cid:111) , i = , ( X , X , (cid:98) X , (cid:98) X ) is jointly Gaussian.Similarly for Q X i | (cid:98) X , i = ,
2. Consequently, for jointly Gaus-sian RVs ( X , X , (cid:98) X , (cid:98) X ) , and the two-component vector RV E (cid:52) = ( E T , E T ) T , we have Q ( X , X ) | (cid:98) X = Σ ( E , E ) . A. Structural Properties of Test Channels for Arbitrary Dis-tributed Sources
In this section, we identify a structural property of the tuple ( (cid:98) X , (cid:98) X ) to achieve a lower bound on I ( X , X ; (cid:98) X , (cid:98) X ) , for anytuple of RVs ( X , X ) with arbitrary distribution P X , X . Theorem II.1.
Let ( X , X , (cid:98) X , (cid:98) X ) be arbitrary RVs takingvalues in the abstract spaces X × X × (cid:98) X × (cid:98) X , with arbitraryjoint distribution P X , X , (cid:98) X , (cid:98) X , and joint marginal the fixeddistribution P X , X of ( X , X ) .(a) Define the conditional mean of X i conditioned on (cid:98) X =( (cid:98) X T , (cid:98) X T ) T , byX cm i = g cm i (cid:0) (cid:98) X , (cid:98) X (cid:1) (cid:52) = E (cid:110) X i (cid:12)(cid:12)(cid:12) (cid:98) X (cid:111) , i = , , (II.8) g cm i : (cid:98) X × (cid:98) X → (cid:98) X i , g cm i ( · ) are measurable functions, i = , . Then, the following inequality holds:I ( X , X ; (cid:98) X , (cid:98) X ) ≥ I (cid:0) X , X ; g cm1 ( (cid:98) X , (cid:98) X ) , g cm2 ( (cid:98) X , (cid:98) X ) (cid:1) . (II.9) Moreover, if there exist RVs ( (cid:98) X , (cid:98) X ) such that the functionsg cm i ( · , · ) satisfy g cm i ( (cid:98) X , (cid:98) X ) = (cid:98) X i − a.s for i = , , then theinequality in (II.9) holds with equality.(b) Let X × X × (cid:98) X × (cid:98) X = R p × R p × R p × R p , ( p , p ) ∈ Z + . For all measurable functions g i ( (cid:98) X , (cid:98) X ) , i = , then E (cid:110)(cid:12)(cid:12)(cid:12)(cid:12) X i − g i ( (cid:98) X , (cid:98) X ) (cid:12)(cid:12)(cid:12)(cid:12) R pi (cid:111) ≥ E (cid:110)(cid:12)(cid:12)(cid:12)(cid:12) X i − E (cid:110) X i (cid:12)(cid:12)(cid:12) (cid:98) X (cid:111)(cid:12)(cid:12)(cid:12)(cid:12) R pi (cid:111) , i = , . c) If the conditions of parts (a) hold, i.e., g cm i ( (cid:98) X , (cid:98) X ) = (cid:98) X i − a.s , i = , , and X × X × (cid:98) X × (cid:98) X = R p × R p × R p × R p , ( p , p ) ∈ Z + , then the joint RDF of (I.1) is characterized byR X , X ( ∆ , ∆ ) = inf M cm ( ∆ , ∆ ) I ( X , X ; (cid:98) X , (cid:98) X ) (II.10) where (cid:98) X i = E (cid:110) X i (cid:12)(cid:12)(cid:12) (cid:98) X (cid:111) , i = , , and M cm ( ∆ , ∆ ) is specifiedby the subset M cm ( ∆ , ∆ ) ⊆ M ( ∆ , ∆ ) , with the additionalrestriction (cid:98) X i = E (cid:110) X i (cid:12)(cid:12)(cid:12) (cid:98) X (cid:111) for i = , .Proof. (a) By properties of mutual information, we have I ( X , X ; (cid:98) X , (cid:98) X ) ( ) = I ( X , X ; (cid:98) X , (cid:98) X , X cm1 , X cm2 ) ( ) = I ( X , X ; (cid:98) X , (cid:98) X | X cm1 , X cm2 )+ I ( X , X ; X cm1 , X cm2 ) ( ) ≥ I ( X , X ; X cm1 , X cm2 ) , (II.11)where ( ) is due to X cm i , i = ,
2, are functions of ( (cid:98) X , (cid:98) X ) , ( ) is due to the chain rule of mutual information, and ( ) is due to I ( X , X ; (cid:98) X , (cid:98) X | X cm1 , X cm2 ) ≥
0. Thus, (II.9)is obtained. If g cm i ( (cid:98) X , (cid:98) X ) = (cid:98) X i − a.s , i = , I ( X , X ; (cid:98) X , (cid:98) X | X cm1 , X cm2 ) =
0, and hence the inequality (II.11)become equality. ( b ) The inequality is well-known, due to theorthogonal projection theorem. (c) This is due to (a), (b).III. S TRUCTURAL P ROPERTIES OF T EST C HANNELS AND C HARACTERIZATION OF J OINT
RDF
FOR M ULTIVARIATE J OINTLY G AUSSIAN S OURCES
This section makes use of Theorem II.1 to derive addi-tional structural properties of test channels for the joint RDF R X , X ( ∆ , ∆ ) of jointly Gaussian sources with square-errordistortions, defined by (I.3)-(I.6). Theorem III.1 (Sufficient conditions for the lower boundsof Theorem II.1 to be achieved) . Consider the quadruple ofzero mean RVs ( X , X , (cid:98) X , (cid:98) X ) taking values in R p × R p × R p × R p , p , p ∈ Z + , with jointly Gaussian distribution i.e, P X , X , (cid:98) X , (cid:98) X = P GX , X , (cid:98) X , (cid:98) X and joint marginal the fixed Gaussiandistribution P X , X = P GX , X of ( X , X ) . Define the vectors,X = (cid:18) X X (cid:19) , (cid:98) X = (cid:32) (cid:98) X (cid:98) X (cid:33) , X cm (cid:52) = E (cid:40) (cid:18) X X (cid:19) (cid:12)(cid:12)(cid:12) (cid:98) X (cid:41) = (cid:18) X cm1 X cm2 (cid:19) . Suppose Condition 1 below holds:Condition 1. cov (cid:0) X , (cid:98) X (cid:1)(cid:8) cov (cid:0) (cid:98) X , (cid:98) X (cid:1)(cid:9) − = I p + p . (III.12) Then, the following equalities hold.X cm1 (cid:52) = E (cid:110) X (cid:12)(cid:12)(cid:12) (cid:98) X (cid:111) = (cid:98) X , X cm2 (cid:52) = E (cid:110) X (cid:12)(cid:12)(cid:12) (cid:98) X (cid:111) = (cid:98) X . (III.13) Moreover, the statements of Theorem II.1 hold, provided thereexist RVs ( (cid:98) X , (cid:98) X ) such that Condition 1 holds.Proof. By properties of jointly Gaussian RVs, X cm = E (cid:8) X (cid:9) + cov (cid:0) X , (cid:98) X (cid:1)(cid:8) cov (cid:0) (cid:98) X , (cid:98) X (cid:1)(cid:9) − (cid:16) (cid:98) X − E (cid:8) (cid:98) X (cid:9)(cid:17) = (cid:98) X (by Condition 1 and zero mean assumption) . The second part is due to Condition 1.In the next lemma, we apply Theorem II.1 and Theo-rem III.1 to find a parametric jointly Gaussian realization of ( (cid:98) X , (cid:98) X ) , that induces the set of test channels of the joint RDF R X , X ( ∆ , ∆ ) for (I.3)-(I.6). Lemma III.1 (Preliminary parametrization of test channel) . Consider the joint RDF R X , X ( ∆ , ∆ ) for (I.3)-(I.6). Thefollowing hold.(a) A jointly Gaussian distribution P X , X , (cid:98) X , (cid:98) X minimizesI ( X , X ; (cid:98) X , (cid:98) X ) , subject to two average distortions.(b) The test channel distribution P (cid:98) X , (cid:98) X | X , X of the jointRDF R X , X ( ∆ , ∆ ) is induced by the parametric Gaussianrealization of ( (cid:98) X , (cid:98) X ) , in terms of the matrices ( H , Q V ) , as (cid:98) X = HX + V (III.14) H ∈ R ( p + p ) × ( p + p ) , V : Ω → R ( p + p ) , (III.15) V ∈ G ( , Q ( V , V ) ) , Q ( V , V ) (cid:23) , V and X indep. , (III.16) (c) Consider part (b) and suppose there exists matrices ( H , Q ( V , V ) ) such that Condition 1 of Theorem III.1 holds, i.e.X cm i = (cid:98) X i -a.s. for i = , . Then the infimum of R X , X ( ∆ , ∆ ) is taken over the subset M cm , G ( ∆ , ∆ ) ⊆ M cm ( ∆ , ∆ ) , M cm , G ( ∆ , ∆ ) (cid:52) = (cid:110) (cid:98) X : Ω → R ( p + p ) (cid:12)(cid:12)(cid:12) (III.14) − (III.16) hold , X cm i = (cid:98) X i , E (cid:8) || X i − (cid:98) X i || R pi (cid:9) ≤ ∆ i , i = , (cid:111) (III.17) Proof. (a) This is similar to the classical RDF R X ( ∆ ) ofa Gaussian RV X ∈ G ( , Q X ) with square-error distortion.(b) By part (a), the test channel distribution P (cid:98) X , (cid:98) X | X , X isconditionally Gaussian with linear conditional mean E (cid:8) X | (cid:98) X (cid:9) and non-random covariance cov ( X , (cid:98) X | X ) . Such a distributionis induced by the realizations (III.14)-(III.16). (c) If Condition1 of Theorem III.1 holds, then (III.13) holds, and we obtainan achievable lower bound due to Theorem III.1. (b), (c).Next, we construct ( H , Q ( V , V ) ) such that X cm i = E (cid:8) X i | (cid:98) X (cid:9) = (cid:98) X i − a . s for i = ,
2, and characterize R X , X ( ∆ , ∆ ) . Theorem III.2 (Realization of optimal test channels andcharacterization of joint RDF) . Consider the joint RDFR X , X ( ∆ , ∆ ) for (I.3)-(I.6).(a) The test channel distribution P (cid:98) X , (cid:98) X | X , X of the RDFR X , X ( ∆ , ∆ ) is induced by the parametrize realization (III.14) - (III.16) , where the matrices, ( H , Q V ) satisfy,HQ ( X , X ) = Q ( X , X ) − Σ ( E , E ) = Q ( X , X ) H T (cid:23) , (III.18) Q ( V , V ) = HQ ( X , X ) − HQ ( X , X ) H T (cid:23) . (III.19) Moreover, R X , X ( ∆ , ∆ ) is characterized by,R X , X ( ∆ , ∆ ) = inf Q † ( ∆ , ∆ )
12 log (cid:110) det (cid:0) Q ( X , X ) (cid:1) det (cid:0) Σ ( E , E ) (cid:1) (cid:111) , (III.20) Q † ( ∆ , ∆ ) (cid:52) = (cid:110) Σ ( E , E ) : ( H , Q ( V , V ) ) satisfy (III.18), (III.19) , tr (cid:0) Σ E (cid:1) ≤ ∆ , tr (cid:0) Σ E (cid:1) ≤ ∆ (cid:111) . (III.21) b) Suppose Q ( X , X ) (cid:31) . If R X , X ( ∆ , ∆ ) < ∞ , then thematrices, ( H , Q ( V , V ) ) , of part (a) reduce to,H = I p + p − Σ ( E , E ) Q − ( X , X ) , (III.22) Q ( V , V ) = Σ ( E , E ) − Σ ( E , E ) Q − ( X , X ) Σ ( E , E ) (cid:23) , (III.23) Q ( X , X ) − Σ ( E , E ) (cid:23) , ⇐⇒ (III.24) Σ ( E , E ) − Σ ( E , E ) Q − ( X , X ) Σ ( E , E ) (cid:23) . (III.25) and Q † ( ∆ , ∆ ) in (III.20) is replaced by ◦ Q ( ∆ , ∆ ) , given by ◦ Q ( ∆ , ∆ ) (cid:52) = (cid:110) Σ ( E , E ) : Q ( X , X ) (cid:23) Σ ( E , E ) (cid:23) , tr (cid:0) Σ E (cid:1) ≤ ∆ , tr (cid:0) Σ E (cid:1) ≤ ∆ (cid:111) . (III.26) Proof.
See Appendix VI.
Lemma III.2.
Consider R X , X ( ∆ , ∆ ) of Theorem III.2, de-fined by (III.20) and assume Q ( X , X ) (cid:31) , and R X , X ( ∆ , ∆ ) < + ∞ . The Lagrange functional is, L (cid:52) =
12 log (cid:110) det (cid:0) Q ( X , X ) (cid:1) det (cid:0) Σ ( E , E ) (cid:1) (cid:111) + tr (cid:16) Θ (cid:16) Σ ( E , E ) − Q ( X , X ) (cid:17)(cid:17) + λ (cid:16) tr (cid:16) Σ E (cid:17) − ∆ (cid:17) + λ (cid:16) tr (cid:16) Σ E (cid:17) − ∆ (cid:17) − tr (cid:16) V Σ ( E , E ) (cid:17) where Θ (cid:23) , V (cid:23) , λ i ∈ [ , ∞ ) , i = , . The optimal Σ ( E , E ) ∈ ◦ Q ( ∆ , ∆ ) for R X , X ( ∆ , ∆ ) is found as follows.(i) Stationarity: − Σ − ( E , E ) + (cid:20) λ I p λ I p (cid:21) + Θ + V = . (III.27) (ii) Complementary Slackness: λ (cid:16) tr (cid:16) Σ E (cid:17) − ∆ (cid:17) = , λ (cid:16) tr (cid:16) Σ E (cid:17) − ∆ (cid:17) = , (III.28)tr (cid:16) V Σ ( E , E ) (cid:17) = , tr (cid:16) Θ (cid:16) Σ ( E , E ) − Q ( X , X ) (cid:17)(cid:17) = . (III.29) (iii) Primal Feasibility: Defined by ◦ Q ( ∆ , ∆ ) .(iv) Dual Feasibility: λ ≥ , λ ≥ , Θ (cid:23) , V (cid:23) .Moreover, the following hold.(a) V = , and Σ ( E , E ) = (cid:32) (cid:20) λ I p λ I p (cid:21) + Θ (cid:33) − (cid:31) . (III.30) (b) If Q ( X , X ) − Σ ( E , E ) (cid:31) then Θ = , and Σ ( E , E ) = (cid:32) (cid:20) λ I p λ I p (cid:21) (cid:33) − (cid:31) . (III.31) Proof.
The derivation is standard hence it is omitted.The next two theorems are obtained from Lemma III.2.
Theorem III.3 (Joint RDF for positive surface) . Consider thecharacterization of joint RDF R X , X ( ∆ , ∆ ) of Theorem III.2,defined by (III.20), and assume Q ( X , X ) (cid:31) (i.e., Q ( X , X ) (cid:31) implies Q X (cid:31) , Q X (cid:31) ). Define the set D ( X , X ) = (cid:26) ( ∆ , ∆ ) ∈ [ , ∞ ) × [ , ∞ ) (cid:12)(cid:12)(cid:12)(cid:12) Q ( X , X ) − Σ ( E , E ) (cid:31) (cid:27) . The joint RDF R X , X ( ∆ , ∆ ) for ( ∆ , ∆ (cid:1) ∈ D ( X , X ) isR X , X (cid:0) ∆ , ∆ (cid:1) =
12 log (cid:26) det (cid:0) Q ( X , X ) (cid:1) det (cid:0) Σ E (cid:1) det (cid:0) Σ E (cid:1) (cid:27) = (I.7) Σ E = diag (cid:16) ∆ p , . . . , ∆ p (cid:17) , Σ E = diag (cid:16) ∆ p , . . . , ∆ p (cid:17) and this is achieved by the covariance matrix Σ ( E , E ) with Σ E , E = Q X , X | (cid:98) X = , and Gray’s lower bound (I.7) holds.Proof. For any element of the set D ( X , X ) then Q ( X , X ) − Σ ( E , E ) (cid:31)
0, and the statements follow from Lemma III.2.
Remark III.1.
For the scalar-valued RVs, i.e., p = p = ,we have verified that Lemma III.2 produces the closed-formexpression of R X , X ( ∆ , ∆ ) as derived in [2, Theorem 6].However, for the multivariate case of Lemma III.2, to obtainthe closed-form expression is challenging. To make the prob-lem tractable, in Theorem III.4, we use the canonical variableform of the tuple ( X , X ) , as described in [8] and [9]. Theorem III.4.
Consider the statement of Theorem III.2.(b),with ( X , X ) ∈ G ( , Q ( X , X ) ) , Q ( X , X ) (cid:31) . Determine thecanonical variable form of the tuple ( X , X ) , according to [9,Definition 2.2] by using algorithm [9, Algorithm 2.10], andrestrict attention to indices, p = p = (i.e., the identicalcomponents of X and X are removed). Then, p = p (i.e.,the number of correlated components of X and X ), p , p (i.e., the number of independent components of X wrt X andvice versa), and n = p = p , p = p + p , p = p + p .Similarly, transform ( E , E ) ∈ G ( , Σ ( E , E ) ) to its canonicalvariable form with p = p = (i.e., identical componentsare removed). Then, p = p , p , p , and n = p = p ,p = p + p , p = p + p .The above yield nonsingular matrices X i (cid:55)→ X ci (cid:52) = S i X i , E i (cid:55)→ E ci (cid:52) = S i E i , i = , , and the covariance matrices,Q ( X c , X c ) = Q cv f = (cid:18) I p D D T I p (cid:19) , D = (cid:18) D
00 0 (cid:19) ∈ R p × p , D = diag ( d , , ..., d , n ) ∈ R n × n , > d , ≥ . . . ≥ d , n > , Σ ( E c , E c ) = Σ cv f = (cid:18) I p D , D T I p (cid:19) , D = (cid:18) D
00 0 (cid:19) ∈ R p × p , D = diag ( d , , ..., d , n ) ∈ R n × n , > d , ≥ . . . ≥ d , n > . The joint RDF R X , X ( ∆ , ∆ ) of Theorem III.2.(b), whenQ ( X , X ) − Σ ( E , E ) (cid:23) but not Q ( X , X ) − Σ ( E , E ) (cid:31) is equiv-alently characterized byR X , X ( ∆ , ∆ ) = inf ◦ Q ( ∆ , ∆ )
12 log (cid:110) det (cid:0) D (cid:1) det (cid:0) D (cid:1) det (cid:0) Q cv f (cid:1) det (cid:0) D (cid:1) det (cid:0) D (cid:1) det (cid:0) Σ cv f (cid:1) (cid:111) , ◦ Q ( ∆ , ∆ ) (cid:52) = (cid:110) n ∈ Z + , d , i ∈ ( , ) , i = , . . . , n , d , i ∈ ( , ∞ ) , i = , . . . , p , d , i ∈ ( , ∞ ) , i = , . . . , p : p ∑ i = d , i ≤ ∆ , p ∑ i = d , i ≤ ∆ , Q ( X , X ) − Σ ( E , E ) (cid:23) (cid:111) ig. III.2. Joint RDF R X , X ( ∆ , ∆ ) of source of Sectiion IV, p = p = where det (cid:0) Q cv f (cid:1) = det (cid:0) I p − D D T (cid:1) = (cid:40) , if p > , p > , p = p = , ∏ ni = (cid:16) − d , i (cid:17) , if p = p = n , p ≥ , p ≥ , det (cid:0) Σ cv f (cid:1) = det (cid:0) I p − D D T (cid:1) = (cid:40) , if p > , p > , p = p = , ∏ ni = (cid:16) − d , i (cid:17) , if p = p = n , p ≥ , p ≥ . Proof.
By Theorem III.2.(b) and applying [9, Definition 2.2]and algorithm [9, Algorithm 2.10] we obtain the results.
Remark III.2. R X , X ( ∆ , ∆ ) of Theorem III.4, is much easierto optimize, due to its structure. IV. N
UMERICAL E VALUATION OF THE J OINT
RDFWe can express the optimization problems of Theorem III.2as a semidefinite program (SDP) as follows, define Ξ T = blkdiag (cid:0) I p p (cid:1) and Ξ T = blkdiag (cid:0) p I p (cid:1) ,min Σ ( E , E )
12 log (cid:110) det (cid:0) Q ( X , X ) (cid:1) det (cid:0) Σ ( E , E ) (cid:1) (cid:111) s . t . Q ( X , X ) − Σ ( E , E ) (cid:23) , Σ ( E , E ) (cid:23) , tr (cid:0) Ξ T i Σ ( E , E ) Ξ i (cid:1) ≤ ∆ i , i = , Σ ( E , E ) for amultivariate example X i : Ω → R , i = ,
2, with covariance, Q ( X , X ) = . − .
11 0 .
642 0 . − .
11 2 . − .
859 0 . . − .
859 2 .
142 1 . .
976 0 .
337 1 .
797 3 . . Fig. III.2 depicts R X , X ( ∆ , ∆ ) , ( ∆ , ∆ ) ∈ [ , ∞ ) × [ , ∞ ) . Be-low we distinguish two cases. Case 1.
Given distortions ( ∆ , ∆ ) = ( . , . ) , the solutionof (III.20), (III.26) is given by Σ ( E , E ) = diag (cid:0) . , . , . , . (cid:1) , Q ( X , X ) − Σ ( E , E ) (cid:31) ∆ and ∆ are equally divided among the diagonalelements of the first and second 2-by-2 diagonal blocks of Σ ( E , E ) respectively, and the rest of the values are zero. Hence, ( . , . ) ∈ D ( X , X ) ; this re-confirms Theorem III.3. Case 2.
Given distortions ( ∆ , ∆ ) = ( . , . ) , the optimalerror covariance matrix is given by, Σ ( E , E ) = . − . − . . − . . − .
144 0 . − . − .
144 0 .
804 0 . . . .
293 1 . and Q ( X , X ) − Σ ( E , E ) (cid:23)
0. Unlike Case 1, Σ ( E , E ) is notblock-diagonal, i.e., Σ E , E (cid:54) =
0, as in Theorem III.3, hence ( . , . ) / ∈ D ( X , X ) . This choice of distortions correspondsto Lemma III.2.(b). V. C ONCLUSION
The joint RDF R X , X ( ∆ , ∆ ) , with individual distortioncriteria, is analyzed, with emphasis on the structural prop-erties of realizations of the reproduction RVs ( (cid:98) X , (cid:98) X ) of ( X , X ) , and corresponding optimal test channel distribution, P (cid:98) X , (cid:98) X | X , X . Closed-form expression of R X , X ( ∆ , ∆ ) are de-rived for strictly positive surface of distortion region, and anumerical technique is presented, which can be easily carriedout for any finite p and p , and verified the closed-formexpressions. VI. A PPENDICES
Proof of Theorem III.2 . Consider (III.14)-(III.16). To iden-tify ( H , Q ( V , V ) ) such that X cm i = E (cid:110) X i (cid:12)(cid:12)(cid:12) (cid:98) X (cid:111) = (cid:98) X i , i = , X and (cid:98) X is, Q X , (cid:98) X = E (cid:110) X (cid:16) HX + V (cid:17) T (cid:111) = Q ( X , X ) H T . (VI.33)By (III.14)-(III.16), the covariance of (cid:98) X = HX + V is Q (cid:98) X = E (cid:110) (cid:98) X (cid:98) X T (cid:111) = HQ ( X , X ) H T + Q ( V , V ) , (VI.34)By Condition 1, i.e (III.12), thencov (cid:0) X , (cid:98) X (cid:1)(cid:8) cov (cid:0) (cid:98) X , (cid:98) X (cid:1)(cid:9) − = I p + p ⇐⇒ Q X , (cid:98) X Q − (cid:98) X = I p + p = ⇒ Q X H T = HQ X H T + Q ( V , V ) by (VI.33), (VI.34) = ⇒ Q ( V , V ) = Q ( X , X ) H T − HQ ( X , X ) H T . (VI.35)Next, we turn to the identification of H . By the definition ofcovariance of the errors, then Σ ( E , E ) (cid:52) = cov ( X , X | (cid:98) X ) , and Σ ( E , E ) = cov ( X , X ) − cov ( X , (cid:98) X ) (cid:8) cov ( (cid:98) X , (cid:98) X ) (cid:9) − cov ( X , (cid:98) X ) T = Q ( X , X ) − HQ ( X , X ) , by (III.12),(VI.33) = ⇒ HQ ( X , X ) = Q ( X , X ) − Σ ( E , E ) = Q ( X , X ) H T (VI.36) = ⇒ H = I p + p − Σ ( E , E ) Q − ( X , X ) , if Q ( X , X ) (cid:31) . Using (VI.36) into (VI.35) then we have Q ( V , V ) = Q ( X , X ) H T − HQ ( X , X ) H T = Q T ( V , V ) (VI.37) = Q ( X , X ) − Σ ( E , E ) − HQ ( X , X ) H T . (VI.38)From the above follow ( H , Q ( V , V ) ) and the rest of thestatements are easily obtained. EFERENCES[1] R. Gray, “A new class of lower bounds to information rates of stationarysources via conditional rate-distortion functions,”
IEEE Transactions onInformation Theory , vol. 19, no. 4, pp. 480–489, 1973.[2] J.-J. Xiao and Z.-Q. Luo, “Compression of correlated gaussian sourcesunder individual distortion criteria,” in , 2005, pp. 438–447.[3] R. Gray and A. Wyner, “Source coding for a simple network,”
BellSystem Technical Journal , vol. 53, no. 9, pp. 1681–1721, 1974.[4] A. Lapidoth and S. Tinguely, “Sending a bivariate gaussian over agaussian mac,”
IEEE Transactions on Information Theory , vol. 56, no. 6,pp. 2714–2752, 2010.[5] G. Xu, W. Liu, and B. Chen, “A lossy source coding interpretationof wyner’s common information,”
IEEE Transactions on InformationTheory , vol. 62, no. 2, pp. 754–768, 2015.[6] K. B. Viswanatha, E. Akyol, and K. Rose, “The lossy common informa-tion of correlated sources,”
IEEE Transactions on Information Theory ,vol. 60, no. 6, pp. 3238–3253, 2014.[7] A. Wyner, “The common information of two dependent random vari-ables,”
IEEE Transactions on Information Theory , vol. 21, no. 2, pp.163–179, 1975.[8] C. D. Charalambous and J. H. van Schuppen, “Characterization ofconditional independence and weak realizations of multivariate gaussianrandom variables: Applications to networks,” in . IEEE, 2020, pp. 2444–2449.[9] ——, “A new approach to lossy network compression of a tuple ofcorrelated multivariate gaussian rvs,” arXiv preprint arXiv:1905.12695arXiv preprint arXiv:1905.12695