Photios A. Stavrou

Nonanticipative Rate Distortion Function and Relations to Filtering Theory

The relation between nonanticipative rate distortion function (RDF) and filtering theory is discussed on abstract spaces. The relation is established by imposing a realizability constraint on the reconstruction conditional distribution of the classical RDF. Existence of the extremum solution of the nonanticipative RDF is shown using weak *-convergence on appropriate topology. The extremum reconstruction conditional distribution is derived in closed form, for the case of stationary processes. The realization of the reconstruction conditional distribution which achieves the infimum of the nonanticipative RDF is described. Finally, an example is presented to illustrate the concepts.

Directed Information on Abstract Spaces: Properties and Variational Equalities

Directed information or its variants are utilized extensively in the characterization of the capacity of channels with memory and feedback, nonanticipative lossy data compression, and their generalizations to networks. In this paper, we derive several functional and topological properties of directed information, defined on general abstract alphabets (complete separable metric spaces), using the topology of weak convergence of probability measures. These include the convexity of the set of consistent distributions, which uniquely define causally conditioned distributions, convexity, and concavity of directed information with respect to the sets of consistent distributions, weak compactness of such sets of distributions, their joint distributions, and their marginals. Furthermore, we show lower semicontinuity of directed information, and under certain conditions, we also establish continuity. Finally, we derive variational equalities for directed information, including sequential versions. These may be viewed as the analog of the variational equalities of mutual information (utilized in Blahut-Arimoto algorithms). In summary, we extend the basic functional and topological properties of mutual information to directed information. These properties are discussed throughout this paper, in the context of extremum problems of directed information.

Directed information on abstract spaces: Properties and extremum problems

This paper describes a framework in which directed information is defined on abstract spaces. The framework is employed to derive properties of directed information such as convexity, concavity, lower semicontinuity, by using the topology of weak convergence of probability measures on Polish spaces. Two extremum problems of directed information related to capacity of channels with memory and feedback, and non-anticipative and sequential rate distortion are analyzed showing existence of maximizing and minimizing distributions, respectively.

Variational equalities of directed information and applications

In this paper we introduce two variational equalities of directed information, which are analogous to those of mutual information employed in the Blahut-Arimoto Algorithm (BAA). Subsequently, we introduce nonanticipative Rate Distortion Function (RDF) R_{o, n}^na(D) defined via directed information introduced in, and we establish its equivalence to Gorbunov-Pinskers nonanticipatory ε-entropy R_{o, n}^ε(D). By invoking certain results we first establish existence of the infimizing reproduction distribution for R_{o, n}^na(D), and then we give its implicit form for the stationary case. Finally, we utilize one of the variational equalities and the closed form expression of the optimal reproduction distribution to provide an algorithm for the computation of R_{o, n}^na(D).

Information Nonanticipative Rate Distortion Function and Its Applications

This paper investigates applications of nonanticipative Rate Distortion Function (RDF) in a) zero-delay Joint Source-Channel Coding (JSCC) design based on average and excess distortion probability, b) in bounding the Optimal Performance Theoretically Attainable (OPTA) by noncausal and causal codes, and computing the Rate Loss (RL) of zero-delay and causal codes with respect to noncausal codes. These applications are described using two running examples, the Binary Symmetric Markov Source with parameter p, (BSMS(p)) and the multidimensional partially observed Gaussian-Markov source. For the multidimensional Gaussian-Markov source with square error distortion, the solution of the nonanticipative RDF is derived, its operational meaning using JSCC design via a noisy coding theorem is shown by providing the optimal encoding-decoding scheme over a vector Gaussian channel, and the RL of causal and zero-delay codes with respect to noncausal codes is computed. For the BSMS(p) with Hamming distortion, the solution of the nonanticipative RDF is derived, the RL of causal codes with respect to noncausal codes is computed, and an uncoded noisy coding theorem based on excess distortion probability is shown. The information nonanticipative RDF is shown to be equivalent to the nonanticipatory -entropy, which corresponds to the classical RDF with an additional causality or nonanticipative condition imposed on the optimal reproduction conditional distribution. Index Terms Nonanticipative RDF, sources with memory, Joint Source-Channel Coding (JSCC), Binary Symmetric Markov Source (BSMS), multidimensional stationary Gaussian-Markov source, excess distortion probability, bounds.

Sequential Necessary and Sufficient Conditions for optimal channel input distributions of channels with memory and feedback

We derive Sequential Necessary and Sufficient Conditions (SNSC) for any channel input distribution P_0,n=̑{P((X_t)|X^t-1,Y^t-1):t=0,1,...,n} to maximize directed information for channel distributions of the form {P(Y_t|Y_t-M^t-1,x_t): t=0,1,...,n} where Xⁿ=̑{X₀...,X_n} , and Yⁿ=̑{Y₀,...,Y_n} are the channel input and output random variables, and M is nonnegative and finite. The results are obtained using the information structures of the optimal channel input distributions and the corresponding Finite Transmission Feedback Information (FTFI) capacity, convexity properties of directed information, and dynamic programming recursions. The conditions are applied to a finite alphabet channel with M = 1 to derive recursive closed form expressions for the optimal (nonstationary) distributions, which achieve the FTFI capacity. Further, ergodic feedback capacity is obtained in closed form, using the asymptotic properties of the optimal distributions. A numerical example is presented to illustrate the convergence properties of the per unit time limiting version of the FTFI capacity.

Filtering with fidelity for time-varying Gauss-Markov processes

In this paper, we revisit the relation between Nonanticipative Rate Distortion (NRD) theory and real-time realizable filtering theory. Specifically, we give the closed form expression for the optimal nonstationary (time-varying) reproduction distribution of the Finite Time Horizon (FTH) Nonanticipative Rate Distortion Function (NRDF) and we establish its connection to real-time realizable filtering theory via a realization scheme utilizing time-varying fully observable multidimensional Gauss-Markov processes. As an application we provide the optimal filter with respect to a mean square error constraint. Unlike classical filtering theory, our filtering approach based on FTH NRDF is performed with waterfilling. We also derive a universal lower bound to the mean square error of any causal estimator to Gaussian processes based on the closed form expression of FTH NRDF. Our theoretical results are demonstrated via an illustrative example.

Applications of information Nonanticipative Rate Distortion Function

The objective of this paper is to further investigate various applications of information Nonanticipative Rate Distortion Function (NRDF) by discussing two working examples, the Binary Symmetric Markov Source with parameter p (BSMS(p)) with Hamming distance distortion, and the multidimensional partially observed Gaussian-Markov source. For the BSMS(p), we give the solution to the NRDF, and we use it to compute the Rate Loss (RL) of causal codes with respect to noncausal codes. For the multidimensional Gaussian-Markov source, we give the solution to the NRDF, we show its operational meaning via joint source-channel matching over a vector of parallel Gaussian channels, and we compute the RL of causal and zero-delay codes with respect to noncausal codes.

Optimization of directed information and relations to filtering theory

In this paper we generalize the relation between nonanticipative Rate Distortion Function (RDF) and filtering theory, to processes which are affected by the reproduction process, by utilizing the topology of weak convergence of probability measures. Specifically, this generalization is established via an optimization on the space of conditional distributions of the so-called directed information, subject to a fidelity constraint. Existence of the optimal reproduction distribution of the general nonanticipative RDF is shown, while the closed form expression of the optimal reproduction distribution is obtained for nonstationary processes. The expression of the optimal reproduction conditional distribution is recursively computed, backward in time. Finally, the realization procedure of the general nonanticipative RDF which is equivalent to joint-source channel matching for symbol-by-symbol transmission is described.

On Shannon’s Duality of a Source and a Channel and Nonanticipative Communication and Communication for Control

This chapter provides a brief introduction to the Fundamental Problem of Communication, as formulated by Shannon, and evolved over the years into various generalities, including the authors’ views on Duality of a Source to a Channel. Suggestions for further research are described, with emphasis on the importance of this duality in nonanticipative or real-time information transmission in both communication and communication for control, of delay-sensitive applications.