Lee D. Davisson

Universal noiseless coding

Universal coding is any asymptotically optimum method of block-to-block memoryless source coding for sources with unknown parameters. This paper considers noiseless coding for such sources, primarily in terms of variable-length coding, with performance measured as a function of the coding redundancy relative to the per-letter conditional source entropy given the unknown parameter. It is found that universal (i.e., zero redundancy) coding in a weighted sense is possible if and only if the per-letter average mutual information between the parameter space and the message space is zero. Universal coding is possible in a maximin sense if and only if the channel capacity between the two spaces is zero. Universal coding is possible in a minimax sense if and only if a probability mass function exists, independent of the unknown parameter, for which the relative entropy of the known conditional-probability mass-function is zero. Several examples are given to illustrate the ideas. Particular attention is given to sources that are stationary and ergodic for any fixed parameter although the whole ensemble is not. For such sources, weighted universal codes always exist if the alphabet is finite, or more generally if the entropy is finite. Minimax universal codes result if an additional entropy stability constraint is applied. A discussion of fixed-rate universal coding is also given briefly with performance measured by a probability of error.

Fixed rate universal block source coding with a fidelity criterion

A unified theory is developed for fixed rate block source encoding subject to a fidelity criterion in incompletely or inaccurately specified stationary statistical environments. Several definitions of universal encoding are given and compared, and the appropriate theorems are stated and proved for each. The new results and approaches are compared and contrasted with earlier related results of Ziv.

A source matching approach to finding minimax codes

A source matching approach is presented to the problem of finding minimax cedes for classes of memoryless sources. The approach leads to a channel capacity problem so that Blahuts algorithm can be used to find approximations to the minimax code. Closed form solutions are presented for the class of monotonic sources and for a class of Bernoulli-like sources. For extensions of finite alphabet memoryless sources, a modified Lynch-Davisson code has performance close to that of the minimax code. The exact solution to the source matching problem and the resulting codes are presented for the extensions of binary codes up to blocklength 31.

Source coding theorems without the ergodic assumption

Source coding theorems are proved for discrete-time stationary processes subject to a fidelity criterion. The alphabet of the process is assumed to be a separable metric space, but the process is not assumed to be ergodic. When the process is not ergodic, the minimum average distortion for a fixed-rate code is not given by the distortion-rate function of the source as usually defined. It is given instead by a weighted average of the distortion-rate functions of ergodic subsources comprising the ergodic decomposition of the source. Potential applications to universal source coding with a fidelity criterion are discussed.

Minimax noiseless universal coding for Markov sources

Upper and lower bounds are presented on the minimax redundancy for Markov source noiseless block-to-variable universal coding. The number of states, block size, and Markov order are arbitrary but finite. Unlike earlier results, the upper and lower bounds are absolute, that is, not merely asymptotic. The upper bound is established by combinatorial bounds. The lower bound is established by a bound on the average redundancy for an arbitrary distribution on the transition probabilities. The bound can be optimized across the choice of distribution.

Adaptive linear estimation for stationary M-dependent processes

In recent years adaptive linear estimation based upon the gradient-following algorithm has been proposed in a wide range of applications. However, little analysis on the convergence of the estimation has appeared when the elements of the data sequence are dependent. This paper presents such an analysis under the assumptions of stationarity and M -dependence (all data sets separated by more than a constant M are statistically independent). It is shown that for a sufficiently small adaptation constant, the mean error in the estimator weights converges to a finite limit, generally nonzero. In addition, hounds on the norm of the mean weight-deviation and on the mean norm-square of the weight-deviation are found and shown to converge to asymptotic bounds, which can be made arbitrarily small by decreasing the adaptation constant and increasing the data block length over which gradient estimates are made.

Efficient universal noiseless source codes

Although the existence of universal noiseless variable-rate codes for the class of discrete stationary ergodic sources has previously been established, very few practical universal encoding methods are available. Efficient implementable universal source coding techniques are discussed in this paper. Results are presented on source codes for which a small value of the maximum redundancy is achieved with a relatively short block length. A constructive proof of the existence of universal noiseless codes for discrete stationary sources is first presented. The proof is shown to provide a method for obtaining efficient universal noiseless variable-rate codes for various classes of sources. For memoryless sources, upper and lower bounds are obtained for the minimax redundancy as a function of the block length of the code. Several techniques for constructing universal noiseless source codes for memoryless sources are presented and their redundancies are compared with the bounds. Consideration is given to possible applications to data compression for certain nonstationary sources.

The ergodic decomposition of stationary discrete random processes

The ergodic decomposition is discussed, and a version focusing on the structure of individual sample functions of stationary processes is proved for the special case of discrete-time random processes with discrete alphabets. The result is stronger in this case than the usual theorem, and the proof is both intuitive and simple. Estimation-theoretic and information-theoretic interpretations are developed and applied to prove existence theorems for universal source codes, both noiseless and with a fidelity criterion.

Fast single-element PN acquisition for the TDRSS MA system

The authors present a description and performance analysis of a new PN code acquisition approach for the tracking and data relay satellite system with multiple access (TDRSS MA) return subsystem. The MA subsystem uses a phased array on-board the TDRS with 30 elements whose received signals are each independently retransmitted to the ground for beam forming. In the current operational system, acquisition is performed by using a serial search of the possible PN epochs on the beam-formed 30-element signal. In contrast, the new approach performs acquisition on a single (arbitrary) element signal using a parallel (i.e., simultaneous) search of all possible PN epochs. Since only one element signal is used, no beam forming is required prior to PN acquisition. The approach is based on sampling and digital processing of the received signal. The new method also incorporates techniques which counteract the effects of Doppler and asynchronous data modulation. The probability of correct acquisition as a function of acquisition time, signal-to-noise ratio, data rate, and user satellite Doppler is analyzed. >

The asymptotic relative efficiency of mixed statistical tests

Mixed statistical tests are described. It is shown that these tests have a much higher efficiency than conventionally used statistics such as the sign test and polarity coincidence correlation without the high operational complexity of the Wilcoxon, Mann-Whitney, Kendall \tau , or Fisher-Yates: Terry-Hoeffding tests.