Narayana P. Santhanam

Universal compression of memoryless sources over unknown alphabets

It has long been known that the compression redundancy of independent and identically distributed (i.i.d.) strings increases to infinity as the alphabet size grows. It is also apparent that any string can be described by separately conveying its symbols, and its pattern-the order in which the symbols appear. Concentrating on the latter, we show that the patterns of i.i.d. strings over all, including infinite and even unknown, alphabets, can be compressed with diminishing redundancy, both in block and sequentially, and that the compression can be performed in linear time. To establish these results, we show that the number of patterns is the Bell number, that the number of patterns with a given number of symbols is the Stirling number of the second kind, and that the redundancy of patterns can be bounded using results of Hardy and Ramanujan on the number of integer partitions. The results also imply an asymptotically optimal solution for the Good-Turing probability-estimation problem.

Always Good Turing: asymptotically optimal probability estimation

While deciphering the German Enigma code during World War II, I.J. Good and A.M. Turing considered the problem of estimating a probability distribution from a sample of data. They derived a surprising and unintuitive formula that has since been used in a variety of applications and studied by a number of researchers. Borrowing an information-theoretic and machine-learning framework, we define the attenuation of a probability estimator as the largest possible ratio between the per-symbol probability assigned to an arbitrarily-long sequence by any distribution, and the corresponding probability assigned by the estimator. We show that some common estimators have infinite attenuation and that the attenuation of the Good-Turing estimator is low, yet larger than one. We then derive an estimator whose attenuation is one, namely, as the length of any sequence increases, the per-symbol probability assigned by the estimator is at least the highest possible. Interestingly, some of the proofs use celebrated results by Hardy and Ramanujan on the number of partitions of an integer. To better understand the behavior of the estimator, we study the probability it assigns to several simple sequences. We show that some sequences this probability agrees with our intuition, while for others it is rather unexpected.

Limit results on pattern entropy

We determine the entropy rate of patterns of certain random processes including all finite-entropy stationary processes. For independent and identically distributed (i.i.d.) processes, we also bound the speed at which the per-symbol pattern entropy converges to this rate, and show that patterns satisfy an asymptotic equipartition property. To derive some of these results we upper bound the probability that the nth variable in a random process differs from all preceding ones.

Universal compression of unknown alphabets

We consider universal compression of strings where the symbols are drawn independently according to the same unknown distribution over an unknown alphabet. We show that the order of the symbols can be conveyed using essentially as many bits as needed when the distribution is known in advance.

Performance of universal codes over infinite alphabets

It was known that universal compression of strings generated by independent and identically distributed sources over infinite alphabets entails infinite per-symbol redundancy. Alternative compression schemes, which decompose the description of such strings into a description of the symbols appearing in the string, and a description of the arrangement of the symbols form were presented. Two descriptions of the symbol arrangement were considered: shapes and patterns. Roughly speaking, shapes describe the relative magnitude of the symbols while patterns describe only the order in which they appear. The per-symbol worst-case redundancy of compressing shapes is a positive constant less than one, and the per-symbol redundancy of compressing patterns diminishes to zero as the block-length increases were proven. Some results on sequential pattern compression were also mentioned.

Optimal Detector for Multilevel NAND Flash Memory Channels with Intercell Interference

In this paper we derive the optimal detector for multilevel cell (MLC) flash memory channels with intercell interference (ICI). We start with the MLC channel model proposed by Dong et al. and just slightly alter the model to guarantee mathematical tractability of the optimal detectors (maximum likelihood and maximum a-posteriori sequence and symbol detectors). The optimal detector is obtained by computing branch metrics using Fourier transforms of analytically computable characteristic functions (corresponding to likelihood functions). We derive the detectors for both simple one-dimensional (1D) channel models and more realistic page-orientated two-dimensional (2D) channel models. Simulation results show that the hard-output bit error rate (BER) performance matches some previously known detectors, but that the soft-output detector outperforms previously known detectors by 0.35 dB.

Population estimation with performance guarantees

We estimate the population size by sampling uniformly from the population. Given an accuracy to which we need to estimate the population with a pre-specified confidence, we provide a simple stopping rule for the sampling process.

Algorithms for modeling distributions over large alphabets

We consider the problem of modeling a distribution whose alphabet size is large relative to the amount of observed data. It is well known that conventional maximum-likelihood estimates do not perform well in that regime. Instead, we find the distribution maximizing the probability of the datas pattern. We derive an efficient algorithm for approximating this distribution. Simulations show that the computed distribution models the data well and yields general estimators that evaluate various data attributes as well as specific estimators designed especially for these tasks

Spatial correlations for solar PV generation and its tree approximation analysis

Smart grids present interesting challenges as we integrate renewable energy sources such as solar PV cells in residential areas. This paper discusses the spatial modeling for solar irradiation and also deals with the problem of graph representation of the model. For this graph, the Chow-Liu minimum spanning tree algorithm helps us to achieve the optimal tree approximation of the graph by minimizing the Kullback-Leibler divergence which has a more sparse graph representation. We consider normalizing data using the zenith angle and also a standard method by subtracting mean and dividing by deviation (at time interval of day, a year moving average). We compare simulation results for Oahu solar measurement grid (Hawaii) sites and six sites near Denver, Colorado. Simulation results reveal that the KullbackLeibler divergence distance between the graph representation of these sites and their optimal tree approximation is bigger in winter than in summer. Moreover, the position of solar PV cells and their angles have an impact on the connection of the graph during the day and also its optimal tree approximation and the accuracy cost that the tree approximation algorithm pays.

Information-theoretic limits of graphical model selection in high dimensions

The problem of graphical model selection is to correctly estimate the graph structure of a Markov random field given samples from the underlying distribution. We analyze the information-theoretic limitations of this problem under high-dimensional scaling, in which the graph size p and the number of edges k (or the maximum degree d) are allowed to increase to infinity as a function of the sample size n. For pairwise binary Markov random fields, we derive both necessary and sufficient conditions on the scaling of the triplet (n, p, k) (or the triplet (n, p, d)) for asympotically reliable reocovery of the graph structure.