Yanjun Han

Maximum Likelihood Estimation of Functionals of Discrete Distributions

The Dirichlet prior is widely used in estimating discrete distributions and functionals of discrete distributions. In terms of Shannon entropy estimation, one approach is to plug-in the Dirichlet prior smoothed distribution into the entropy functional, while the other one is to calculate the Bayes estimator for entropy under the Dirichlet prior for squared error, which is the conditional expectation. We show that in general they do not improve over the maximum likelihood estimator, which plugs-in the empirical distribution into the entropy functional. No matter how we tune the parameters in the Dirichlet prior, this approach cannot achieve the minimax rates in entropy estimation, as recently characterized by Jiao, Venkat, Han, and Weissman [1], and Wu and Yang [2]. The performance of the minimax rate-optimal estimator with n samples is essentially at least as good as that of the Dirichlet smoothed entropy estimators with n ln n samples. We harness the theory of approximation using positive linear operators for analyzing the bias of plug-in estimators for general functionals under arbitrary statistical models, thereby further consolidating the interplay between these two fields, which was thoroughly exploited by Jiao, Venkat, Han, and Weissman [3] in estimating various functionals of discrete distributions. We establish new results in approximation theory, and apply them to analyze the bias of the Dirichlet prior smoothed plug-in entropy estimator. This interplay between bias analysis and approximation theory is of relevance and consequence far beyond the specific problem setting in this paper.

On the Ergodic Capacity of MIMO Free-Space Optical Systems Over Turbulence Channels

Free-space optical (FSO) communications can achieve high capacity with huge unlicensed optical spectrum and low operational costs. The corresponding performance analysis of FSO systems over turbulence channels is very limited, particularly when using multiple apertures at both transmitter and receiver sides. This paper aims to provide the ergodic capacity characterization of multiple-input-multiple-output (MIMO) FSO systems over atmospheric turbulence-induced fading channels. The fluctuations of the irradiance of optical channels distorted by atmospheric conditions is usually described by a gamma-gamma (rr) distribution, and the distribution of the sum of rr random variables (RVs) is required to model the MIMO optical links. We use an α - μ distribution to efficiently approximate the probability density function (pdf) of the sum of independent and identical distributed ΓΓ RVs through moment-based estimators. Furthermore, the pdf of the sum of independent, but not necessarily identically distributed ΓΓ RVs can be efficiently approximated by a finite weighted sum of pdfs of ΓΓ distributions. Based on these reliable approximations, novel and precise analytical expressions for the ergodic capacity of MIMO FSO systems are derived. Additionally, we deduce the asymptotic simple expressions in high signal-to-noise ratio regimes, which provide useful insights into the impact of the system parameters on the ergodic capacity. Finally, our proposed results are validated via Monte Carlo simulations.

Performance Limits and Geometric Properties of Array Localization

Location-aware networks are of great importance and interest in both civil and military applications. This paper determines the localization accuracy of an agent, which is equipped with an antenna array and localizes itself using wireless measurements with anchor nodes, in a far-field environment. In view of the Cramér-Rao bound, we first derive the localization information for static scenarios and demonstrate that such information is a weighed sum of Fisher information matrices from each anchor-antenna measurement pair. Each matrix can be further decomposed into two parts: 1) a distance part with intensity proportional to the squared baseband effective bandwidth of the transmitted signal and 2) a direction part with intensity associated with the normalized anchor-antenna visual angle. Moreover, in dynamic scenarios, we show that the Doppler shift contributes additional direction information, with intensity determined by the agent velocity and the root mean squared time duration of the transmitted signal. In addition, two measures are proposed to evaluate the localization performance of wireless networks with different anchor-agent and array-antenna geometries, and both formulae and simulations are provided for typical anchor deployments and antenna arrays.

Minimax estimation of the L 1 distance

We consider the problem of estimating the L1 distance between two discrete probability measures P and Q from empirical data in a nonasymptotic and large alphabet setting. We construct minimax rate-optimal estimators for L1(P,Q) when Q is either known or unknown, and show that the performance of the optimal estimators with n samples is essentially that of the Maximum Likelihood Estimators (MLE) with n ln n samples. Hence, we demonstrate that the effective sample size enlargement phenomenon, discovered and discussed in Jiao et al. (2015), holds for this problem as well. However, the construction of optimal estimators for L1(P,Q) requires new techniques and insights outside the scope of the Approximation methodology of functional estimation in Jiao et al. (2015).

Minimax Estimation of Discrete Distributions Under

We consider the problem of discrete distribution estimation under l1 loss. We provide tight upper and lower bounds on the maximum risk of the empirical distribution (the maximum likelihood estimator), and the minimax risk in regimes where the support size S may grow with the number of observations n. We show that among distributions with bounded entropy H, the asymptotic maximum risk for the empirical distribution is 2H/ln n, while the asymptotic minimax risk is H/ ln n. Moreover, we show that a hard-thresholding estimator oblivious to the unknown upper bound H, is essentially minimax. However, if we constrain the estimates to lie in the simplex of probability distributions, then the asymptotic minimax risk is again 2H/ ln n. We draw connections between our work and the literature on density estimation, entropy estimation, total variation distance (I1 divergence) estimation, joint distribution estimation in stochastic processes, normal mean estimation, and adaptive estimation.

\ell _{1}

We consider estimating the Shannon entropy of a discrete distribution P from n i.i.d. samples. Recently, Jiao, Venkat, Han, and Weissman (JVHW), and Wu and Yang constructed approximation theoretic estimators that achieve the minimax L2 rates in estimating entropy. Their estimators are consistent given n ≫ S/lnS samples, where S is the support size, and it is the best possible sample complexity. In contrast, the Maximum Likelihood Estimator (MLE), which is the empirical entropy, requires n ≫ S samples. In the present paper we significantly refine the minimax results of existing work. To alleviate the pessimism of minimaxity, we adopt the adaptive estimation framework, and show that the JVHW estimator is an adaptive estimator, i.e., it achieves the minimax rates simultaneously over a nested sequence of subsets of distributions P, without knowing the support size S or which subset P lies in. We also characterize the maximum risk of the MLE over this nested sequence, and show, for every subset in the sequence, that the performance of the minimax rate-optimal estimator with n samples is essentially that of the MLE with n ln n samples, thereby further substantiating the generality of “effective sample size enlargement” phenomenon discovered by Jiao, Venkat, Han, and Weissman. We provide a “pointwise” explanation of the sample size enlargement phenomenon, which states that for sufficiently small probabilities, the bias function of the JVHW estimator with n samples is nearly that of the MLE with n ln n samples.

Loss

The Dirichlet prior is widely used in estimating discrete distributions and functionals of discrete distributions. In terms of Shannon entropy estimation, one approach is to plug-in the Dirichlet prior smoothed distribution into the entropy functional, while the other one is to calculate the Bayes estimator for entropy under the Dirichlet prior for squared error, which is the conditional expectation. We show that in general they do not improve over the maximum likelihood estimator, which plugs-in the empirical distribution into the entropy functional. No matter how we tune the parameters in the Dirichlet prior, this approach cannot achieve the minimax rates in entropy estimation, as recently characterized by Jiao, Venkat, Han, and Weissman [1], and Wu and Yang [2]. The performance of the minimax rate-optimal estimator with n samples is essentially at least as good as that of the Dirichlet smoothed entropy estimators with n ln n samples. We harness the theory of approximation using positive linear operators for analyzing the bias of plug-in estimators for general functionals under arbitrary statistical models, thereby further consolidating the interplay between these two fields, which was thoroughly exploited by Jiao, Venkat, Han, and Weissman [3] in estimating various functionals of discrete distributions. We establish new results in approximation theory, and apply them to analyze the bias of the Dirichlet prior smoothed plug-in entropy estimator. This interplay between bias analysis and approximation theory is of relevance and consequence far beyond the specific problem setting in this paper.

Adaptive estimation of Shannon entropy

We show that in high dimensional distributions, i.e., the regime where the alphabet size of each node is comparable to the number of observations, the Chow-Liu algorithm on learning graphical models is highly sub-optimal. We propose a new approach, where the key ingredient is to replace the empirical mutual information in the Chow-Liu algorithm with a minimax rate-optimal estimator proposed recently by Jiao, Venkat, Han, and Weissman [1]. We demonstrate the improved performance of the new approach in two problems: learning tree graphical models and Bayesian network classification.

Does dirichlet prior smoothing solve the Shannon entropy estimation problem

The Maximum Likelihood Estimator (MLE) is widely used in estimating information measures, and involves “plugging-in” the empirical distribution of the data to estimate a given functional of the unknown distribution. In this work we propose a general framework and procedure to analyze the nonasymptotic performance of the MLE in estimating functionals of discrete distributions, under the worst-case mean squared error criterion. We show that existing theory is insufficient for analyzing the bias of the MLE, and propose to apply the theory of approximation using positive linear operators to study this bias. The variance is controlled using the well-known tools from the literature on concentration inequalities. Our techniques completely characterize the maximum L₂ risk incurred by the MLE in estimating the Shannon entropy H(P) = Σ_i=1^S -p_iln p_i, and F_α(P) = Σ_i=1^Sp_i^α up to a multiplicative constant. As a corollary, for Shannon entropy estimation, we show that it is necessary and sufficient to have n ≪ S observations for the MLE to be consistent, where S represents the support size. In addition, we obtain that it is necessary and sufficient to consider n ≪ S^1/α samples for the MLE to consistently estimate F_α(P); 0 <;α <; 1. The minimax rate-optimal estimators for both problems require S/ln S and S^1/α / ln S samples, which implies that the MLE is strictly sub-optimal. When 1 <; α <; 3/2, we show that the maximum L₂ rate of convergence for the MLE is n^-2(α-1) for infinite support size, while the minimax L₂ rate is (n ln n)^-2(α-1). When α ≥ 3/2, the MLE achieves the minimax optimal L₂ convergence rate n^-1 regardless of the support size.

Beyond maximum likelihood: Boosting the Chow-Liu algorithm for large alphabets

We analyze the problem of discrete distribution estimation under ℓ1 loss. We provide non-asymptotic upper and lower bounds on the maximum risk of the empirical distribution (the maximum likelihood estimator), and the minimax risk in regimes where the alphabet size S may grow with the number of observations n. We show that among distributions with bounded entropy H, the asymptotic maximum risk for the empirical distribution is 2H / ln n, while the asymptotic minimax risk is H / ln n. Moreover, a hard-thresholding estimator, whose threshold does not depend on the unknown upper bound H, is asymptotically minimax. We draw connections between our work and the literature on density estimation, entropy estimation, total variation distance (ℓ1 divergence) estimation, joint distribution estimation in stochastic processes, normal mean estimation, and adaptive estimation.