Salimeh Yasaei Sekeh

Basic inequalities for weighted entropies

The concept of weighted entropy takes into account values of different outcomes, i.e., makes entropy context-dependent, through the weight function. In this paper, we establish a number of simple inequalities for the weighted entropies (general as well as specific), mirroring similar bounds on standard (Shannon) entropies and related quantities. The required assumptions are written in terms of various expectations of weight functions. Examples are weighted Ky Fan and weighted Hadamard inequalities involving determinants of positive-definite matrices, and weighted Cramér-Rao inequalities involving the weighted Fisher information matrix.

Direct estimation of information divergence using nearest neighbor ratios

We propose a direct estimation method for Rényi and f-divergence measures based on a new graph theoretical interpretation. Suppose that we are given two sample sets X and Y, respectively with N and M samples, where η := M/N is a constant value. Considering the k-nearest neighbor (k-NN) graph of Y in the joint data set (X, Y), we show that the average powered ratio of the number of X points to the number of Y points among all k-NN points is proportional to Rényi divergence of X and Y densities. A similar method can also be used to estimate f-divergence measures. We derive bias and variance rates, and show that for the class of γ-Hölder smooth functions, the estimator achieves the MSE rate of O(N−2γ/(γ+d)). Furthermore, by using a weighted ensemble estimation technique, for density functions with continuous and bounded derivatives of up to the order d, and some extra conditions at the support set boundary, we derive an ensemble estimator that achieves the parametric MSE rate of O(1/N). Our estimator requires no boundary correction, and remarkably, the boundary issues do not show up. Our approach is also more computationally tractable than other competing estimators, which makes them appealing in many practical applications.

Information theoretic structure learning with confidence

Information theoretic measures (e.g. the Kullback Liebler divergence and Shannon mutual information) have been used for exploring possibly nonlinear multivariate dependencies in high dimension. If these dependencies are assumed to follow a Markov factor graph model, this exploration process is called structure discovery. For discrete-valued samples, estimates of the information divergence over the parametric class of multinomial models lead to structure discovery methods whose mean squared error achieves parametric convergence rates as the sample size grows. However, a naive application of this method to continuous nonparametric multivariate models converges much more slowly. In this paper we introduce a new method for nonparametric structure discovery that uses weighted ensemble divergence estimators that achieve parametric convergence rates and obey an asymptotic central limit theorem that facilitates hypothesis testing and other types of statistical validation.

New Stochastic Orders Based on the Inactivity Time

Measure of uncertainty in past lifetime distribution is particularly suitable measure to describe the information in problems related to ageing properties of reliability theory based on distribution of components or systems. This measure has been de ned by Ruiz and Navarro (1996) on using the physical signi cance. In this paper, we introduce a new measure related to the moment orders and de ne new stochastic orders based on that. We also provide some stochastic comparisons with other certain well-known ageing stochastic orders. Finally, a few properties for series systems and mixture with respect to de ned measures are discussed.

A Note on Double Truncated (Interval) Weighted Cumulative Entropies

Measure of the weighted cumulative entropy about the predictability of failure time of a system have been introduced in [3]. Referring properties of doubly truncated (interval) cumulative residual and past entropy, several bounds and properties in terms of the weighted cumulative entropy is proposed.

Bayesian Weighted Information Measures

Following Ebrahimi et al. (J Stat Res Iran 3:113–137, 2006), we study weighted information measure in univariate case. In particular, we address the concept of comparison models based on information measure and, in our case, specially Kullback–Leibler discrimination measure. The main result is presenting the relationship of weighted mutual information measure and weighted entropy. Indeed, the importance of Weibull distribution family in weighted Kullback–Leibler information and Kullback–Leibler information has been carefully examined, which is useful in comparison models. As a notable application of the result, we study normal distributions, which can prove the expected motivation.