156
72

Estimating Renyi Entropy of Discrete Distributions

Abstract

It was recently shown that estimating the Shannon entropy H(p)H(p) of a discrete kk-symbol distribution pp requires Θ(k/logk)\Theta(k/\log k) samples, a number that grows near-linearly in the support size. In many applications H(p)H(p) can be replaced by the more general Renyi entropy of order α\alpha, Hα(p)H_\alpha(p). We determine the number of samples needed to estimate Hα(p)H_\alpha(p) for all α\alpha, showing that α<1\alpha < 1 requires a super-linear, roughly k1/αk^{1/\alpha} samples, noninteger α>1\alpha>1 requires a near-linear kk samples, but, perhaps surprisingly, integer α>1\alpha>1 requires only Θ(k11/α)\Theta(k^{1-1/\alpha}) samples. Furthermore, developing on a recently established connection between polynomial approximation and estimation of additive functions of the form xf(px)\sum_x f (p_x), we reduce the sample complexity for noninteger values of α\alpha by a factor of logk\log k compared to the empirical estimator. The estimators achieving these bounds are simple and run in time linear in the number of samples. Our lower bound provides an explicit construction of distributions with different Renyi entropies that are hard to distinguish.

View on arXiv
Comments on this paper