Minimax Estimation of Functionals of Discrete Distributions

We propose a general framework for the construction and analysis of minimax estimators for a wide class of functionals of discrete distributions, where the alphabet size $S$ is unknown and may be scaling with the number of observations $n$. We treat the respective regions where the functional is "nonsmooth" and "smooth" separately. In the "nonsmooth" regime, we apply an unbiased estimator for the best polynomial approximation of the functional whereas, in the "smooth" regime, we apply a bias-corrected version of the Maximum Likelihood Estimator (MLE). We illustrate the merit of this approach by thoroughly analyzing the performance of the resulting schemes for estimating two important information measures: the entropy $H(P) = \sum_{i = 1}^S -p_i \ln p_i$ and $F_\alpha(P) = \sum_{i = 1}^S p_i^\alpha,\alpha>0$. We obtain the minimax $L_2$ rates for estimating these functionals. In particular, we demonstrate that our estimator achieves the optimal sample complexity $n = \Theta(S/\ln S)$ for entropy estimation. We also demonstrate that the sample complexity for estimating $F_\alpha(P),0<\alpha<1$ is $\Theta(S^{1/\alpha}/ \ln S)$, which can be achieved by our estimator but not the MLE. For $1<\alpha<3/2$, we show the minimax $L_2$ rate for estimating $F_\alpha(P)$ is $(n\ln n)^{-2(\alpha-1)}$ regardless of the alphabet size, while the exact $L_2$ rate for the MLE is $n^{-2(\alpha-1)}$. For all the above cases, the behavior of the optimal estimators with $n$ samples is essentially that of the MLE with $n\ln n$ samples. We highlight the practical advantages of our schemes for the estimation of entropy and mutual information. We compare our performance with the popular MLE and with the order-optimal entropy estimator of Valiant and Valiant. As we illustrate with a few experiments, our approach reduces running time and boosts the accuracy.