A Bernstein-Von Mises Theorem for discrete probability distributions

We investigate the asymptotic normality of the the posterior distribution in the discrete case, when model dimension increases with sample size. We consider a probability mass function $\theta_0$ on $\mathbbm {N}\setminus \{0\}$ and a sequence of trunction levels $(k_n)_n$ satisfying $k_n^3\leq n\inf_{i\leq k_n}\theta_0(i).$ Then, under some mild conditions on $\theta_0$ and on the sequence of prior probabilities on the $k_n$-dimensional simplices. Let $\hat{\theta}$ denote the maximum likelihood estimate of $(\theta_0(i))_{i\leq k_n}$ and $\Delta_n(\theta_0)=\sqrt{n}(\hat{\theta_n}-\theta_0)$. We check that after centering and rescaling, the variation distance between the posterior distribution and the Gaussian distribution $\mathcal{N}(\Delta_n(\theta_0),I^{-1}(\theta_0))$ converges in probability to $0.$ This theorem can be used to prove the asymptotic normality of some Bayesian estimators of the Shannon and R\'{e}nyi entropies. The proofs are based on concentration inequalities for centered and non-centered Chi-square (Pearson) statistics. The latter allow to establish posterior concentration rates with respect to Fisher distance rather than with respect to the Hellinger distance as it is commonplace in non-parametric Bayesian statistics.