Efficient multivariate entropy estimation via $k$-nearest neighbour distances

Many statistical procedures, including goodness-of-fit tests and methods for independent component analysis, rely critically on the estimation of the entropy of a distribution. In this paper we study a generalisation of the entropy estimator originally proposed by \citet{Kozachenko:87}, based on the $k$-nearest neighbour distances of a sample of $n$ independent and identically distributed random vectors in $\mathbb{R}^d$. When $d \geq 3$, and under regularity conditions, we derive the leading term in the asymptotic expansion of the bias of the estimator, while when $d \leq 2$ we provide bounds on the bias (which is typically negligible in these instances). We also prove that, when $d \leq 3$ and provided $k/\log^5 n \rightarrow \infty$, the estimator is efficient, in that it achieves the local asymptotic minimax lower bound; on the other hand, when $d\geq 4$, a non-trivial bias precludes its efficiency regardless of the choice of $k$. In addition to the theoretical understanding provided, our results also have several methodological implications; in particular, they motivate the prewhitening of the data before applying the estimator, facilitate the construction of asymptotically valid confidence intervals of asymptotically minimal width, and suggest methods for bias reduction to obtain root-$n$ consistency in higher dimensions.