Empirically Estimable Classification Bounds Based on a New Divergence Measure

Information divergence functions play a critical role in statistics and information theory. In this paper we introduce a divergence function between distributions and describe a number of properties that make it appealing for classification applications. Based on an extension of a multivariate two-sample test, we identify a nonparametric estimator of the divergence that does not impose strong assumptions on the data distribution. Furthermore, we show that this measure bounds the minimum binary classification error for the case when the training and test data are drawn from the same distribution and for the case where there exists some mismatch between training and test distributions. We confirm the theoretical results by designing feature selection algorithms using the criteria from these bounds and evaluating the algorithms on a series of pathological speech classification tasks.