In various situations one is given the predictions of multiple classifiers over a large unlabeled test data. This scenario raises the following questions: Without any labeled data and without any a-priori knowledge about the reliability of these different classifiers, is it possible to consistently and computationally efficiently estimate their accuracy? Furthermore, also without any labeled data can one construct a more accurate unsupervised ensemble classifier? In this paper we make the following contributions: Under the standard assumption that classifiers make independent errors, we provide (partial)\ positive answers to these questions. In the binary case, we present two different methods to estimate the class imbalance and classifiers specificities and sensitivities. This directly gives a novel unsupervised ensemble learner. In the multi-class case, we show how to estimate the class probabilities and the diagonal entries of the classifiers confusion matrices. We illustrate our algorithms with empirical results on both artificial and real data.
View on arXiv