Assessing binary classifiers using only positive and unlabeled data

Assessing the performance of a learned model is a crucial part of machine learning. Most evaluation metrics can only be computed with labeled data. Unfortunately, in many domains we have many more unlabeled than labeled examples. Furthermore, in some domains only positive and unlabeled examples are available, in which case most standard metrics cannot be computed at all. In this paper, we propose an approach that is able to estimate several widely used metrics including ROC and PR curves using only positive and unlabeled data. We provide theoretical bounds on the quality of our estimates. Empirically, we demonstrate that even given only a small number of positive examples and unlabeled data, we are able to reliable estimate both ROC and PR curves.