Optimal stopping rules for active learning

Active learning algorithms aim to minimise the amount of labelled data used to learn a target concept. However, there is no formal framework for expressing the trade-off between needed accuracy and the cost of label acquisition, rendering the objective evaluation of algorithms problematic and the development of criteria for deciding when to terminate data acquisition impossible. This paper aims to increase awareness of these problems and to introduce a formal notion of optimality for active learning, thus leading to the development of stopping algorithms and finally to a procedure for assessing the real world performance of active learning algorithms.