Unsupervised Data Selection for Data-Centric Semi-Supervised Learning

We study unsupervised data selection for semi-supervised learning (SSL), where a large-scale unlabeled dataset is available and a small subset of data is budgeted for label acquisition. Existing SSL methods focus on learning a model that effectively integrates information from given small labeled data and large unlabeled data, whereas we focus on selecting the right data to annotate for SSL without requiring any label or task information. Intuitively, instances to be labeled shall collectively have maximum diversity and coverage for downstream tasks, and individually have maximum information propagation utility for SSL. We formalize these concepts in a three-step data-centric SSL method that improves FixMatch in stability and accuracy by 8% on CIFAR-10 (0.08% labeled) and 14% on ImageNet-1K (0.2% labeled). It is also a universal framework that works with various SSL methods, delivering consistent performance gains. Our work demonstrates that small computation spent on carefully selecting data for annotation brings big annotation efficiency and model performance gain without changing the learning pipeline. Our completely unsupervised data selection can be easily extended to other weakly supervised learning settings.