Temporal Ensembling for Semi-Supervised Learning

In this paper, we present a simple and efficient method for training deep neural networks in a semi-supervised setting where only a small portion of training data is labeled. We introduce temporal ensembling, where we form a consensus prediction of the unknown labels under multiple instances of the network-in-training on different epochs, and most importantly, under different regularization and input augmentation conditions. This ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Using our method, we set new records for two standard semi-supervised learning benchmarks, reducing the classification error rate from 18.63% to 12.89% in CIFAR-10 with 4000 labels and from 18.44% to 6.83% in SVHN with 500 labels.