Detecting Label Errors using Pre-Trained Language Models

For identifying label errors in natural language datasets, we show that large pre-trained language models are extremely capable: simply hand-verifying data points in descending order of out-of-distribution model loss significantly outperforms more complex mechanisms proposed in previous work. We also contribute a novel method for producing human-originated label noise using existing crowdsourced datasets, apply it to SNLI and TweetNLP, and show that the resulting errors have similar characteristics to real label errors. We present evidence that pre-training provides limited robustness to this more realistic form of label noise. Finally, we use crowdsourced verification to evaluate performance on IMDB, Amazon Reviews, and Recon, and find that pre-trained models detect errors with 9-36% higher absolute Area Under the Precision-Recall Curve compared to existing models.