Evaluating the Evaluators: Which UDA validation methods are most effective? Can they be improved?

This paper compares and ranks 8 UDA validation methods. Validators estimate model accuracy, which makes them an essential component of any UDA train-test pipeline. We rank these validators to indicate which of them are most useful for the purpose of selecting optimal model checkpoints and hyperparameters. To the best of our knowledge, this large-scale benchmark study is the first of its kind in the UDA field. In addition, we propose three new validators that outperform all the existing checkpoint-based validators that we were able to find in the existing literature. Code is available at https://www.github.com/KevinMusgrave/powerful-benchmarker.