Bipartite ranking is a fundamental supervised learning problem, with the goal of learning a ranking over instances with maximal area under the ROC curve (AUC) against a single binary target label. However, one may often observe multiple binary target labels, e.g., from distinct human annotators. How can one synthesize such labels into a single coherent ranking? In this work, we formally analyze two approaches to this problem -- loss aggregation and label aggregation -- by characterizing their Bayes-optimal solutions. Based on this, we show that while both methods can yield Pareto-optimal solutions, loss aggregation can exhibit label dictatorship: one can inadvertently (and undesirably) favor one label over others. This suggests that label aggregation can be preferable to loss aggregation, which we empirically verify.
View on arXiv@article{lukasik2025_2504.11284,
title={ Bipartite Ranking From Multiple Labels: On Loss Versus Label Aggregation },
author={ Michal Lukasik and Lin Chen and Harikrishna Narasimhan and Aditya Krishna Menon and Wittawat Jitkrittum and Felix X. Yu and Sashank J. Reddi and Gang Fu and Mohammadhossein Bateni and Sanjiv Kumar },
journal={arXiv preprint arXiv:2504.11284},
year={ 2025 }
}