Why Ask One When You Can Ask $k$? Two-Stage Learning-to-Defer to a Set of Experts

Learning-to-Defer (L2D) enables decision-making systems to improve reliability by selectively deferring uncertain predictions to more competent agents. However, most existing approaches focus exclusively on single-agent deferral, which is often inadequate in high-stakes scenarios that require collective expertise. We propose Top-$k$ Learning-to-Defer, a generalization of the classical two-stage L2D framework that allocates each query to the $k$ most confident agents instead of a single one. To further enhance flexibility and cost-efficiency, we introduce Top-$k(x)$ Learning-to-Defer, an adaptive extension that learns the optimal number of agents to consult for each query, based on input complexity, agent competency distributions, and consultation costs. For both settings, we derive a novel surrogate loss and prove that it is Bayes-consistent and $(\mathcal{R}, \mathcal{G})$-consistent, ensuring convergence to the Bayes-optimal allocation. Notably, we show that the well-established model cascades paradigm arises as a restricted instance of our Top-$k$ and Top-$k(x)$ formulations. Extensive experiments across diverse benchmarks demonstrate the effectiveness of our framework on both classification and regression tasks.

@article{montreuil2025_2504.12988,
  title={ Why Ask One When You Can Ask $k$? Two-Stage Learning-to-Defer to the Top-$k$ Experts },
  author={ Yannis Montreuil and Axel Carlier and Lai Xing Ng and Wei Tsang Ooi },
  journal={arXiv preprint arXiv:2504.12988},
  year={ 2025 }
}