Exploring the Effectiveness of Multi-stage Fine-tuning for Cross-encoder Re-rankers

State-of-the-art cross-encoders can be fine-tuned to be highly effective in passage re-ranking. The typical fine-tuning process of cross-encoders as re-rankers requires large amounts of manually labelled data, a contrastive learning objective, and a set of heuristically sampled negatives. An alternative recent approach for fine-tuning instead involves teaching the model to mimic the rankings of a highly effective large language model using a distillation objective. These fine-tuning strategies can be applied either individually, or in sequence. In this work, we systematically investigate the effectiveness of point-wise cross-encoders when fine-tuned independently in a single stage, or sequentially in two stages. Our experiments show that the effectiveness of point-wise cross-encoders fine-tuned using contrastive learning is indeed on par with that of models fine-tuned with multi-stage approaches. Code is available for reproduction atthis https URL.

@article{pezzuti2025_2503.22672,
  title={ Exploring the Effectiveness of Multi-stage Fine-tuning for Cross-encoder Re-rankers },
  author={ Francesca Pezzuti and Sean MacAvaney and Nicola Tonellotto },
  journal={arXiv preprint arXiv:2503.22672},
  year={ 2025 }
}