Large language models (LLMs) are increasingly applied to outpatient referral tasks across healthcare systems. However, there is a lack of standardized evaluation criteria to assess their effectiveness, particularly in dynamic, interactive scenarios. In this study, we systematically examine the capabilities and limitations of LLMs in managing tasks within Intelligent Outpatient Referral (IOR) systems and propose a comprehensive evaluation framework specifically designed for such systems. This framework comprises two core tasks: static evaluation, which focuses on evaluating the ability of predefined outpatient referrals, and dynamic evaluation, which evaluates capabilities of refining outpatient referral recommendations through iterative dialogues. Our findings suggest that LLMs offer limited advantages over BERT-like models, but show promise in asking effective questions during interactive dialogues.
View on arXiv@article{liu2025_2503.08292,
title={ Large Language Models for Outpatient Referral: Problem Definition, Benchmarking and Challenges },
author={ Xiaoxiao Liu and Qingying Xiao and Junying Chen and Xiangyi Feng and Xiangbo Wu and Bairui Zhang and Xiang Wan and Jian Chang and Guangjun Yu and Yan Hu and Benyou Wang },
journal={arXiv preprint arXiv:2503.08292},
year={ 2025 }
}