Tuning Language Models for Robust Prediction of Diverse User Behaviors

Predicting user behavior is essential for intelligent assistant services, yet deep learning models often struggle to capture long-tailed behaviors. Large language models (LLMs), with their pretraining on vast corpora containing rich behavioral knowledge, offer promise. However, existing fine-tuning approaches tend to overfit to frequent ``anchor'' behaviors, reducing their ability to predict less common ``tail'' behaviors. In this paper, we introduce BehaviorLM, a progressive fine-tuning approach that addresses this issue. In the first stage, LLMs are fine-tuned on anchor behaviors while preserving general behavioral knowledge. In the second stage, fine-tuning uses a balanced subset of all behaviors based on sample difficulty to improve tail behavior predictions without sacrificing anchor performance. Experimental results on two real-world datasets demonstrate that BehaviorLM robustly predicts both anchor and tail behaviors and effectively leverages LLM behavioral knowledge to master tail behavior prediction with few-shot examples.

@article{meng2025_2505.17682,
  title={ Tuning Language Models for Robust Prediction of Diverse User Behaviors },
  author={ Fanjin Meng and Jingtao Ding and Jiahui Gong and Chen Yang and Hong Chen and Zuojian Wang and Haisheng Lu and Yong Li },
  journal={arXiv preprint arXiv:2505.17682},
  year={ 2025 }
}