Supervised fine-tuning with synthesized instructions has been a common practice for adapting LLMs to domain-specific QA tasks. However, the synthesized instructions deviate from real user questions and expected answers. This study proposes a novel framework called DeepThink to generate high-quality instructions. DeepThink first generates a few seed questions to mimic actual user questions, simulates conversations to uncover the hidden user needs, and refines the answer by conversational contexts and the retrieved documents for more comprehensive answers. Experiments demonstrate that DeepThink achieves an average performance improvement of 7.92% compared to a GPT-4-turbo+RAG-based assistant on the real user test set in the advertising domain across dimensions such as relevance, completeness, clarity, accuracy, and actionability.
View on arXiv@article{li2025_2502.05497,
title={ DeepThink: Aligning Language Models with Domain-Specific User Intents },
author={ Yang Li and Mingxuan Luo and Yeyun Gong and Chen Lin and Jian Jiao and Yi Liu and Kaili Huang },
journal={arXiv preprint arXiv:2502.05497},
year={ 2025 }
}