Expanding Chatbot Knowledge in Customer Service: Context-Aware Similar Question Generation Using Large Language Models

Service chatbots play an important role in enhancing customer support by delivering timely responses to diverse queries. Traditionally, these chatbots rely on retrieval-based methods constrained by a predefined knowledge base of question-answer (QA) pairs to guarantee reliable responses. To effectively handle varied customer inquiries, augmenting the knowledge base with similar questions that maintain semantic consistency and linguistic variability is crucial. This paper presents methodologies for a novel approach that utilizes Large Language Models (LLMs) for generating similar questions and selecting an optimal subset of questions for knowledge base augmentation in industrial chatbots. Specifically, we define the SQG task in the context of LLM training and propose a one-to-many objective that incorporates contextual information. We also introduce an optimization framework that selects a diverse subset of similar questions within predefined resource constraints. Experimental results demonstrate significant improvements over traditional methods, achieving greater semantic diversity while aligning with source QA pairs, with over 120% relative improvement in meeting business-specific requirements with human evaluation. Combined with several best practices, we provide a robust, application-driven solution for enhancing chatbot performance and improving customer service satisfaction.

@article{hong2025_2410.12444,
  title={ Expanding Chatbot Knowledge in Customer Service: Context-Aware Similar Question Generation Using Large Language Models },
  author={ Mengze Hong and Chen Jason Zhang and Di Jiang and Yuanfeng Song and Lu Wang and Yuanqin He and Zhiyang Su and Qing Li },
  journal={arXiv preprint arXiv:2410.12444},
  year={ 2025 }
}