A Survey on Knowledge Distillation of Large Language Models

20 February 2024

Papers citing "A Survey on Knowledge Distillation of Large Language Models"

35 / 35 papers shown

Title
Optimization Problem Solving Can Transition to Evolutionary Agentic Workflows Wenhao Li Bo Jin Mingyi Hong Changhong Lu Xiangfeng Wang 36 0 0 07 May 2025
Knowledge Distillation of Domain-adapted LLMs for Question-Answering in Telecom Rishika Sen Sujoy Roychowdhury Sumit Soman H. G. Ranjani Srikhetra Mohanty 48 0 0 28 Apr 2025
Towards Faster and More Compact Foundation Models for Molecular Property Prediction Yasir Ghunaim Andrés Villa Gergo Ignacz Gyorgy Szekely Motasem Alfarra Bernard Ghanem AI4CE 78 0 0 28 Apr 2025
Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge Wenhan Mu Ling Xu Shuren Pei Le Mi Huichi Zhou AAML ELM 45 0 0 28 Apr 2025
Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill? Chenrui Fan Ming Li Lichao Sun Tianyi Zhou LRM 41 2 0 09 Apr 2025
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 48 5 0 22 Oct 2024
Future-Guided Learning: A Predictive Approach To Enhance Time-Series Forecasting Skye Gunasekaran Assel Kembay Hugo J. Ladret Rui-Jie Zhu Laurent Udo Perrinet Omid Kavehei Jason Eshraghian AI4TS 24 0 0 19 Oct 2024
Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling W. Xu Rujun Han Z. Wang L. Le Dhruv Madeka Lei Li W. Wang Rishabh Agarwal Chen-Yu Lee Tomas Pfister 66 8 0 15 Oct 2024
LUK: Empowering Log Understanding with Expert Knowledge from Large Language Models Lipeng Ma Weidong Yang Sihang Jiang Ben Fei Mingjie Zhou Shuhao Li Bo Xu Bo Xu Yanghua Xiao 38 0 0 03 Sep 2024
Mobile Edge Intelligence for Large Language Models: A Contemporary Survey Guanqiao Qu Qiyuan Chen Wei Wei Zheng Lin Xianhao Chen Kaibin Huang 27 37 0 09 Jul 2024
Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statements Ming Li Jiuhai Chen Lichang Chen Tianyi Zhou 63 17 0 16 Feb 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 215 291 0 18 Jan 2024
Knowledge Distillation for Closed-Source Language Models Hongzhan Chen Xiaojun Quan Hehong Chen Ming Yan Ji Zhang BDL 20 2 0 13 Jan 2024
Silkie: Preference Distillation for Large Visual Language Models Lei Li Zhihui Xie Mukai Li Shunian Chen Peiyi Wang Liang Chen Yazheng Yang Benyou Wang Lingpeng Kong MLLM 96 67 0 17 Dec 2023
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models Avi Singh John D. Co-Reyes Rishabh Agarwal Ankesh Anand Piyush Patil ... Yamini Bansal Ethan Dyer Behnam Neyshabur Jascha Narain Sohl-Dickstein Noah Fiedel ALM LRM ReLM SyDa 144 143 0 11 Dec 2023
CITB: A Benchmark for Continual Instruction Tuning Zihan Zhang Meng Fang Ling-Hao Chen Mohammad-Reza Namazi-Rad ALM CLL 28 20 0 23 Oct 2023
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection Akari Asai Zeqiu Wu Yizhong Wang Avirup Sil Hannaneh Hajishirzi RALM 138 600 0 17 Oct 2023
An Empirical Study of Instruction-tuning Large Language Models in Chinese Q. Si Tong Wang Zheng Lin Xu Zhang Yanan Cao Weiping Wang ALM 45 16 0 11 Oct 2023
FireAct: Toward Language Agent Fine-tuning Baian Chen Chang Shu Ehsan Shareghi Nigel Collier Karthik Narasimhan Shunyu Yao ALM LLMAG 96 96 0 09 Oct 2023
OWL: A Large Language Model for IT Operations Hongcheng Guo Jian Yang Jiaheng Liu Liqun Yang Linzheng Chai ... Tieqiao Zheng Liangfan Zheng Bo-Wen Zhang Ke Xu Zhoujun Li VLM 52 40 0 17 Sep 2023
Aligning Large Language Models through Synthetic Feedback Sungdong Kim Sanghwan Bae Jamin Shin Soyoung Kang Donghyun Kwak Kang Min Yoo Minjoon Seo ALM SyDa 73 46 0 23 May 2023
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes Lokesh Nagalapatti Chun-Liang Li Chih-Kuan Yeh Hootan Nakhost Yasuhisa Fujii Alexander Ratner Ranjay Krishna Chen-Yu Lee Tomas Pfister ALM 198 283 0 03 May 2023
Zero-Shot Listwise Document Reranking with a Large Language Model Xueguang Ma Xinyu Crystina Zhang Ronak Pradeep Jimmy J. Lin 65 48 0 03 May 2023
PMC-LLaMA: Towards Building Open-source Language Models for Medicine Chaoyi Wu Weixiong Lin Xiaoman Zhang Ya-Qin Zhang Yanfeng Wang Weidi Xie LM&MA AI4MH 78 74 0 27 Apr 2023
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions Minghao Wu Abdul Waheed Chiyu Zhang Muhammad Abdul-Mageed Alham Fikri Aji ALM 115 115 0 27 Apr 2023
Large Language Models are Strong Zero-Shot Retriever Tao Shen Guodong Long Xiubo Geng Chongyang Tao Tianyi Zhou Daxin Jiang RALM KELM 32 33 0 27 Apr 2023
Instruction Tuning with GPT-4 Baolin Peng Chunyuan Li Pengcheng He Michel Galley Jianfeng Gao SyDa ALM LM&MA 154 576 0 06 Apr 2023
ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge Yunxiang Li Zihan Li Kai Zhang Ruilong Dan Steven Jiang You Zhang LM&MA AI4MH 114 366 0 24 Mar 2023
Improving alignment of dialogue agents via targeted human judgements Amelia Glaese Nat McAleese Maja Trkebacz John Aslanides Vlad Firoiu ... John F. J. Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks G. Irving ALM AAML 220 495 0 28 Sep 2022
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models Ishika Singh Valts Blukis Arsalan Mousavian Ankit Goyal Danfei Xu Jonathan Tremblay D. Fox Jesse Thomason Animesh Garg LM&Ro LLMAG 104 616 0 22 Sep 2022
Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning Jiahui Gao Renjie Pi Yong Lin Hang Xu Jiacheng Ye Zhiyong Wu Weizhong Zhang Xiaodan Liang Zhenguo Li Lingpeng Kong SyDa VLM 47 45 0 25 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022
Towards Zero-Label Language Learning Zirui Wang Adams Wei Yu Orhan Firat Yuan Cao SyDa 167 101 0 19 Sep 2021
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 273 1,561 0 18 Sep 2019