f-Divergence Minimization for Sequence-Level Knowledge Distillation

27 July 2023

Papers citing "f-Divergence Minimization for Sequence-Level Knowledge Distillation"

50 / 52 papers shown

Title
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $α$ - $β$ -Divergence Guanghui Wang Zhiyong Yang Z. Wang Shi Wang Qianqian Xu Q. Huang 28 0 0 07 May 2025
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation Jiabin Fan Guoqing Luo Michael Bowling Lili Mou OffRL 61 0 0 26 Apr 2025
DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models Chengyu Wang Junbing Yan Yuanhao Yue Jun Huang 14 0 0 21 Apr 2025
Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models Junjie Yang Junhao Song Xudong Han Ziqian Bi Tianyang Wang ... Y. Zhang Qian Niu Benji Peng Keyu Chen Ming Liu VLM 40 0 0 18 Apr 2025
A Dual-Space Framework for General Knowledge Distillation of Large Language Models X. Zhang Songming Zhang Yunlong Liang Fandong Meng Yufeng Chen Jinan Xu Jie Zhou 17 0 0 15 Apr 2025
Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs Anshumann Mohd Abbas Zaidi Akhil Kedia Jinwoo Ahn Taehwak Kwon Kangwook Lee Haejun Lee Joohyung Lee FedML 71 0 0 21 Mar 2025
MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modeling Yingyue Li Bencheng Liao Wenyu Liu Xinggang Wang Mamba 58 0 0 17 Mar 2025
Training Domain Draft Models for Speculative Decoding: Best Practices and Insights Fenglu Hong Ravi Raju Jonathan Li Bo Li Urmish Thakker Avinash Ravichandran Swayambhoo Jain Changran Hu 33 0 0 10 Mar 2025
DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs Jongwoo Ko Tianyi Chen Sungnyun Kim Tianyu Ding Luming Liang Ilya Zharkov Se-Young Yun VLM 58 0 0 10 Mar 2025
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation Songming Zhang Xue Zhang Tong Zhang Bojie Hu Yufeng Chen Jinan Xu 42 1 0 04 Mar 2025
Erasing Without Remembering: Safeguarding Knowledge Forgetting in Large Language Models Huazheng Wang Yongcheng Jing Haifeng Sun Yingjie Wang J. Wang Jianxin Liao Dacheng Tao KELM MU 42 0 0 27 Feb 2025
Advantage-Guided Distillation for Preference Alignment in Small Language Models Shiping Gao Fanqi Wan Jiajian Guo Xiaojun Quan Qifan Wang ALM 53 0 0 25 Feb 2025
SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters Teng Xiao Yige Yuan Z. Chen Mingxiao Li Shangsong Liang Z. Ren V. Honavar 93 5 0 21 Feb 2025
Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models Gyeongman Kim Gyouk Chu Eunho Yang MoE 54 0 0 18 Feb 2025
Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation Verna Dankers Vikas Raunak VLM 56 0 0 03 Feb 2025
TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models Makoto Shing Kou Misaki Han Bao Sho Yokoi Takuya Akiba VLM 51 1 0 28 Jan 2025
Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models Xiao Cui Mo Zhu Yulei Qin Liang Xie Wengang Zhou H. Li 81 2 0 19 Dec 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 59 5 0 28 Oct 2024
Beyond Autoregression: Fast LLMs via Self-Distillation Through Time Justin Deschenaux Çağlar Gülçehre 39 2 0 28 Oct 2024
SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models Jahyun Koo Yerin Hwang Yongil Kim Taegwan Kang Hyunkyung Bae Kyomin Jung 21 0 0 25 Oct 2024
A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs A. S. Rawat Veeranjaneyulu Sadhanala Afshin Rostamizadeh Ayan Chakrabarti Wittawat Jitkrittum ... Rakesh Shivanna Sashank J. Reddi A. Menon Rohan Anil Sanjiv Kumar 20 2 0 24 Oct 2024
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 58 5 0 22 Oct 2024
Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQL Qihuang Zhong Kunfeng Chen Liang Ding Juhua Liu Bo Du Dacheng Tao 26 0 0 15 Oct 2024
Efficient Inference for Large Language Model-based Generative Recommendation Xinyu Lin Chaoqun Yang Wenjie Wang Yongqi Li Cunxiao Du Fuli Feng See-Kiong Ng Tat-Seng Chua 58 4 0 07 Oct 2024
Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment Tianyu Peng Jiajun Zhang 21 2 0 19 Sep 2024
Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models Jun Rao Xuebo Liu Zepeng Lin Liang Ding Jing Li Dacheng Tao Min Zhang 30 2 0 19 Sep 2024
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward Dongheng Li Yongchang Hao Lili Mou 30 1 0 19 Sep 2024
LLAVADI: What Matters For Multimodal Large Language Models Distillation Shilin Xu Xiangtai Li Haobo Yuan Lu Qi Yunhai Tong Ming-Hsuan Yang 34 0 0 28 Jul 2024
Multi-Granularity Semantic Revision for Large Language Model Distillation Xiaoyu Liu Yun-feng Zhang Wei Li Simiao Li Xu Huang Hanting Chen Yehui Tang Jie Hu Zhiwei Xiong Yunhe Wang 22 1 0 14 Jul 2024
Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application Chuanpeng Yang Wang Lu Yao Zhu Yidong Wang Qian Chen Chenlong Gao Bingjie Yan Yiqiang Chen ALM KELM 44 20 0 02 Jul 2024
Direct Preference Knowledge Distillation for Large Language Models Yixing Li Yuxian Gu Li Dong Dequan Wang Yu Cheng Furu Wei 18 6 0 28 Jun 2024
Dual-Space Knowledge Distillation for Large Language Models Songming Zhang Xue Zhang Zengkui Sun Yufeng Chen Jinan Xu 35 0 0 25 Jun 2024
PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs Rongzhi Zhang Jiaming Shen Tianqi Liu Haorui Wang Zhen Qin Feng Han Jialu Liu Simon Baumgartner Michael Bendersky Chao Zhang 29 6 0 05 Jun 2024
Efficient Compression of Multitask Multilingual Speech Models Thomas Palmeira Ferraz 33 0 0 02 May 2024
Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models Taiqiang Wu Chaofan Tao Jiahao Wang Zhe Zhao Ngai Wong ALM 35 14 0 03 Apr 2024
Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs Raghavv Goel Mukul Gagrani Wonseok Jeon Junyoung Park Mingu Lee Christopher Lott ALM 16 5 0 29 Feb 2024
EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation Yuqiao Wen Behzad Shayegh Chenyang Huang Yanshuai Cao Lili Mou 37 4 0 29 Feb 2024
Sinkhorn Distance Minimization for Knowledge Distillation Xiao Cui Yulei Qin Yuting Gao Enwei Zhang Zihan Xu Tong Wu Ke Li Xing Sun Wen-gang Zhou Houqiang Li 52 3 0 27 Feb 2024
A Survey on Knowledge Distillation of Large Language Models Xiaohan Xu Ming Li Chongyang Tao Tao Shen Reynold Cheng Jinyang Li Can Xu Dacheng Tao Tianyi Zhou KELM VLM 42 94 0 20 Feb 2024
Revisiting Knowledge Distillation for Autoregressive Language Models Qihuang Zhong Liang Ding Li Shen Juhua Liu Bo Du Dacheng Tao KELM 31 15 0 19 Feb 2024
NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models Shengrui Li Junzhe Chen Xueting Han Jing Bai 9 6 0 15 Feb 2024
DistiLLM: Towards Streamlined Distillation for Large Language Models Jongwoo Ko Sungnyun Kim Tianyi Chen SeYoung Yun 44 25 0 06 Feb 2024
Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning Seonghak Kim Gyeongdo Ham Yucheol Cho Daeshik Kim 9 2 0 23 Nov 2023
Towards the Law of Capacity Gap in Distilling Language Models Chen Zhang Dawei Song Zheyu Ye Yan Gao ELM 10 20 0 13 Nov 2023
Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts Thomas Palmeira Ferraz Marcely Zanon Boito Caroline Brun Vassilina Nikoulina 13 12 0 02 Nov 2023
DistillSpec: Improving Speculative Decoding via Knowledge Distillation Yongchao Zhou Kaifeng Lyu A. S. Rawat A. Menon Afshin Rostamizadeh Sanjiv Kumar Jean-François Kagy Rishabh Agarwal 34 77 0 12 Oct 2023
Ensemble Distillation for Unsupervised Constituency Parsing Behzad Shayegh Yanshuai Cao Xiaodan Zhu Jackie C.K. Cheung Lili Mou 36 5 0 03 Oct 2023
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes Rishabh Agarwal Nino Vieillard Yongchao Zhou Piotr Stańczyk Sabela Ramos Matthieu Geist Olivier Bachem 27 84 0 23 Jun 2023
MiniLLM: Knowledge Distillation of Large Language Models Yuxian Gu Li Dong Furu Wei Minlie Huang ALM 18 75 0 14 Jun 2023
An Equal-Size Hard EM Algorithm for Diverse Dialogue Generation Yuqiao Wen Yongchang Hao Yanshuai Cao Lili Mou 44 10 0 29 Sep 2022