Sequence-Level Knowledge Distillation

25 June 2016

Papers citing "Sequence-Level Knowledge Distillation"

50 / 173 papers shown

Title
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $α$ - $β$ -Divergence Guanghui Wang Zhiyong Yang Z. Wang Shi Wang Qianqian Xu Q. Huang 39 0 0 07 May 2025
Group Relative Knowledge Distillation: Learning from Teacher's Relational Inductive Bias Chao Li Changhua Zhou Jia Chen 24 0 0 29 Apr 2025
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation Jiabin Fan Guoqing Luo Michael Bowling Lili Mou OffRL 63 0 0 26 Apr 2025
Teach Me How to Denoise: A Universal Framework for Denoising Multi-modal Recommender Systems via Guided Calibration H. Li Hanwen Du Y. Li Junchen Fu Chunxiao Li Ziyi Zhuang Jiakang Li Yongxin Ni AI4TS 24 0 0 19 Apr 2025
From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs Jiliang Ni Jiachen Pu Zhongyi Yang Kun Zhou Hui Wang Xiaoliang Xiao Dakui Wang Xin Li Jingfeng Luo Conggang Hu 32 0 0 18 Apr 2025
FourierNAT: A Fourier-Mixing-Based Non-Autoregressive Transformer for Parallel Sequence Generation Andrew Kiruluta Eric Lundy Andreas Lemos AI4TS 37 0 0 04 Mar 2025
LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning Zhibin Lan Liqiang Niu Fandong Meng Jie Zhou Jinsong Su VLM 67 0 0 04 Mar 2025
Advantage-Guided Distillation for Preference Alignment in Small Language Models Shiping Gao Fanqi Wan Jiajian Guo Xiaojun Quan Qifan Wang ALM 58 0 0 25 Feb 2025
Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillation Hieu Nguyen Zihao He Shoumik Atul Gandre Ujjwal Pasupulety Sharanya Kumari Shivakumar Kristina Lerman HILM 52 1 0 16 Feb 2025
TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models Makoto Shing Kou Misaki Han Bao Sho Yokoi Takuya Akiba VLM 57 1 0 28 Jan 2025
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs Nicolas Boizard Kevin El Haddad C´eline Hudelot Pierre Colombo 68 14 0 28 Jan 2025
InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion Zhaoyi Yan Zhijie Sang Y. Zhang Yuhao Fu Baoyi He Qi Zhou Yining Di Chunlin Ji Shengyu Zhang Fei Wu MoMe LRM 55 1 0 06 Jan 2025
Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison Tsz Kin Lam Marco Gaido Sara Papi L. Bentivogli Barry Haddow 31 0 0 04 Jan 2025
Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models Xiao Cui Mo Zhu Yulei Qin Liang Xie Wengang Zhou H. Li 81 4 0 19 Dec 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 68 5 0 28 Oct 2024
SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models Jahyun Koo Yerin Hwang Yongil Kim Taegwan Kang Hyunkyung Bae Kyomin Jung 46 0 0 25 Oct 2024
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 65 5 0 22 Oct 2024
Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling W. Xu Rujun Han Z. Wang L. Le Dhruv Madeka Lei Li W. Wang Rishabh Agarwal Chen-Yu Lee Tomas Pfister 72 8 0 15 Oct 2024
Efficient Inference for Large Language Model-based Generative Recommendation Xinyu Lin Chaoqun Yang Wenjie Wang Yongqi Li Cunxiao Du Fuli Feng See-Kiong Ng Tat-Seng Chua 65 4 0 07 Oct 2024
HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models Seanie Lee Haebin Seong Dong Bok Lee Minki Kang Xiaoyin Chen Dominik Wagner Yoshua Bengio Juho Lee Sung Ju Hwang 65 2 0 02 Oct 2024
Explanation Bottleneck Models Shinýa Yamaguchi Kosuke Nishida LRM BDL 49 1 0 26 Sep 2024
Beyond Next Token Prediction: Patch-Level Training for Large Language Models Chenze Shao Fandong Meng Jie Zhou 41 1 0 17 Jul 2024
Don't Throw Away Data: Better Sequence Knowledge Distillation Jun Wang Eleftheria Briakou Hamid Dadkhahi Rishabh Agarwal Colin Cherry Trevor Cohn 39 5 0 15 Jul 2024
Direct Preference Knowledge Distillation for Large Language Models Yixing Li Yuxian Gu Li Dong Dequan Wang Yu Cheng Furu Wei 34 6 0 28 Jun 2024
Factual Dialogue Summarization via Learning from Large Language Models Rongxin Zhu Jey Han Lau Jianzhong Qi HILM 46 1 0 20 Jun 2024
BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLM Zhewen Shen Aditya Joshi Ruey-Cheng Chen CLL 39 2 0 17 Jun 2024
Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Models Dominik Wagner Ilja Baumann K. Riedhammer Tobias Bocklet MQ 30 1 0 16 Jun 2024
CTC-based Non-autoregressive Textless Speech-to-Speech Translation Qingkai Fang Zhengrui Ma Yan Zhou Min Zhang Yang Feng 52 0 0 11 Jun 2024
A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation Zhengrui Ma Qingkai Fang Shaolei Zhang Shoutao Guo Yang Feng Min Zhang 53 9 0 11 Jun 2024
$$\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning$ $\textit{Trans-LoRA}$ : towards data-free Transferable Parameter Efficient Finetuning Runqian Wang Soumya Ghosh David D. Cox Diego Antognini Aude Oliva Rogerio Feris Leonid Karlinsky 30 1 0 27 May 2024
Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation Jingxuan Wei Linzhuang Sun Yichong Leng Xu Tan Bihui Yu Ruifeng Guo 43 3 0 23 Apr 2024
Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation Lasal Jayawardena Prasan Yapa BDL 34 1 0 19 Apr 2024
The Unreasonable Ineffectiveness of the Deeper Layers Andrey Gromov Kushal Tirumala Hassan Shapourian Paolo Glorioso Daniel A. Roberts 41 79 0 26 Mar 2024
EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation Yuqiao Wen Behzad Shayegh Chenyang Huang Yanshuai Cao Lili Mou 45 4 0 29 Feb 2024
Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages Yuan Zhang Yile Wang Zijun Liu Shuo Wang Xiaolong Wang Peng Li Maosong Sun Yang Janet Liu LRM 27 9 0 19 Feb 2024
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads Tianle Cai Yuhong Li Zhengyang Geng Hongwu Peng Jason D. Lee De-huai Chen Tri Dao 36 246 0 19 Jan 2024
DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiency Azhar Shaikh Michael Cochez Denis Diachkov Michiel de Rijcke Sahar Yousefi 25 0 0 09 Nov 2023
Non-autoregressive Streaming Transformer for Simultaneous Translation Zhengrui Ma Shaolei Zhang Shoutao Guo Chenze Shao Min Zhang Yang Feng 24 12 0 23 Oct 2023
Tuna: Instruction Tuning using Feedback from Large Language Models Haoran Li Yiran Liu Xingxing Zhang Wei Lu Furu Wei ALM 30 3 0 20 Oct 2023
Teacher-Student Architecture for Knowledge Distillation: A Survey Chengming Hu Xuan Li Danyang Liu Haolun Wu Xi Chen Ju Wang Xue Liu 21 16 0 08 Aug 2023
Learning Evaluation Models from Large Language Models for Sequence Generation Chenglong Wang Hang Zhou Kai-Chun Chang Tongran Liu Chunliang Zhang Quan Du Tong Xiao Yue Zhang Jingbo Zhu ELM 34 3 0 08 Aug 2023
f-Divergence Minimization for Sequence-Level Knowledge Distillation Yuqiao Wen Zichao Li Wenyu Du Lili Mou 30 53 0 27 Jul 2023
RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment Kevin Kaichuang Yang Dan Klein Asli Celikyilmaz Nanyun Peng Yuandong Tian ALM 25 31 0 24 Jul 2023
Intuitive Access to Smartphone Settings Using Relevance Model Trained by Contrastive Learning Joonyoung Kim Kangwook Lee Haebin Shin Hurnjoo Lee Sechun Kang Byunguk Choi Dong Shin Joohyung Lee 18 0 0 15 Jul 2023
Target-Side Augmentation for Document-Level Machine Translation Guangsheng Bao Zhiyang Teng Yue Zhang 21 10 0 08 May 2023
DSD $^2$ : Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free? Victor Quétu Enzo Tartaglione 24 7 0 02 Mar 2023
Towards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation Rehan Ahmad Md. Asif Jalal Muhammad Umar Farooq A. Ollerenshaw Thomas Hain 16 2 0 01 Mar 2023
A Reparameterized Discrete Diffusion Model for Text Generation Lin Zheng Jianbo Yuan Lei Yu Lingpeng Kong DiffM 28 57 0 11 Feb 2023
N-Gram Nearest Neighbor Machine Translation Rui Lv Junliang Guo Rui Wang Xu Tan Qi Liu Tao Qin 23 2 0 30 Jan 2023
How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling? Kazuma Hashimoto Iftekhar Naim K. Raman UQLM 27 2 0 21 Dec 2022