Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning

21 February 2024

Papers citing "Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning"

27 / 27 papers shown

Title
FineScope : Precision Pruning for Domain-Specialized Large Language Models Using SAE-Guided Self-Data Cultivation Chaitali Bhattacharyya Yeseong Kim 40 0 0 01 May 2025
A Short Survey on Small Reasoning Models: Training, Inference, Applications and Research Directions Chengyu Wang Taolin Zhang Richang Hong Jun Huang ReLM LRM 32 1 0 12 Apr 2025
SD $^2$ : Self-Distilled Sparse Drafters Mike Lasby Nish Sinnadurai Valavan Manohararajah Sean Lie Vithursan Thangarasa 46 0 0 10 Apr 2025
InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization Yunan Wang Jijie Li Bo Zhang Liangdong Wang Guang Liu 53 0 0 20 Mar 2025
Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Sampling Yiwen Ding Zhiheng Xi Wei He Zhuoyuan Li Yitao Zhai Xiaowei Shi Xunliang Cai Tao Gui Qi Zhang Xuanjing Huang LRM 55 3 0 24 Feb 2025
Escaping Collapse: The Strength of Weak Data for Large Language Model Training Kareem Amin Sara Babakniya Alex Bie Weiwei Kong Umar Syed Sergei Vassilvitskii 61 1 0 13 Feb 2025
OntoTune: Ontology-Driven Self-training for Aligning Large Language Models Zhiqiang Liu Chengtao Gan Junjie Wang Y. Zhang Zhongpu Bo Mengshu Sun H. Chen Wen Zhang 60 0 0 08 Feb 2025
The Best Instruction-Tuning Data are Those That Fit Dylan Zhang Qirun Dai Hao Peng ALM 111 3 0 06 Feb 2025
Aligning Instruction Tuning with Pre-training Yiming Liang Tianyu Zheng Xinrun Du Ge Zhang J. Liu ... Zhaoxiang Zhang Wenhao Huang Jiajun Zhang Xiang Yue Jiajun Zhang 73 1 0 16 Jan 2025
Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation Junhong Wu Yang Zhao Yangyifan Xu Bing Liu Chengqing Zong CLL 20 1 0 17 Oct 2024
Self-Data Distillation for Recovering Quality in Pruned Large Language Models Vithursan Thangarasa Ganesh Venkatesh Mike Lasby Nish Sinnadurai Sean Lie SyDa 28 0 0 13 Oct 2024
Self-Powered LLM Modality Expansion for Large Speech-Text Models Tengfei Yu Xuebo Liu Zhiyi Hou Liang Ding Dacheng Tao Min Zhang 22 0 0 04 Oct 2024
SAG: Style-Aligned Article Generation via Model Collaboration Chenning Xu Fangxun Shu Dian Jin Jinghao Wei Hao Jiang ALM SyDa 20 0 0 04 Oct 2024
Uncertainty-Guided Self-Questioning and Answering for Video-Language Alignment Jin Chen Kaijing Ma Haojian Huang Jiayu Shen Han Fang Xianghao Zang Chao Ban 73 2 0 17 Sep 2024
Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs Jiancheng Dong Lei Jiang Wei Jin Lu Cheng 26 1 0 18 Aug 2024
LLAVADI: What Matters For Multimodal Large Language Models Distillation Shilin Xu Xiangtai Li Haobo Yuan Lu Qi Yunhai Tong Ming-Hsuan Yang 21 0 0 28 Jul 2024
Unified Active Retrieval for Retrieval Augmented Generation Qinyuan Cheng Xiaonan Li Shimin Li Qin Zhu Zhangyue Yin Yunfan Shao Linyang Li Tianxiang Sun Hang Yan Xipeng Qiu 20 0 0 18 Jun 2024
Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression Zilun Zhang Yutao Sun Tiancheng Zhao Leigang Sha Ruochen Xu Kyusong Lee Jianwei Yin CLL KELM 38 0 0 17 Jun 2024
SCAR: Efficient Instruction-Tuning for Large Language Models via Style Consistency-Aware Response Ranking Zhuang Li Yuncheng Hua Thuy-Trang Vu Haolan Zhan Lizhen Qu Gholamreza Haffari 21 2 0 16 Jun 2024
TAIA: Large Language Models are Out-of-Distribution Data Learners Shuyang Jiang Yusheng Liao Ya-Qin Zhang Yu Wang Yanfeng Wang 21 2 0 30 May 2024
A Survey on Self-Evolution of Large Language Models Zhengwei Tao Ting-En Lin Xiancai Chen Hangyu Li Yuchuan Wu Yongbin Li Zhi Jin Fei Huang Dacheng Tao Jingren Zhou LRM LM&Ro 35 21 0 22 Apr 2024
Large Language Models for Data Annotation: A Survey Zhen Tan Dawei Li Song Wang Alimohammad Beigi Bohan Jiang Amrita Bhattacharjee Mansooreh Karami Jundong Li Lu Cheng Huan Liu SyDa 34 44 0 21 Feb 2024
I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses Xuan Ren Biao Wu Lingqiao Liu 23 0 0 17 Feb 2024
Improving Demonstration Diversity by Human-Free Fusing for Text-to-SQL Dingzirui Wang Longxu Dou Xuanliang Zhang Qingfu Zhu Wanxiang Che 6 7 0 16 Feb 2024
KnowGPT: Knowledge Graph based Prompting for Large Language Models Qinggang Zhang Junnan Dong Hao Chen Daochen Zha Zailiang Yu Xiao Huang KELM RALM 22 4 0 11 Dec 2023
Fine-tuned Language Models are Continual Learners Thomas Scialom Tuhin Chakrabarty Smaranda Muresan CLL LRM 132 116 0 24 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022