In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models

20 December 2022

Kathleen McKeown

Papers citing "In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models"

33 / 33 papers shown

Title
Improving In-Context Learning with Reasoning Distillation Nafis Sadeq Xin Xu Zhouhang Xie Julian McAuley Byungkyu Kang Prarit Lamba Xiang Gao RALM ReLM LRM 33 0 0 14 Apr 2025
Enhancing Contrastive Demonstration Selection with Semantic Diversity for Robust In-Context Machine Translation Owen Patterson Chee Ng 29 0 0 12 Apr 2025
When Reasoning Meets Compression: Benchmarking Compressed Large Reasoning Models on Complex Reasoning Tasks Nan Zhang Yusen Zhang Prasenjit Mitra Rui Zhang MQ LRM 51 2 0 02 Apr 2025
LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment Binrui Zeng Bin Ji Xiaodong Liu Jie Yu Shasha Li Jun Ma Xiaopeng Li Shangwen Wang Xinran Hong Yongtao Tang MQ 42 1 0 24 Dec 2024
FASTNav: Fine-tuned Adaptive Small-language-models Trained for Multi-point Robot Navigation Yuxuan Chen Yixin Han Xiao Li 69 1 0 20 Nov 2024
HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference Peng Tang Jiacheng Liu X. Hou Yifei Pu Jing Wang Pheng-Ann Heng C. Li M. Guo MoE 59 7 0 03 Nov 2024
Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression Jingcun Wang Yu-Guang Chen Ing-Chao Lin Bing Li Grace Li Zhang 33 4 0 02 Oct 2024
CogniDual Framework: Self-Training Large Language Models within a Dual-System Theoretical Framework for Improving Cognitive Tasks Yongxin Deng Xihe Qiu Xiaoyu Tan Chao Qu Jing Pan Yuan-Chia Cheng Yinghui Xu Wei Chu 34 2 0 05 Sep 2024
LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation Fangxun Shu Yue Liao Le Zhuo Chenning Xu Guanghao Zhang ... Bolin Li Zhelun Yu Si Liu Hongsheng Li Hao Jiang VLM MoE 32 8 0 28 Aug 2024
Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations Leo Donisch Sigurd Schacht Carsten Lanquillon 22 2 0 06 Aug 2024
In-Context Editing: Learning Knowledge from Self-Induced Distributions Siyuan Qi Bangcheng Yang Kailin Jiang Xiaobo Wang Jiaqi Li Yifan Zhong Yaodong Yang Zilong Zheng KELM 103 8 0 17 Jun 2024
AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers Emil Biju Anirudh Sriram Mert Pilanci 39 0 0 13 Jun 2024
LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions Chuanneng Sun Songjun Huang D. Pompili LLMAG 39 27 0 17 May 2024
PatentGPT: A Large Language Model for Intellectual Property Zilong Bai Ruiji Zhang Linqing Chen Qijun Cai Yuan Zhong ... Fu Bian Xiaolong Gu Lisha Zhang Weilei Wang Changyang Tu 41 3 0 28 Apr 2024
A Survey on Efficient Inference for Large Language Models Zixuan Zhou Xuefei Ning Ke Hong Tianyu Fu Jiaming Xu ... Shengen Yan Guohao Dai Xiao-Ping Zhang Yuhan Dong Yu-Xiang Wang 46 82 0 22 Apr 2024
LLM Inference Unveiled: Survey and Roofline Model Insights Zhihang Yuan Yuzhang Shang Yang Zhou Zhen Dong Zhe Zhou ... Yong Jae Lee Yan Yan Beidi Chen Guangyu Sun Kurt Keutzer 37 79 0 26 Feb 2024
A Survey on Knowledge Distillation of Large Language Models Xiaohan Xu Ming Li Chongyang Tao Tao Shen Reynold Cheng Jinyang Li Can Xu Dacheng Tao Tianyi Zhou KELM VLM 44 100 0 20 Feb 2024
Model Compression and Efficient Inference for Large Language Models: A Survey Wenxiao Wang Wei Chen Yicong Luo Yongliu Long Zhengkai Lin Liye Zhang Binbin Lin Deng Cai Xiaofei He MQ 41 47 0 15 Feb 2024
Accurate LoRA-Finetuning Quantization of LLMs via Information Retention Haotong Qin Xudong Ma Xingyu Zheng Xiaoyang Li Yang Zhang Shouda Liu Jie Luo Xianglong Liu Michele Magno MQ 23 37 0 08 Feb 2024
A Survey on Transformer Compression Yehui Tang Yunhe Wang Jianyuan Guo Zhijun Tu Kai Han Hailin Hu Dacheng Tao 31 27 0 05 Feb 2024
Improving In-context Learning via Bidirectional Alignment Chengwei Qin Wenhan Xia Fangkai Jiao Chen Chen Yuchen Hu Bosheng Ding Shafiq R. Joty 35 7 0 28 Dec 2023
A Performance Evaluation of a Quantized Large Language Model on Various Smartphones Tolga Çöplü Marc Loedi Arto Bendiken Mykhailo Makohin Joshua J. Bouw Stephen Cobb MQ 16 5 0 19 Dec 2023
HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models Cheng Chen Yuchen Hu Chao-Han Huck Yang Sabato Marco Siniscalchi Pin-Yu Chen E. Chng 24 42 0 27 Sep 2023
A Survey on Model Compression for Large Language Models Xunyu Zhu Jian Li Yong Liu Can Ma Weiping Wang 26 191 0 15 Aug 2023
On the Relation between Sensitivity and Accuracy in In-context Learning Yanda Chen Chen Zhao Zhou Yu Kathleen McKeown He He 182 77 0 16 Sep 2022
Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher Mehdi Rezagholizadeh A. Jafari Puneeth Salad Pranav Sharma Ali Saheb Pasand A. Ghodsi 71 17 0 16 Oct 2021
A Short Study on Compressing Decoder-Based Language Models Tianda Li Yassir El Mesbahi I. Kobyzev Ahmad Rashid A. Mahmud Nithin Anchuri Habib Hajimolahoseini Yang Liu Mehdi Rezagholizadeh 86 25 0 16 Oct 2021
Meta-learning via Language Model In-context Tuning Yanda Chen Ruiqi Zhong Sheng Zha George Karypis He He 234 156 0 15 Oct 2021
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP Qinyuan Ye Bill Yuchen Lin Xiang Ren 211 179 0 18 Apr 2021
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity Yao Lu Max Bartolo Alastair Moore Sebastian Riedel Pontus Stenetorp AILaw LRM 279 1,120 0 18 Apr 2021
What Makes Good In-Context Examples for GPT- $3$ ? Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin Weizhu Chen AAML RALM 275 1,312 0 17 Jan 2021
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing Canwen Xu Wangchunshu Zhou Tao Ge Furu Wei Ming Zhou 221 197 0 07 Feb 2020
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 415 2,584 0 03 Sep 2019