Learning to Skip for Language Modeling

Learning to Skip for Language Modeling

26 November 2023

Papers citing "Learning to Skip for Language Modeling"

13 / 13 papers shown

Title
Adaptive Layer-skipping in Pre-trained LLMs Xuan Luo Weizhi Wang Xifeng Yan 134 0 0 31 Mar 2025
Position-Aware Depth Decay Decoding ( $D^3$ ): Boosting Large Language Model Inference Efficiency Siqi Fan Xuezhi Fang Xingrun Xing Peng Han Shuo Shang Yequan Wang 58 0 0 11 Mar 2025
UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths Weijia Mao Z. Yang Mike Zheng Shou MoE 71 0 0 10 Feb 2025
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 76 5 0 28 Oct 2024
Dynamic layer selection in decoder-only transformers Theodore Glavas Joud Chataoui Florence Regol Wassim Jabbour Antonios Valkanas Boris N. Oreshkin Mark J. Coates AI4CE 24 0 0 26 Oct 2024
Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules Zhuocheng Gong Ang Lv Jian-Yu Guan Junxi Yan Wei Yu Wu Huishuai Zhang Minlie Huang Dongyan Zhao Rui Yan MoE 50 6 0 09 Jul 2024
Not All Layers of LLMs Are Necessary During Inference Siqi Fan Xin Jiang Xiang Li Xuying Meng Peng Han Shuo Shang Aixin Sun Yequan Wang Zhongyuan Wang 44 32 0 04 Mar 2024
Model Compression and Efficient Inference for Large Language Models: A Survey Wenxiao Wang Wei Chen Yicong Luo Yongliu Long Zhengkai Lin Liye Zhang Binbin Lin Deng Cai Xiaofei He MQ 38 47 0 15 Feb 2024
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems Xupeng Miao Gabriele Oliaro Zhihao Zhang Xinhao Cheng Hongyi Jin Tianqi Chen Zhihao Jia 65 76 0 23 Dec 2023
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism Yanxi Chen Xuchen Pan Yaliang Li Bolin Ding Jingren Zhou LRM 33 31 0 08 Dec 2023
Mixture-of-Experts with Expert Choice Routing Yan-Quan Zhou Tao Lei Han-Chu Liu Nan Du Yanping Huang Vincent Zhao Andrew M. Dai Zhifeng Chen Quoc V. Le James Laudon MoE 160 327 0 18 Feb 2022
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 241 643 0 21 Apr 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,817 0 17 Sep 2019