LaCo: Large Language Model Pruning via Layer Collapse

17 February 2024

Papers citing "LaCo: Large Language Model Pruning via Layer Collapse"

44 / 44 papers shown

Title
SPAP: Structured Pruning via Alternating Optimization and Penalty Methods Hanyu Hu Xiaoming Yuan 46 0 0 06 May 2025
SD $^2$ : Self-Distilled Sparse Drafters Mike Lasby Nish Sinnadurai Valavan Manohararajah Sean Lie Vithursan Thangarasa 89 0 0 10 Apr 2025
Compression Laws for Large Language Models Ayan Sengupta Siddhant Chaudhary Tanmoy Chakraborty 26 0 0 06 Apr 2025
Entropy-Based Block Pruning for Efficient Large Language Models Liangwei Yang Yuhui Xu Juntao Tan Doyen Sahoo S. Caiming Xiong H. Wang Shelby Heinecke AAML 23 0 0 04 Apr 2025
Adaptive Layer-skipping in Pre-trained LLMs Xuan Luo Weizhi Wang Xifeng Yan 95 0 0 31 Mar 2025
Triad: Empowering LMM-based Anomaly Detection with Vision Expert-guided Visual Tokenizer and Manufacturing Process Yuanze Li Shihao Yuan Haolin Wang Qizhang Li Ming-Yu Liu Chen Xu Guangming Shi Wangmeng Zuo 51 0 0 17 Mar 2025
Position-Aware Depth Decay Decoding ( $D^3$ ): Boosting Large Language Model Inference Efficiency Siqi Fan Xuezhi Fang Xingrun Xing Peng Han Shuo Shang Yequan Wang 51 0 0 11 Mar 2025
IteRABRe: Iterative Recovery-Aided Block Reduction Haryo Akbarianto Wibowo Haiyue Song Hideki Tanaka Masao Utiyama Alham Fikri Aji Raj Dabre 57 0 0 08 Mar 2025
Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models Benyamin Jamialahmadi Parsa Kavehzadeh Mehdi Rezagholizadeh Parsa Farinneya Hossein Rajabzadeh A. Jafari Boxing Chen Marzieh S. Tahaei 42 0 0 06 Mar 2025
PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation Tao Fan Guoqiang Ma Yuanfeng Song Lixin Fan Kai Chen Qiang Yang 46 1 0 21 Feb 2025
LESA: Learnable LLM Layer Scaling-Up Yifei Yang Zouying Cao Xinbei Ma Yao Yao L. Qin Z. Chen Hai Zhao 59 0 0 20 Feb 2025
Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation H. Seo Wongi Jeong Jae-sun Seo Se Young Chun 55 0 0 12 Feb 2025
You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning Ayan Sengupta Siddhant Chaudhary Tanmoy Chakraborty 44 3 0 25 Jan 2025
How Redundant Is the Transformer Stack in Speech Representation Models? Teresa Dorszewski Albert Kjøller Jacobsen Lenka Tětková Lars Kai Hansen 104 0 0 20 Jan 2025
GREEN-CODE: Learning to Optimize Energy Efficiency in LLM-based Code Generation Shashikant Ilager Lukas Florian Briem Ivona Brandić 29 0 0 19 Jan 2025
Merging Feed-Forward Sublayers for Compressed Transformers Neha Verma Kenton W. Murray Kevin Duh AI4CE 45 0 0 10 Jan 2025
Optimizing Small Language Models for In-Vehicle Function-Calling Yahya Sowti Khiabani Farris Atif Chieh Hsu Sven Stahlmann Tobias Michels Sebastian Kramer Benedikt Heidrich M. Saquib Sarfraz Julian Merten Faezeh Tafazzoli 26 1 0 04 Jan 2025
LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment Binrui Zeng Bin Ji Xiaodong Liu Jie Yu Shasha Li Jun Ma Xiaopeng Li Shangwen Wang Xinran Hong Yongtao Tang MQ 36 1 0 24 Dec 2024
GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inference Chao Zeng Songwei Liu Shu Yang Fangmin Chen Xing Mei Lean Fu MQ 38 0 0 23 Dec 2024
Reassessing Layer Pruning in LLMs: New Insights and Methods Yao Lu Hao Cheng Yujie Fang Zeyu Wang Jiaheng Wei Dongwei Xu Qi Xuan Xiaoniu Yang Zhaowei Zhu 61 0 0 23 Nov 2024
P $^2$ Law: Scaling Law for Post-Training After Model Pruning Xiaodong Chen Yuxuan Hu Jing Zhang Xiaokang Zhang C. Li H. Chen Jing Zhang 34 0 0 15 Nov 2024
Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy Razvan-Gabriel Dumitru Paul-Ioan Clotan Vikas Yadav Darius Peteleaza Mihai Surdeanu 22 4 0 05 Nov 2024
Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design Ruisi Cai Yeonju Ro Geon-Woo Kim Peihao Wang Babak Ehteshami Bejnordi Aditya Akella Z. Wang MoE 25 3 0 24 Oct 2024
KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing Yifei Yang Zouying Cao Qiguang Chen L. Qin Dongjie Yang Hai Zhao Zhi Chen 28 5 0 24 Oct 2024
Chip-Tuning: Classify Before Language Models Say Fangwei Zhu Dian Li Jiajun Huang Gang Liu Hui Wang Zhifang Sui 25 0 0 09 Oct 2024
CFSP: An Efficient Structured Pruning Framework for LLMs with Coarse-to-Fine Activation Information Yuxin Wang Minghua Ma Zekun Wang Jingchang Chen Huiming Fan Liping Shan Qing Yang Dongliang Xu Ming Liu Bing Qin 24 3 0 20 Sep 2024
Application Specific Compression of Deep Learning Models Rohit Raj Rai Angana Borah Amit Awekar 14 0 0 09 Sep 2024
MPruner: Optimizing Neural Network Size with CKA-Based Mutual Information Pruning Seungbeom Hu ChanJun Park Andrew Ferraiuolo Sang-Ki Ko Jinwoo Kim Haein Song Jieung Kim 16 1 0 24 Aug 2024
MoDeGPT: Modular Decomposition for Large Language Model Compression Chi-Heng Lin Shangqian Gao James Seale Smith Abhishek Patel Shikhar Tuli Yilin Shen Hongxia Jin Yen-Chang Hsu 71 6 0 19 Aug 2024
Research on Personalized Compression Algorithm for Pre-trained Models Based on Homomorphic Entropy Increase Yicong Li Xing Guo Haohua Du 20 0 0 16 Aug 2024
Compact Language Models via Pruning and Knowledge Distillation Saurav Muralidharan Sharath Turuvekere Sreenivas Raviraj Joshi Marcin Chochowski M. Patwary M. Shoeybi Bryan Catanzaro Jan Kautz Pavlo Molchanov SyDa MQ 27 37 0 19 Jul 2024
Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules Zhuocheng Gong Ang Lv Jian-Yu Guan Junxi Yan Wei Yu Wu Huishuai Zhang Minlie Huang Dongyan Zhao Rui Yan MoE 41 6 0 09 Jul 2024
FoldGPT: Simple and Effective Large Language Model Compression Scheme Songwei Liu Chao Zeng Lianqiang Li Chenqian Yan Lean Fu Xing Mei Fangmin Chen 40 4 0 01 Jul 2024
Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels Razvan-Gabriel Dumitru Vikas Yadav Rishabh Maheshwary Paul-Ioan Clotan Sathwik Tejaswi Madhusudhan Mihai Surdeanu MQ 30 2 0 25 Jun 2024
Efficient Continual Pre-training by Mitigating the Stability Gap Yiduo Guo Jie Fu Huishuai Zhang Dongyan Zhao Yikang Shen 30 12 0 21 Jun 2024
BlockPruner: Fine-grained Pruning for Large Language Models Longguang Zhong Fanqi Wan Ruijun Chen Xiaojun Quan Liangzhi Li 18 7 0 15 Jun 2024
LaCoOT: Layer Collapse through Optimal Transport Victor Quétu Nour Hezbri Enzo Tartaglione 23 0 0 13 Jun 2024
Flextron: Many-in-One Flexible Large Language Model Ruisi Cai Saurav Muralidharan Greg Heinrich Hongxu Yin Zhangyang Wang Jan Kautz Pavlo Molchanov 32 10 0 11 Jun 2024
FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models Yang Zhang Yawei Li Xinpeng Wang Qianli Shen Barbara Plank Bernd Bischl Mina Rezaei Kenji Kawaguchi 47 7 0 28 May 2024
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect Xin Men Mingyu Xu Qingyu Zhang Bingning Wang Hongyu Lin Yaojie Lu Xianpei Han Weipeng Chen 25 103 0 06 Mar 2024
Not All Layers of LLMs Are Necessary During Inference Siqi Fan Xin Jiang Xiang Li Xuying Meng Peng Han Shuo Shang Aixin Sun Yequan Wang Zhongyuan Wang 44 32 0 04 Mar 2024
Head-wise Shareable Attention for Large Language Models Zouying Cao Yifei Yang Hai Zhao 36 3 0 19 Feb 2024
SliceGPT: Compress Large Language Models by Deleting Rows and Columns Saleh Ashkboos Maximilian L. Croci Marcelo Gennari do Nascimento Torsten Hoefler James Hensman VLM 125 145 0 26 Jan 2024
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes Lokesh Nagalapatti Chun-Liang Li Chih-Kuan Yeh Hootan Nakhost Yasuhisa Fujii Alexander Ratner Ranjay Krishna Chen-Yu Lee Tomas Pfister ALM 206 498 0 03 May 2023