A Simple and Effective Pruning Approach for Large Language Models

20 June 2023

J. Zico Kolter

Papers citing "A Simple and Effective Pruning Approach for Large Language Models"

50 / 271 papers shown

Title
Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations Dhananjay Saikumar Blesson Varghese 31 0 0 18 Feb 2025
An Efficient Row-Based Sparse Fine-Tuning Cen-Jhih Li Aditya Bhaskara 49 0 0 17 Feb 2025
Forget the Data and Fine-Tuning! Just Fold the Network to Compress Dong Wang Haris Šikić Lothar Thiele O. Saukh 44 0 0 17 Feb 2025
Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation H. Seo Wongi Jeong Jae-sun Seo Se Young Chun 55 0 0 12 Feb 2025
EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models Xingrun Xing Zheng Liu Shitao Xiao Boyan Gao Yiming Liang Wanpeng Zhang Haokun Lin Guoqi Li Jiajun Zhang LRM 56 1 0 10 Feb 2025
Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective Yuan Feng Junlin Lv Y. Cao Xike Xie S.Kevin Zhou 71 2 0 06 Feb 2025
M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inference Nikhil Bhendawade Mahyar Najibi Devang Naik Irina Belousova MoE 85 0 0 04 Feb 2025
Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities Zora Che Stephen Casper Robert Kirk Anirudh Satheesh Stewart Slocum ... Zikui Cai Bilal Chughtai Y. Gal Furong Huang Dylan Hadfield-Menell MU AAML ELM 80 3 0 03 Feb 2025
Progressive Binarization with Semi-Structured Pruning for LLMs X. Yan Tianao Zhang Zhiteng Li Yulun Zhang MQ 54 0 0 03 Feb 2025
Position: AI Scaling: From Up to Down and Out Yunke Wang Yanxi Li Chang Xu HAI 74 1 0 02 Feb 2025
Brain-inspired sparse training enables Transformers and LLMs to perform as fully connected Yingtao Zhang Jialin Zhao Wenjing Wu Ziheng Liao Umberto Michieli C. Cannistraci 46 0 0 31 Jan 2025
Symmetric Pruning of Large Language Models Kai Yi Peter Richtárik AAML VLM 57 0 0 31 Jan 2025
B-FPGM: Lightweight Face Detection via Bayesian-Optimized Soft FPGM Pruning Nikolaos Kaparinos Vasileios Mezaris CVBM 43 0 0 28 Jan 2025
Sparse High Rank Adapters K. Bhardwaj N. Pandey Sweta Priyadarshi Viswanath Ganapathy Rafael Esteves ... P. Whatmough Risheek Garrepalli M. V. Baalen Harris Teague Markus Nagel MQ 33 4 0 28 Jan 2025
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs Mohammad Mozaffari Amir Yazdanbakhsh Zhao Zhang M. Dehnavi 65 5 0 28 Jan 2025
DiscQuant: A Quantization Method for Neural Networks Inspired by Discrepancy Theory Jerry Chee A. Backurs Rainie Heck Li Zhang Janardhan Kulkarni Thomas Rothvoss Sivakanth Gopi MQ 49 0 0 11 Jan 2025
Tailored-LLaMA: Optimizing Few-Shot Learning in Pruned LLaMA Models with Task-Specific Prompts Danyal Aftab Steven Davy ALM 49 0 0 10 Jan 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 80 0 0 08 Jan 2025
CURing Large Models: Compression via CUR Decomposition Sanghyeon Park Soo-Mook Moon 38 0 0 08 Jan 2025
Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic Yifei He Yuzheng Hu Yong Lin Tong Zhang Han Zhao FedML MoMe 62 17 0 08 Jan 2025
Pushing the Envelope of Low-Bit LLM via Dynamic Error Compensation Y. Park Jake Hyun Hojoon Kim Jae W. Lee MQ 36 0 0 31 Dec 2024
SlimGPT: Layer-wise Structured Pruning for Large Language Models Gui Ling Ziyang Wang Yuliang Yan Qingwen Liu 21 2 0 24 Dec 2024
HyperCLIP: Adapting Vision-Language models with Hypernetworks Victor Akinwande Mohammad Sadegh Norouzzadeh Devin Willmott Anna Bair Madan Ravi Ganesh J. Zico Kolter CLIP VLM 84 0 0 21 Dec 2024
Deploying Foundation Model Powered Agent Services: A Survey Wenchao Xu Jinyu Chen Peirong Zheng Xiaoquan Yi Tianyi Tian ... Quan Wan Haozhao Wang Yunfeng Fan Qinliang Su Xuemin Shen AI4CE 112 1 0 18 Dec 2024
PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation Ao Wang Hui Chen Jianchao Tan K. Zhang Xunliang Cai Zijia Lin J. Han Guiguang Ding VLM 77 3 0 04 Dec 2024
Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking Marco Federici Davide Belli M. V. Baalen Amir Jalalirad Andrii Skliar Bence Major Markus Nagel Paul N. Whatmough 76 0 0 02 Dec 2024
Neutralizing Backdoors through Information Conflicts for Large Language Models Chen Chen Yuchen Sun Xueluan Gong Jiaxin Gao K. Lam KELM AAML 67 0 0 27 Nov 2024
Reassessing Layer Pruning in LLMs: New Insights and Methods Yao Lu Hao Cheng Yujie Fang Zeyu Wang Jiaheng Wei Dongwei Xu Qi Xuan Xiaoniu Yang Zhaowei Zhu 61 0 0 23 Nov 2024
freePruner: A Training-free Approach for Large Multimodal Model Acceleration Bingxin Xu Yuzhang Shang Yunhao Ge Qian Lou Yan Yan 94 3 0 23 Nov 2024
Layer Pruning with Consensus: A Triple-Win Solution Leandro Giusti Mugnaini Carolina Tavares Duarte Anna H. Reali Costa Artur Jordao 61 0 0 21 Nov 2024
SAM Decoding: Speculative Decoding via Suffix Automaton Yuxuan Hu Ke Wang Jing Zhang Fanjin Zhang C. Li H. Chen Jing Zhang 42 1 0 16 Nov 2024
AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment Y. Fu Zhongzhi Yu Junwei Li Jiayi Qian Yongan Zhang Xiangchi Yuan Dachuan Shi Roman Yakunin Y. Lin 24 2 0 15 Nov 2024
Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training Elia Cunegatti Leonardo Lucio Custode Giovanni Iacca 36 0 0 11 Nov 2024
DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution Yang Yue Yulin Wang Bingyi Kang Yizeng Han Shenzhi Wang Shiji Song Jiashi Feng Gao Huang VLM 40 16 0 04 Nov 2024
Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference Shashank Nag Alan T. L. Bacellar Zachary Susskind Anshul Jha Logan Liberty ... Krishnan Kailas P. Lima Neeraja J. Yadwadkar F. M. G. França L. John 28 0 0 04 Nov 2024
HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference Peng Tang Jiacheng Liu X. Hou Yifei Pu Jing Wang Pheng-Ann Heng C. Li M. Guo MoE 59 6 0 03 Nov 2024
Fast and Memory-Efficient Video Diffusion Using Streamlined Inference Zheng Zhan Yushu Wu Yifan Gong Zichong Meng Zhenglun Kong Changdi Yang Geng Yuan Pu Zhao Wei Niu Yanzhi Wang VGen 31 4 0 02 Nov 2024
LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment Ge Yang Changyi He J. Guo Jianyu Wu Yifu Ding Aishan Liu Haotong Qin Pengliang Ji Xianglong Liu MQ 31 4 0 28 Oct 2024
EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation Shih-yang Liu Huck Yang Nai Chit Fung Nai Chit Fung Hongxu Yin ... Jan Kautz Yu-Chun Wang Pavlo Molchanov Min-Hung Chen Min-Hung Chen MQ 29 0 0 28 Oct 2024
Beware of Calibration Data for Pruning Large Language Models Yixin Ji Yang Xiang Juntao Li Qingrong Xia Ping Li Xinyu Duan Zhefeng Wang Min Zhang 34 2 0 23 Oct 2024
WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models Jinghan Jia Jiancheng Liu Yihua Zhang Parikshit Ram Nathalie Baracaldo Sijia Liu MU 35 2 0 23 Oct 2024
Small Contributions, Small Networks: Efficient Neural Network Pruning Based on Relative Importance Mostafa Hussien Mahmoud Afifi K. Nguyen M. Cheriet 16 0 0 21 Oct 2024
GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning Guibin Zhang Haonan Dong Yuchen Zhang Zhixun Li Dingshuo Chen Kai Wang Tianlong Chen Yuxuan Liang Dawei Cheng Kun Wang 32 2 0 17 Oct 2024
Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching Jie Peng Zhang Cao Huaizhi Qu Zhengyu Zhang Chang Guo Yanyong Zhang Zhichao Cao Tianlong Chen 24 2 0 17 Oct 2024
LLM-Rank: A Graph Theoretical Approach to Pruning Large Language Models David Hoffmann Kailash Budhathoki Matthaeus Kleindessner 32 0 0 17 Oct 2024
On the Role of Attention Heads in Large Language Model Safety Z. Zhou Haiyang Yu Xinghua Zhang Rongwu Xu Fei Huang Kun Wang Yang Liu Junfeng Fang Yongbin Li 57 5 0 17 Oct 2024
MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router Yanyue Xie Zhi Zhang Ding Zhou Cong Xie Ziang Song Xin Liu Yanzhi Wang Xue Lin An Xu LLMAG 30 3 0 15 Oct 2024
DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models Shangqian Gao Chi-Heng Lin Ting Hua Tang Zheng Yilin Shen Hongxia Jin Yen-Chang Hsu 28 3 0 15 Oct 2024
AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models Haiquan Lu Yefan Zhou Shiwei Liu Zhangyang Wang Michael W. Mahoney Yaoqing Yang 15 0 0 14 Oct 2024
RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates Md. Kowsher Tara Esmaeilbeig Chun-Nam Yu Mojtaba Soltanalian Niloofar Yousefi 27 0 0 14 Oct 2024