SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

2 January 2023

Dan Alistarh

Papers citing "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot"

19 / 119 papers shown

Title
Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models Arnav Chavan Nahush Lele Deepak Gupta 19 1 0 12 Dec 2023
Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs Yu-xin Zhang Lirui Zhao Mingbao Lin Yunyun Sun Yiwu Yao Xingjia Han Jared Tanner Shiwei Liu Rongrong Ji SyDa 37 40 0 13 Oct 2023
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models Huiqiang Jiang Qianhui Wu Chin-Yew Lin Yuqing Yang Lili Qiu 24 100 0 09 Oct 2023
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin You Wu Zhenyu (Allen) Zhang Cheng-Yu Hsieh Yaqing Wang ... Mykola Pechenizkiy Yi Liang Michael Bendersky Zhangyang Wang Shiwei Liu 28 78 0 08 Oct 2023
Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation Chen Dun Mirian Hipolito Garcia Guoqing Zheng Ahmed Hassan Awadallah Anastasios Kyrillidis Robert Sim 81 6 0 04 Oct 2023
Norm Tweaking: High-performance Low-bit Quantization of Large Language Models Liang Li Qingyuan Li Bo-Wen Zhang Xiangxiang Chu MQ 37 29 0 06 Sep 2023
Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA Yuhan Ma Haiqi Jiang Chenyou Fan LRM 26 14 0 09 Aug 2023
Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models Seungcheol Park Ho-Jin Choi U. Kang VLM 25 5 0 07 Aug 2023
A Simple and Effective Pruning Approach for Large Language Models Mingjie Sun Zhuang Liu Anna Bair J. Zico Kolter 56 353 0 20 Jun 2023
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time Zichang Liu Aditya Desai Fangshuo Liao Weitao Wang Victor Xie Zhaozhuo Xu Anastasios Kyrillidis Anshumali Shrivastava 28 201 0 26 May 2023
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers Sotiris Anagnostidis Dario Pavllo Luca Biggio Lorenzo Noci Aurélien Lucchi Thomas Hofmann 34 53 0 25 May 2023
Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models D. Honegger Konstantin Schurholt Damian Borth 20 4 0 26 Apr 2023
Towards Compute-Optimal Transfer Learning Massimo Caccia Alexandre Galashov Arthur Douillard Amal Rannen-Triki Dushyant Rao Michela Paganini Laurent Charlin MarcÁurelio Ranzato Razvan Pascanu 8 3 0 25 Apr 2023
Blockwise Compression of Transformer-based Models without Retraining Gaochen Dong W. Chen 11 3 0 04 Apr 2023
RPTQ: Reorder-based Post-training Quantization for Large Language Models Zhihang Yuan Lin Niu Jia-Wen Liu Wenyu Liu Xinggang Wang Yuzhang Shang Guangyu Sun Qiang Wu Jiaxiang Wu Bingzhe Wu MQ 29 78 0 03 Apr 2023
Broken Neural Scaling Laws Ethan Caballero Kshitij Gupta Irina Rish David M. Krueger 19 74 0 26 Oct 2022
Playing Lottery Tickets with Vision and Language Zhe Gan Yen-Chun Chen Linjie Li Tianlong Chen Yu Cheng Shuohang Wang Jingjing Liu Lijuan Wang Zicheng Liu VLM 103 53 0 23 Apr 2021
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks Itay Hubara Brian Chmiel Moshe Island Ron Banner S. Naor Daniel Soudry 44 110 0 16 Feb 2021
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 141 684 0 31 Jan 2021