SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

18 November 2022

Song Han

Papers citing "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models"

50 / 526 papers shown

Title
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models Yuhui Xu Lingxi Xie Xiaotao Gu Xin Chen Heng Chang Hengheng Zhang Zhensu Chen Xiaopeng Zhang Qi Tian MQ 13 88 0 26 Sep 2023
Efficient Post-training Quantization with FP8 Formats Haihao Shen Naveen Mellempudi Xin He Q. Gao Chang‐Bao Wang Mengni Wang MQ 23 19 0 26 Sep 2023
LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression Ayush Kaushal Tejas Vaidhya Irina Rish 52 15 0 25 Sep 2023
Scaling Laws for Sparsely-Connected Foundation Models Elias Frantar C. Riquelme N. Houlsby Dan Alistarh Utku Evci 16 34 0 15 Sep 2023
Large Language Models for Compiler Optimization Chris Cummins Volker Seeker Dejan Grubisic Mostafa Elhoushi Youwei Liang ... Jonas Gehring Fabian Gloeckle Kim M. Hazelwood Gabriel Synnaeve Hugh Leather 18 47 0 11 Sep 2023
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs Wenhua Cheng Weiwei Zhang Haihao Shen Yiyang Cai Xin He Kaokao Lv Yi. Liu MQ 24 21 0 11 Sep 2023
Understanding the Impact of Post-Training Quantization on Large Language Models Somnath Roy MQ 27 3 0 11 Sep 2023
Norm Tweaking: High-performance Low-bit Quantization of Large Language Models Liang Li Qingyuan Li Bo-Wen Zhang Xiangxiang Chu MQ 30 28 0 06 Sep 2023
QuantEase: Optimization-based Quantization for Language Models Kayhan Behdin Ayan Acharya Aman Gupta Qingquan Song Siyu Zhu S. Keerthi Rahul Mazumder MQ 22 19 0 05 Sep 2023
Concepts is All You Need: A More Direct Path to AGI Peter Voss M. Jovanovic 11 3 0 04 Sep 2023
Memory Efficient Optimizers with 4-bit States Bingrui Li Jianfei Chen Jun Zhu MQ 25 33 0 04 Sep 2023
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills Amey Agrawal Ashish Panwar Jayashree Mohan Nipun Kwatra Bhargav S. Gulavani R. Ramjee AI4TS LRM 28 91 0 31 Aug 2023
FPTQ: Fine-grained Post-Training Quantization for Large Language Models Qingyuan Li Yifan Zhang Liang Li Peng Yao Bo-Wen Zhang Xiangxiang Chu Yerui Sun Li-Qiang Du Yuchen Xie MQ 37 11 0 30 Aug 2023
Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models Kaiyuan Gao Su He Zhenyu He Jiacheng Lin Qizhi Pei Jie Shao Wei Zhang LM&MA SyDa 30 4 0 27 Aug 2023
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models Wenqi Shao Mengzhao Chen Zhaoyang Zhang Peng-Tao Xu Lirui Zhao Zhiqiang Li Kaipeng Zhang Peng Gao Yu Qiao Ping Luo MQ 10 174 0 25 Aug 2023
FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs Young Jin Kim Rawn Henry Raffy Fahim Hany Awadalla MQ 15 19 0 16 Aug 2023
A Survey on Model Compression for Large Language Models Xunyu Zhu Jian Li Yong Liu Can Ma Weiping Wang 24 189 0 15 Aug 2023
Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling Marija vSakota Maxime Peyrard Robert West 17 44 0 11 Aug 2023
Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs Or Sharir Anima Anandkumar 22 0 0 27 Jul 2023
QuIP: 2-Bit Quantization of Large Language Models With Guarantees Jerry Chee Yaohui Cai Volodymyr Kuleshov Chris De Sa MQ 20 186 0 25 Jul 2023
ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats Xiaoxia Wu Z. Yao Yuxiong He MQ 27 43 0 19 Jul 2023
A Survey of Techniques for Optimizing Transformer Inference Krishna Teja Chitty-Venkata Sparsh Mittal M. Emani V. Vishwanath Arun Somani 29 62 0 16 Jul 2023
EFL Students' Attitudes and Contradictions in a Machine-in-the-loop Activity System David James Woo Hengky Susanto Kai Guo AI4CE 11 1 0 13 Jul 2023
A Comprehensive Overview of Large Language Models Humza Naveed Asad Ullah Khan Shi Qiu Muhammad Saqib Saeed Anwar Muhammad Usman Naveed Akhtar Nick Barnes Ajmal Saeed Mian OffRL 46 514 0 12 Jul 2023
QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models Tommaso Pegolotti Elias Frantar Dan Alistarh Markus Püschel MQ 9 3 0 07 Jul 2023
INT-FP-QSim: Mixed Precision and Formats For Large Language Models and Vision Transformers Lakshmi Nair Mikhail Bernadskiy Arulselvan Madhavan Craig Chan Ayon Basumallik D. Bunandar MQ 28 2 0 07 Jul 2023
H $_2$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu (Allen) Zhang Ying Sheng Tianyi Zhou Tianlong Chen Lianmin Zheng ... Yuandong Tian Christopher Ré Clark W. Barrett Zhangyang Wang Beidi Chen VLM 47 248 0 24 Jun 2023
Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing Yelysei Bondarenko Markus Nagel Tijmen Blankevoort MQ 8 87 0 22 Jun 2023
Training Transformers with 4-bit Integers Haocheng Xi Changhao Li Jianfei Chen Jun Zhu MQ 25 46 0 21 Jun 2023
A Simple and Effective Pruning Approach for Large Language Models Mingjie Sun Zhuang Liu Anna Bair J. Zico Kolter 50 353 0 20 Jun 2023
SqueezeLLM: Dense-and-Sparse Quantization Sehoon Kim Coleman Hooper A. Gholami Zhen Dong Xiuyu Li Sheng Shen Michael W. Mahoney Kurt Keutzer MQ 24 165 0 13 Jun 2023
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression Tim Dettmers Ruslan Svirschevski Vage Egiazarian Denis Kuznedelev Elias Frantar Saleh Ashkboos Alexander Borzunov Torsten Hoefler Dan Alistarh MQ 14 228 0 05 Jun 2023
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration Ji Lin Jiaming Tang Haotian Tang Shang Yang Wei-Ming Chen Wei-Chen Wang Guangxuan Xiao Xingyu Dang Chuang Gan Song Han EDL MQ 25 463 0 01 Jun 2023
FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization J. H. Lee Jeonghoon Kim S. Kwon Dongsoo Lee MQ 14 32 0 01 Jun 2023
Intriguing Properties of Quantization at Scale Arash Ahmadian Saurabh Dash Hongyu Chen Bharat Venkitesh Stephen Gou Phil Blunsom A. Ustun Sara Hooker MQ 40 38 0 30 May 2023
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models Zechun Liu Barlas Oğuz Changsheng Zhao Ernie Chang Pierre Stock Yashar Mehdad Yangyang Shi Raghuraman Krishnamoorthi Vikas Chandra MQ 42 187 0 29 May 2023
CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers Dachuan Shi Chaofan Tao Anyi Rao Zhendong Yang Chun Yuan Jiaqi Wang VLM 23 22 0 27 May 2023
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time Zichang Liu Aditya Desai Fangshuo Liao Weitao Wang Victor Xie Zhaozhuo Xu Anastasios Kyrillidis Anshumali Shrivastava 23 201 0 26 May 2023
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers Sotiris Anagnostidis Dario Pavllo Luca Biggio Lorenzo Noci Aurélien Lucchi Thomas Hofmann 32 53 0 25 May 2023
Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model Zirui Liu Guanchu Wang Shaochen Zhong Zhaozhuo Xu Daochen Zha ... Zhimeng Jiang Kaixiong Zhou V. Chaudhary Shuai Xu Xia Hu 30 11 0 24 May 2023
QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers Artidoro Pagnoni Ari Holtzman Luke Zettlemoyer ALM 33 2,327 0 23 May 2023
Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization Jeonghoon Kim J. H. Lee Sungdong Kim Joonsuk Park Kang Min Yoo S. Kwon Dongsoo Lee MQ 36 97 0 23 May 2023
Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models Yijia Zhang Lingran Zhao Shijie Cao Wenqiang Wang Ting Cao Fan Yang Mao Yang Shanghang Zhang Ningyi Xu MQ 12 17 0 21 May 2023
Standalone 16-bit Neural Network Training: Missing Study for Hardware-Limited Deep Learning Practitioners Juyoung Yun Byungkon Kang Francois Rameau Zhoulai Fu Zhoulai Fu MQ 13 1 0 18 May 2023
Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt Zhaozhuo Xu Zirui Liu Beidi Chen Yuxin Tang Jue Wang Kaixiong Zhou Xia Hu Anshumali Shrivastava MQ 24 29 0 17 May 2023
FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention Guangxuan Xiao Tianwei Yin William T. Freeman F. Durand Song Han VGen DiffM 34 238 0 17 May 2023
SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification Xupeng Miao Gabriele Oliaro Zhihao Zhang Xinhao Cheng Zeyu Wang ... Chunan Shi Zhuoming Chen Daiyaan Arfeen Reyna Abhyankar Zhihao Jia LRM 40 28 0 16 May 2023
Fast Distributed Inference Serving for Large Language Models Bingyang Wu Yinmin Zhong Zili Zhang Gang Huang Xuanzhe Liu Xin Jin 22 91 0 10 May 2023
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance Lingjiao Chen Matei A. Zaharia James Y. Zou LLMAG 16 205 0 09 May 2023
CrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptation J. Heo S. Azizi A. Fayyazi Massoud Pedram 23 0 0 08 May 2023