Understanding INT4 Quantization for Transformer Models: Latency Speedup,
Composability, and Failure Cases

Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases

27 January 2023

Reza Yazdani Aminabadi

Yuxiong He

Papers citing "Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases"

17 / 17 papers shown

Title
HOT: Hadamard-based Optimized Training Seonggon Kim Juncheol Shin Seung-taek Woo Eunhyeok Park 43 0 0 27 Mar 2025
MergeQuant: Accurate 4-bit Static Quantization of Large Language Models by Channel-wise Calibration Jinguang Wang J. Wang Haifeng Sun Tingting Yang Zirui Zhuang Wanyi Ning Yuexi Yin Q. Qi Jianxin Liao MQ MoMe 44 0 0 07 Mar 2025
Optimizing Large Language Model Training Using FP4 Quantization Ruizhe Wang Yeyun Gong Xiao Liu Guoshuai Zhao Ziyue Yang Baining Guo Zhengjun Zha Peng Cheng MQ 67 4 0 28 Jan 2025
iServe: An Intent-based Serving System for LLMs Dimitrios Liakopoulos Tianrui Hu Prasoon Sinha N. Yadwadkar VLM 71 0 0 08 Jan 2025
BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching Peizhuang Cong Qizhi Chen Haochen Zhao Tong Yang KELM 21 0 0 24 Oct 2024
Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs Tianyu Guo Druv Pai Yu Bai Jiantao Jiao Michael I. Jordan Song Mei 29 9 0 17 Oct 2024
AERO: Softmax-Only LLMs for Efficient Private Inference N. Jha Brandon Reagen 25 1 0 16 Oct 2024
OutlierTune: Efficient Channel-Wise Quantization for Large Language Models Jinguang Wang Yuexi Yin Haifeng Sun Qi Qi Jingyu Wang Zirui Zhuang Tingting Yang Jianxin Liao 30 2 0 27 Jun 2024
How to Parameterize Asymmetric Quantization Ranges for Quantization-Aware Training Jaeseong You Minseop Park Kyunggeun Lee Seokjun An Chirag I. Patel Markus Nagel MQ 31 1 0 25 Apr 2024
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization Aniruddha Nrusimha Mayank Mishra Naigang Wang Dan Alistarh Rameswar Panda Yoon Kim MQ 54 8 0 04 Apr 2024
Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models Qiong Wu Weihao Ye Yiyi Zhou Xiaoshuai Sun Rongrong Ji MoE 25 1 0 22 Mar 2024
A2Q+: Improving Accumulator-Aware Weight Quantization Ian Colbert Alessandro Pappalardo Jakoba Petri-Koenig Yaman Umuroglu MQ 21 4 0 19 Jan 2024
Token-Scaled Logit Distillation for Ternary Weight Generative Language Models Minsoo Kim Sihwa Lee Jangwhan Lee S. Hong Duhyeuk Chang Wonyong Sung Jungwook Choi MQ 16 14 0 13 Aug 2023
I-BERT: Integer-only BERT Quantization Sehoon Kim A. Gholami Z. Yao Michael W. Mahoney Kurt Keutzer MQ 86 332 0 05 Jan 2021
BinaryBERT: Pushing the Limit of BERT Quantization Haoli Bai Wei Zhang Lu Hou Lifeng Shang Jing Jin Xin Jiang Qun Liu Michael Lyu Irwin King MQ 138 221 0 31 Dec 2020
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT Sheng Shen Zhen Dong Jiayu Ye Linjian Ma Z. Yao A. Gholami Michael W. Mahoney Kurt Keutzer MQ 217 571 0 12 Sep 2019
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 170 3,504 0 10 Jun 2015