Massive Activations in Large Language Models

27 February 2024

J. Zico Kolter

Papers citing "Massive Activations in Large Language Models"

28 / 28 papers shown

Title
MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design Haojie Duanmu Xiuhong Li Zhihang Yuan Size Zheng Jiangfei Duan Xingcheng Zhang Dahua Lin MQ MoE 55 0 0 09 May 2025
Fast and Low-Cost Genomic Foundation Models via Outlier Removal Haozheng Luo Chenghao Qiu Maojiang Su Zhihan Zhou Zoe Mehta Guo Ye Jerry Yao-Chieh Hu Han Liu AAML 53 0 0 01 May 2025
ICQuant: Index Coding enables Low-bit LLM Quantization Xinlin Li Osama A. Hanna Christina Fragouli Suhas Diggavi MQ 50 0 0 01 May 2025
Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based Language Models Lucas Maisonnave Cyril Moineau Olivier Bichler Fabrice Rastello MQ 66 1 0 30 Apr 2025
Outlier dimensions favor frequent tokens in language models Iuri Macocco Nora Graichen Gemma Boleda Marco Baroni 42 0 0 27 Mar 2025
Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing Qi Le Enmao Diao Ziyan Wang Xinran Wang Jie Ding Li Yang Ali Anwar 67 1 0 24 Feb 2025
Zeroth-Order Adaptive Neuron Alignment Based Pruning without Re-Training Elia Cunegatti Leonardo Lucio Custode Giovanni Iacca 36 0 0 11 Nov 2024
HSR-Enhanced Sparse Attention Acceleration Bo Chen Yingyu Liang Zhizhou Sha Zhenmei Shi Zhao-quan Song 73 17 0 14 Oct 2024
FlatQuant: Flatness Matters for LLM Quantization Yuxuan Sun Ruikang Liu Haoli Bai Han Bao Kang Zhao ... Lu Hou Chun Yuan Xin Jiang W. Liu Jun Yao MQ 46 3 0 12 Oct 2024
DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models Wenlong Deng Yize Zhao V. Vakilian Minghui Chen Xiaoxiao Li Christos Thrampoulidis 33 3 0 12 Oct 2024
Differential Transformer Tianzhu Ye Li Dong Yuqing Xia Yutao Sun Yi Zhu Gao Huang Furu Wei 35 0 0 07 Oct 2024
House of Cards: Massive Weights in LLMs Jaehoon Oh Seungjun Shin Dokwan Oh 35 1 0 02 Oct 2024
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition Stephen Zhang V. Papyan VLM 36 1 0 20 Sep 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 46 9 0 24 Jul 2024
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension Yijiong Yu Huiqiang Jiang Xufang Luo Qianhui Wu Chin-Yew Lin Dongsheng Li Yuqing Yang Yongfeng Huang L. Qiu 35 9 0 04 Jun 2024
Understanding and Minimising Outlier Features in Neural Network Training Bobby He Lorenzo Noci Daniele Paliotta Imanol Schlag Thomas Hofmann 21 3 0 29 May 2024
Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models Dennis Wu Jerry Yao-Chieh Hu Teng-Yun Hsiao Han Liu 33 28 0 04 Apr 2024
The Falcon Series of Open Language Models Ebtesam Almazrouei Hamza Alobeidli Abdulaziz Alshamsi Alessandro Cappelli Ruxandra-Aimée Cojocaru ... Quentin Malartic Daniele Mazzotta Badreddine Noune B. Pannier Guilherme Penedo AI4TS ALM 104 389 0 28 Nov 2023
Unveiling A Core Linguistic Region in Large Language Models Jun Zhao Zhihao Zhang Yide Ma Qi Zhang Tao Gui Luhui Gao Xuanjing Huang 59 5 0 23 Oct 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 197 2,232 0 22 Mar 2023
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng-Zhen Zhang Yuxiao Dong Jie Tang BDL LRM 240 1,070 0 05 Oct 2022
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 237 453 0 24 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 301 11,730 0 04 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 315 8,261 0 28 Jan 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 258 7,337 0 11 Nov 2021
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality William Timkey Marten van Schijndel 213 110 0 09 Sep 2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 234 690 0 27 Aug 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 236 1,508 0 31 Dec 2020