Understanding and Minimising Outlier Features in Neural Network Training

Understanding and Minimising Outlier Features in Neural Network Training

29 May 2024

Daniele Paliotta

Papers citing "Understanding and Minimising Outlier Features in Neural Network Training"

13 / 13 papers shown

Title
From Attention to Activation: Unravelling the Enigmas of Large Language Models Prannay Kaul Chengcheng Ma Ismail Elezi Jiankang Deng 18 2 0 22 Oct 2024
AERO: Softmax-Only LLMs for Efficient Private Inference N. Jha Brandon Reagen 20 1 0 16 Oct 2024
ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models N. Jha Brandon Reagen OffRL AI4CE 20 0 0 12 Oct 2024
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization Aniruddha Nrusimha Mayank Mishra Naigang Wang Dan Alistarh Rameswar Panda Yoon Kim MQ 52 8 0 04 Apr 2024
Massive Activations in Large Language Models Mingjie Sun Xinlei Chen J. Zico Kolter Zhuang Liu 60 64 0 27 Feb 2024
Stabilizing Transformer Training by Preventing Attention Entropy Collapse Shuangfei Zhai Tatiana Likhomanenko Etai Littwin Dan Busbridge Jason Ramapuram Yizhe Zhang Jiatao Gu J. Susskind AAML 35 64 0 11 Mar 2023
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng-Zhen Zhang Yuxiao Dong Jie Tang BDL LRM 240 1,070 0 05 Oct 2022
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency Giovanni Puccetti Anna Rogers Aleksandr Drozd F. Dell’Orletta 63 42 0 23 May 2022
Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping James Martens Andy Ballard Guillaume Desjardins G. Swirszcz Valentin Dalibard Jascha Narain Sohl-Dickstein S. Schoenholz 74 40 0 05 Oct 2021
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality William Timkey Marten van Schijndel 213 110 0 09 Sep 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 220 450 0 11 Feb 2021
Stable ResNet Soufiane Hayou Eugenio Clerico Bo He George Deligiannidis Arnaud Doucet Judith Rousseau ODL SSeg 46 50 0 24 Oct 2020
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks Lechao Xiao Yasaman Bahri Jascha Narain Sohl-Dickstein S. Schoenholz Jeffrey Pennington 220 330 0 14 Jun 2018