Outliers with Opposing Signals Have an Outsized Effect on Neural Network
Optimization

Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization

7 November 2023

Andrej Risteski

Papers citing "Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization"

7 / 7 papers shown

Title
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models Frederik Kunstner Robin Yadav Alan Milligan Mark Schmidt Alberto Bietti 29 26 0 29 Feb 2024
Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs Kelvin Guu Albert Webson Ellie Pavlick Lucas Dixon Ian Tenney Tolga Bolukbasi TDI 66 33 0 14 Mar 2023
Understanding Edge-of-Stability Training Dynamics with a Minimalist Example Xingyu Zhu Zixuan Wang Xiang Wang Mo Zhou Rong Ge 64 35 0 07 Oct 2022
Understanding Gradient Descent on Edge of Stability in Deep Learning Sanjeev Arora Zhiyuan Li A. Panigrahi MLT 75 88 0 19 May 2022
Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization Frederik Benzing ODL 35 23 0 28 Jan 2022
Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization Stanislaw Jastrzebski Devansh Arpit Oliver Åstrand Giancarlo Kerg Huan Wang Caiming Xiong R. Socher Kyunghyun Cho Krzysztof J. Geras AI4CE 177 65 0 28 Dec 2020
The large learning rate phase of deep learning: the catapult mechanism Aitor Lewkowycz Yasaman Bahri Ethan Dyer Jascha Narain Sohl-Dickstein Guy Gur-Ari ODL 153 232 0 04 Mar 2020