Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections

v1v2 (latest)

Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections

International Conference on Machine Learning (ICML), 2021

13 February 2021

Chris Holmes

Mert Gurbuzbalaban

ArXiv (abs)PDF HTML

Papers citing "Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections"

12 / 12 papers shown

Title
Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails Haichao Sha Yang Cao Yong Liu Yuncheng Wu Ruixuan Liu Hong Chen 214 4 0 27 May 2024
High Probability Analysis for Non-Convex Stochastic Optimization with ClippingEuropean Conference on Artificial Intelligence (ECAI), 2023 Shaojie Li Yong Liu 170 5 0 25 Jul 2023
Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGDInternational Conference on Machine Learning (ICML), 2023 Yijun Wan Melih Barsbey Milad Sefidgaran Umut Simsekli 204 5 0 13 Jun 2023
Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy PerspectiveInternational Joint Conference on Artificial Intelligence (IJCAI), 2023 Yuxin Dong Tieliang Gong Hao Chen Chen Li 158 5 0 02 May 2023
Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptation Zijian Liu Zhengyuan Zhou 211 18 0 22 Mar 2023
Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient DescentNeural Information Processing Systems (NeurIPS), 2022 Soon Hoe Lim Yijun Wan Umut cSimcsekli 201 14 0 23 May 2022
Mirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise VarianceAnnual Conference Computational Learning Theory (COLT), 2022 Nuri Mert Vural Lu Yu Krishnakumar Balasubramanian S. Volgushev Murat A. Erdogdu 124 31 0 23 Feb 2022
Intrinsic Dimension, Persistent Homology and Generalization in Neural NetworksNeural Information Processing Systems (NeurIPS), 2021 Tolga Birdal Aaron Lou Leonidas Guibas Umut cSimcsekli 171 82 0 25 Nov 2021
On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications Ziqiao Wang Yongyi Mao FedML MLT 252 33 0 07 Oct 2021
Heavy Tails in SGD and Compressibility of Overparametrized Neural NetworksNeural Information Processing Systems (NeurIPS), 2021 Melih Barsbey Romain Chor Murat A. Erdogdu Gaël Richard Umut Simsekli 135 50 0 07 Jun 2021
Variational Autoencoders: A Harmonic PerspectiveInternational Conference on Artificial Intelligence and Statistics (AISTATS), 2021 A. Camuto M. Willetts DRL 177 2 0 31 May 2021
Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks Umut Simsekli Ozan Sener George Deligiannidis Murat A. Erdogdu 185 65 0 16 Jun 2020