v1v2 (latest)

On the interplay between noise and curvature and its effect on optimization and generalization

18 June 2019

Valentin Thomas

Fabian Pedregosa

B. V. Merrienboer

Pierre-Antoine Mangazol

Yoshua Bengio

Nicolas Le Roux

ArXiv (abs)PDF HTML

Papers citing "On the interplay between noise and curvature and its effect on optimization and generalization"

20 / 20 papers shown

Title
An Improved Empirical Fisher Approximation for Natural Gradient Descent Xiaodong Wu Wenyi Yu Chao Zhang Philip Woodland 74 4 0 10 Jun 2024
How to select an objective function using information theory T. Hodson T. Over T. Smith Lucy M. Marshall 27 1 0 10 Dec 2022
Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States Ziqiao Wang Yongyi Mao 111 12 0 19 Nov 2022
Noise Injection as a Probe of Deep Learning Dynamics Noam Levi I. Bloch M. Freytsis T. Volansky 72 2 0 24 Oct 2022
Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes M. Kodryan E. Lobacheva M. Nakhodnov Dmitry Vetrov 105 17 0 08 Sep 2022
Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning Lin Zhang Shaoshuai Shi Wei Wang Yue Liu 65 10 0 30 Jun 2022
Hybrid quantum ResNet for car classification and its hyperparameter optimization Asel Sagingalieva Mohammad Kordzanganeh Andrii Kurkin Artem Melnikov Daniil Kuhmistrov M. Perelshtein A. Melnikov Andrea Skolik David Von Dollen 153 38 0 10 May 2022
A generalization gap estimation for overparameterized models via the Langevin functional variance Akifumi Okuno Keisuke Yano 99 2 0 07 Dec 2021
Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization Alexandre Ramé Corentin Dancette Matthieu Cord OOD 139 210 0 07 Sep 2021
The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion D. Kunin Javier Sagastuy-Breña Lauren Gillespie Eshed Margalit Hidenori Tanaka Surya Ganguli Daniel L. K. Yamins 93 20 0 19 Jul 2021
Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks Shaoshuai Shi Lin Zhang Yue Liu 123 9 0 14 Jul 2021
M-FAC: Efficient Matrix-Free Approximations of Second-Order Information Elias Frantar Eldar Kurtic Dan Alistarh 72 59 0 07 Jul 2021
ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure Felix Dangel Lukas Tatzel Philipp Hennig 83 13 0 04 Jun 2021
Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics Charles H. Martin Michael W. Mahoney 75 20 0 01 Jun 2021
Strength of Minibatch Noise in SGD Liu Ziyin Kangqiao Liu Takashi Mori Masakuni Ueda ODL MLT 46 35 0 10 Feb 2021
Deep Learning is Singular, and That's Good Daniel Murfet Susan Wei Biwei Huang Hui Li Jesse Gell-Redman T. Quella UQCV 79 29 0 22 Oct 2020
Scalable and Practical Natural Gradient for Large-Scale Deep Learning Kazuki Osawa Yohei Tsuji Yuichiro Ueno Akira Naruse Chuan-Sheng Foo Rio Yokota 85 37 0 13 Feb 2020
Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates Jeffrey Negrea Mahdi Haghifam Gintare Karolina Dziugaite Ashish Khisti Daniel M. Roy FedML 195 153 0 06 Nov 2019
Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation Konstantinos Pitas 70 8 0 06 Sep 2019
The Generalization-Stability Tradeoff In Neural Network Pruning Brian Bartoldson Ari S. Morcos Adrian Barbu G. Erlebacher 94 76 0 09 Jun 2019