v1v2v3v4v5v6v7v8v9v10 (latest)

Tractable structured natural gradient descent using local parameterizations

15 February 2021

Wu Lin

Frank Nielsen

Mohammad Emtiyaz Khan

Mark Schmidt

ArXiv (abs)PDF HTML

Papers citing "Tractable structured natural gradient descent using local parameterizations"

20 / 20 papers shown

Title
Spectral-factorized Positive-definite Curvature Learning for NN Training Wu Lin Felix Dangel Runa Eschenhagen Juhan Bae Richard E. Turner Roger B. Grosse 171 0 0 10 Feb 2025
Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradient Shaoqi Wang Chunjie Yang Siwei Lou 40 1 0 23 Sep 2024
Understanding Stochastic Natural Gradient Variational Inference Kaiwen Wu Jacob R. Gardner BDL 87 2 0 04 Jun 2024
Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective Wu Lin Felix Dangel Runa Eschenhagen Juhan Bae Richard Turner Alireza Makhzani ODL 156 13 0 05 Feb 2024
Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC Wu Lin Felix Dangel Runa Eschenhagen Kirill Neklyudov Agustinus Kristiadi Richard Turner Alireza Makhzani 56 4 0 09 Dec 2023
Variational Bayes Made Easy Mohammad Emtiyaz Khan BDL 51 1 0 27 Apr 2023
The Lie-Group Bayesian Learning Rule E. M. Kıral Thomas Möllenhoff Mohammad Emtiyaz Khan BDL 47 3 0 08 Mar 2023
Natural Gradient Hybrid Variational Inference with Application to Deep Mixed Models Weiben Zhang M. Smith Worapree Maneesoonthorn Rubén Loaiza-Maya 26 1 0 27 Feb 2023
Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning Wu Lin Valentin Duruisseaux Melvin Leok Frank Nielsen Mohammad Emtiyaz Khan Mark Schmidt 104 10 0 20 Feb 2023
The Geometry of Neural Nets' Parameter Spaces Under Reparametrization Agustinus Kristiadi Felix Dangel Philipp Hennig 73 14 0 14 Feb 2023
A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models Oleg Arenz Philipp Dahlinger Zihan Ye Michael Volpp Gerhard Neumann 126 17 0 23 Sep 2022
Invariance Properties of the Natural Gradient in Overparametrised Systems Jesse van Oostrum J. Müller Nihat Ay 55 9 0 30 Jun 2022
On the Influence of Enforcing Model Identifiability on Learning dynamics of Gaussian Mixture Models Pascal Esser Frank Nielsen 32 0 0 17 Jun 2022
Towards Modeling and Resolving Singular Parameter Spaces using Stratifolds Pascal Esser Frank Nielsen 29 2 0 07 Dec 2021
PredProp: Bidirectional Stochastic Optimization with Precision Weighted Predictive Coding André Ofner Sebastian Stober 40 2 0 16 Nov 2021
Nys-Newton: Nyström-Approximated Curvature for Stochastic Optimization Dinesh Singh Hardik Tankaria M. Yamada ODL 69 3 0 16 Oct 2021
Analytic natural gradient updates for Cholesky factor in Gaussian variational approximation Linda S. L. Tan 84 12 0 01 Sep 2021
Structured second-order methods via natural gradient descent Wu Lin Frank Nielsen Mohammad Emtiyaz Khan Mark Schmidt ODL 56 10 0 22 Jul 2021
The Bayesian Learning Rule Mohammad Emtiyaz Khan Håvard Rue BDL 159 83 0 09 Jul 2021
Natural Reweighted Wake-Sleep Csongor-Huba Várady Riccardo Volpi Luigi Malagò Nihat Ay BDL 99 0 0 15 Aug 2020