v1v2 (latest)

Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning

17 December 2020

Papers citing "Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning"

50 / 105 papers shown

Title
Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs D. Chistikov Matthias Englert R. Lazic MLT 94 12 0 10 Jun 2023
Gradient descent in matrix factorization: Understanding large initialization Hengchao Chen Xin Chen Mohamad Elmasri Qiang Sun AI4CE 52 1 0 30 May 2023
Pruning at Initialization -- A Sketching Perspective Noga Bar Raja Giryes 112 1 0 27 May 2023
$Implicit bias of SGD in $L_{2}$-regularized linear DNNs: One-way jumps from high to low rank$ Implicit bias of SGD in $L_{2}$ -regularized linear DNNs: One-way jumps from high to low rank Zihan Wang Arthur Jacot 93 21 0 25 May 2023
Robust Sparse Mean Estimation via Incremental Learning Jianhao Ma Ruidi Chen Yinghui He Salar Fattahi Wei Hu 67 0 0 24 May 2023
Deep ReLU Networks Have Surprisingly Simple Polytopes Fenglei Fan Wei Huang Xiang-yu Zhong Lecheng Ruan T. Zeng Huan Xiong Fei Wang 108 5 0 16 May 2023
Saddle-to-Saddle Dynamics in Diagonal Linear Networks Scott Pesme Nicolas Flammarion 126 39 0 02 Apr 2023
First-order ANIL learns linear representations despite misspecified latent dimension Oğuz Kaan Yüksel Etienne Boursier Nicolas Flammarion 82 0 0 02 Mar 2023
Penalising the biases in norm regularisation enforces sparsity Etienne Boursier Nicolas Flammarion 127 17 0 02 Mar 2023
Learning time-scales in two-layers neural networks Raphael Berthier Andrea Montanari Kangjie Zhou 196 38 0 28 Feb 2023
Simplicity Bias in 1-Hidden Layer Neural Networks Depen Morwani Jatin Batra Prateek Jain Praneeth Netrapalli 103 21 0 01 Feb 2023
Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regression Mo Zhou Rong Ge 114 2 0 01 Feb 2023
Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing Jikai Jin Zhiyuan Li Kaifeng Lyu S. Du Jason D. Lee MLT 110 37 0 27 Jan 2023
A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorization JIAN-PENG Cao Chao Qian Yihui Huang Dicheng Chen Yuncheng Gao Jiyang Dong D. Guo X. Qu 114 1 0 29 Dec 2022
Rank-1 Matrix Completion with Gradient Descent and Small Random Initialization Daesung Kim Hye Won Chung 86 2 0 19 Dec 2022
Infinite-width limit of deep linear neural networks Lénaïc Chizat Maria Colombo Xavier Fernández-Real Alessio Figalli 80 16 0 29 Nov 2022
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models Hong Liu Sang Michael Xie Zhiyuan Li Tengyu Ma AI4CE 126 55 0 25 Oct 2022
Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets Edo Cohen-Karlik Itamar Menuhin-Gruman Raja Giryes Nadav Cohen Amir Globerson 76 5 0 25 Oct 2022
Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data Spencer Frei Gal Vardi Peter L. Bartlett Nathan Srebro Wei Hu MLT 83 42 0 13 Oct 2022
Behind the Scenes of Gradient Descent: A Trajectory Analysis via Basis Function Decomposition Jianhao Ma Li-Zhen Guo Salar Fattahi 73 4 0 01 Oct 2022
Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functions Arthur Jacot 125 27 0 29 Sep 2022
Deep Linear Networks can Benignly Overfit when Shallow Ones Do Niladri S. Chatterji Philip M. Long 96 8 0 19 Sep 2022
Incremental Learning in Diagonal Linear Networks Raphael Berthier CLL AI4CE 93 17 0 31 Aug 2022
On the Implicit Bias in Deep-Learning Algorithms Gal Vardi FedML AI4CE 91 81 0 26 Aug 2022
Implicit Regularization with Polynomial Growth in Deep Tensor Factorization Kais Hariz Hachem Kadri Stéphane Ayache Maher Moakher Thierry Artières 72 3 0 18 Jul 2022
Blessing of Nonconvexity in Deep Linear Models: Depth Flattens the Optimization Landscape Around the True Solution Jianhao Ma Salar Fattahi 75 5 0 15 Jul 2022
Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent Zhiyuan Li Tianhao Wang Jason D. Lee Sanjeev Arora 100 29 0 08 Jul 2022
Tensor-on-Tensor Regression: Riemannian Optimization, Over-parameterization, Statistical-computational Gap, and Their Interplay Yuetian Luo Anru R. Zhang 106 20 0 17 Jun 2022
Reconstructing Training Data from Trained Neural Networks Niv Haim Gal Vardi Gilad Yehudai Ohad Shamir Michal Irani 116 141 0 15 Jun 2022
Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction Kaifeng Lyu Zhiyuan Li Sanjeev Arora FAtt 121 75 0 14 Jun 2022
Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs Etienne Boursier Loucas Pillaud-Vivien Nicolas Flammarion ODL 77 61 0 02 Jun 2022
$Feature Learning in $L_{2}$-regularized DNNs: Attraction/Repulsion and Sparsity$ Feature Learning in $L_{2}$ -regularized DNNs: Attraction/Repulsion and Sparsity Arthur Jacot Eugene Golikov Clément Hongler Franck Gabriel MLT 75 18 0 31 May 2022
Machine Learning and Deep Learning -- A review for Ecologists Maximilian Pichler F. Hartig 120 144 0 11 Apr 2022
Robust Training under Label Noise by Over-parameterization Sheng Liu Zhihui Zhu Qing Qu Chong You NoLa OOD 77 109 0 28 Feb 2022
Support Vectors and Gradient Dynamics of Single-Neuron ReLU Networks Sangmin Lee Byeongsu Sim Jong Chul Ye MLT 39 0 0 11 Feb 2022
Implicit Regularization Towards Rank Minimization in ReLU Networks Nadav Timor Gal Vardi Ohad Shamir 93 51 0 30 Jan 2022
Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks Noam Razin Asaf Maman Nadav Cohen 132 29 0 27 Jan 2022
The Implicit Regularization of Momentum Gradient Descent with Early Stopping Li Wang Yingcong Zhou Zhiguo Fu 107 2 0 14 Jan 2022
Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias Frederic Koehler Viraj Mehta Chenghui Zhou Andrej Risteski DRL 87 13 0 13 Dec 2021
Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias Kaifeng Lyu Zhiyuan Li Runzhe Wang Sanjeev Arora MLT 110 76 0 26 Oct 2021
ZerO Initialization: Initializing Neural Networks with only Zeros and Ones Jiawei Zhao Florian Schäfer Anima Anandkumar 87 26 0 25 Oct 2021
What Happens after SGD Reaches Zero Loss? --A Mathematical Framework Zhiyuan Li Tianhao Wang Sanjeev Arora MLT 121 105 0 13 Oct 2021
On Margin Maximization in Linear and ReLU Networks Gal Vardi Ohad Shamir Nathan Srebro 162 30 0 06 Oct 2021
Implicit Sparse Regularization: The Impact of Depth and Early Stopping Jiangyuan Li Thanh V. Nguyen Chinmay Hegde R. K. Wong 81 30 0 12 Aug 2021
Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks Shih-Yu Sun Vimal Thilak Etai Littwin Omid Saremi J. Susskind 20 0 0 02 Jul 2021
Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity Arthur Jacot François Ged Berfin cSimcsek Clément Hongler Franck Gabriel 84 55 0 30 Jun 2021
Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction Dominik Stöger Mahdi Soltanolkotabi ODL 80 78 0 28 Jun 2021
Understanding Deflation Process in Over-parametrized Tensor Decomposition Rong Ge Y. Ren Xiang Wang Mo Zhou 66 19 0 11 Jun 2021
Implicit Regularization in Matrix Sensing via Mirror Descent Fan Wu Patrick Rebeschini 32 11 0 28 May 2021
The Low-Rank Simplicity Bias in Deep Networks Minyoung Huh H. Mobahi Richard Y. Zhang Brian Cheung Pulkit Agrawal Phillip Isola 117 116 0 18 Mar 2021