Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing

27 January 2023

Papers citing "Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing"

31 / 31 papers shown

Title
Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization Ziqing Xu Hancheng Min Lachlan Ewen MacDonald Jinqi Luo Salma Tarmoun Enrique Mallada René Vidal AI4CE 46 0 0 10 Mar 2025
Towards Understanding Text Hallucination of Diffusion Models via Local Generation Bias Rui Lu Runzhe Wang Kaifeng Lyu Xitai Jiang Gao Huang Mengdi Wang DiffM 86 0 0 05 Mar 2025
On the Crucial Role of Initialization for Matrix Factorization Bingcong Li Liang Zhang Aryan Mokhtari Niao He 26 1 0 24 Oct 2024
Task Diversity Shortens the ICL Plateau Jaeyeon Kim Sehyun Kwon Joo Young Choi Jongho Park Jaewoong Cho Jason D. Lee Ernest K. Ryu MoMe 29 2 0 07 Oct 2024
The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization Haihan Zhang Yuanshi Liu Qianwen Chen Cong Fang 24 0 0 15 Sep 2024
Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning Nadav Cohen Noam Razin 19 0 0 25 Aug 2024
Non-convex matrix sensing: Breaking the quadratic rank barrier in the sample complexity Dominik Stoger Yizhe Zhu 16 1 0 20 Aug 2024
Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes Zhenfeng Tu Santiago Aranguri Arthur Jacot 17 1 0 27 May 2024
Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completion Zhiwei Bai Jiajie Zhao Yaoyu Zhang AI4CE 23 0 0 22 May 2024
Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations Akshay Kumar Jarvis D. Haupt ODL 40 3 0 12 Mar 2024
Absence of spurious solutions far from ground truth: A low-rank analysis with high-order losses Ziye Ma Ying Chen Javad Lavaei Somayeh Sojoudi 16 1 0 10 Mar 2024
The Implicit Bias of Heterogeneity towards Invariance: A Study of Multi-Environment Matrix Sensing Yang Xu Yihong Gu Cong Fang 25 0 0 03 Mar 2024
LoRA Training in the NTK Regime has No Spurious Local Minima Uijeong Jang Jason D. Lee Ernest K. Ryu 22 3 0 19 Feb 2024
Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networks Akshay Kumar Jarvis D. Haupt ODL 17 6 0 14 Feb 2024
Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics Soo Min Kwon Zekai Zhang Dogyoon Song Laura Balzano Qing Qu 27 2 0 08 Nov 2023
Algorithmic Regularization in Tensor Optimization: Towards a Lifted Approach in Matrix Sensing Ziye Ma Javad Lavaei Somayeh Sojoudi 24 2 0 24 Oct 2023
A Quadratic Synchronization Rule for Distributed Deep Learning Xinran Gu Kaifeng Lyu Sanjeev Arora Jingzhao Zhang Longbo Huang 28 1 0 22 Oct 2023
How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization Nuoya Xiong Lijun Ding Simon S. Du 11 11 0 03 Oct 2023
Asymmetric matrix sensing by gradient descent with small random initialization J. S. Wind 23 1 0 04 Sep 2023
Trained Transformers Learn Linear Models In-Context Ruiqi Zhang Spencer Frei Peter L. Bartlett 6 172 0 16 Jun 2023
Transformers learn through gradual rank increase Enric Boix-Adserà Etai Littwin Emmanuel Abbe Samy Bengio J. Susskind 29 33 0 12 Jun 2023
Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs D. Chistikov Matthias Englert R. Lazic MLT 27 12 0 10 Jun 2023
Gradient descent in matrix factorization: Understanding large initialization Hengchao Chen Xin Chen Mohamad Elmasri Qiang Sun AI4CE 11 1 0 30 May 2023
Fast and Accurate Estimation of Low-Rank Matrices from Noisy Measurements via Preconditioned Non-Convex Gradient Descent Jialun Zhang Hong-Ming Chiu Richard Y. Zhang 12 5 0 26 May 2023
Robust Sparse Mean Estimation via Incremental Learning Jianhao Ma Ruidi Chen Yinghui He S. Fattahi Wei Hu 13 0 0 24 May 2023
Saddle-to-Saddle Dynamics in Diagonal Linear Networks Scott Pesme Nicolas Flammarion 17 35 0 02 Apr 2023
Greedy Pruning with Group Lasso Provably Generalizes for Matrix Sensing Nived Rajaraman Devvrit Aryan Mokhtari Kannan Ramchandran 16 0 0 20 Mar 2023
Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks Noam Razin Asaf Maman Nadav Cohen 26 29 0 27 Jan 2022
What Happens after SGD Reaches Zero Loss? --A Mathematical Framework Zhiyuan Li Tianhao Wang Sanjeev Arora MLT 83 98 0 13 Oct 2021
Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noise Spencer Frei Yuan Cao Quanquan Gu FedML MLT 45 16 0 04 Jan 2021
Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition Hamed Karimi J. Nutini Mark W. Schmidt 119 1,190 0 16 Aug 2016