Implicit Bias and Fast Convergence Rates for Self-attention

8 February 2024

Papers citing "Implicit Bias and Fast Convergence Rates for Self-attention"

12 / 12 papers shown

Title
How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias Ruiquan Huang Yingbin Liang Jing Yang 43 0 0 02 May 2025
Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity Ruifeng Ren Yong Liu 30 0 0 26 Apr 2025
Gating is Weighting: Understanding Gated Linear Attention through In-context Learning Yingcong Li Davoud Ataee Tarzanagh A. S. Rawat Maryam Fazel Samet Oymak 23 0 0 06 Apr 2025
When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective Alireza Mousavi-Hosseini Clayton Sanford Denny Wu Murat A. Erdogdu 36 0 0 14 Mar 2025
Training Dynamics of In-Context Learning in Linear Attention Yedi Zhang Aaditya K. Singh Peter E. Latham Andrew Saxe MLT 53 1 0 28 Jan 2025
On the Learn-to-Optimize Capabilities of Transformers in In-Context Sparse Recovery Renpu Liu Ruida Zhou Cong Shen Jing Yang 23 0 0 17 Oct 2024
Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context Spencer Frei Gal Vardi MLT 21 3 0 02 Oct 2024
Non-asymptotic Convergence of Training Transformers for Next-token Prediction Ruiquan Huang Yingbin Liang Jing Yang 16 5 0 25 Sep 2024
Implicit Regularization of Gradient Flow on One-Layer Softmax Attention Heejune Sheen Siyu Chen Tianhao Wang Harrison H. Zhou MLT 23 10 0 13 Mar 2024
Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be Frederik Kunstner Jacques Chen J. Lavington Mark W. Schmidt 38 42 0 27 Apr 2023
Benign Overfitting for Two-layer ReLU Convolutional Neural Networks Yiwen Kou Zi-Yuan Chen Yuanzhou Chen Quanquan Gu MLT 39 12 0 07 Mar 2023
Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization Spencer Frei Gal Vardi Peter L. Bartlett Nathan Srebro 24 22 0 02 Mar 2023