Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup

18 June 2019

Lenka Zdeborová

Papers citing "Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup"

32 / 32 papers shown

Title
Analytic theory of dropout regularization Francesco Mori Francesca Mignacco 29 0 0 12 May 2025
Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime Francesco Camilli D. Tieplova Eleonora Bergamin Jean Barbier 109 0 0 06 May 2025
Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networks Devon Jarvis Richard Klein Benjamin Rosman Andrew M. Saxe MLT 66 1 0 08 Mar 2025
Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence Berfin Simsek Amire Bendjeddou Daniel Hsu 44 0 0 13 Nov 2024
How a student becomes a teacher: learning and forgetting through Spectral methods Lorenzo Giambagli L. Buffoni Lorenzo Chicchi Duccio Fanelli 19 7 0 19 Oct 2023
The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensions Nishil Patel Sebastian Lee Stefano Sarao Mannelli Sebastian Goldt Adrew Saxe OffRL 28 3 0 17 Jun 2023
Phase transitions in the mini-batch size for sparse and dense two-layer neural networks Raffaele Marino F. Ricci-Tersenghi 30 14 0 10 May 2023
Online Learning for the Random Feature Model in the Student-Teacher Framework Roman Worschech B. Rosenow 38 0 0 24 Mar 2023
From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks Luca Arnaboldi Ludovic Stephan Florent Krzakala Bruno Loureiro MLT 30 31 0 12 Feb 2023
Learning Single-Index Models with Shallow Neural Networks A. Bietti Joan Bruna Clayton Sanford M. Song 164 67 0 27 Oct 2022
Annihilation of Spurious Minima in Two-Layer ReLU Networks Yossi Arjevani M. Field 16 8 0 12 Oct 2022
Sudakov-Fernique post-AMP, and a new proof of the local convexity of the TAP free energy Michael Celentano 31 20 0 19 Aug 2022
Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit Boaz Barak Benjamin L. Edelman Surbhi Goel Sham Kakade Eran Malach Cyril Zhang 30 123 0 18 Jul 2022
Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting Neil Rohit Mallinar James B. Simon Amirhesam Abedsoltan Parthe Pandit M. Belkin Preetum Nakkiran 24 37 0 14 Jul 2022
Neural Collapse: A Review on Modelling Principles and Generalization Vignesh Kothapalli 21 71 0 08 Jun 2022
High-dimensional limit theorems for SGD: Effective dynamics and critical scaling Gerard Ben Arous Reza Gheissari Aukosh Jagannath 49 58 0 08 Jun 2022
Optimal learning rate schedules in high-dimensional non-convex optimization problems Stéphane dÁscoli Maria Refinetti Giulio Biroli 16 7 0 09 Feb 2022
Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks R. Veiga Ludovic Stephan Bruno Loureiro Florent Krzakala Lenka Zdeborová MLT 10 31 0 01 Feb 2022
Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics Chunheng Jiang Tejaswini Pedapati Pin-Yu Chen Yizhou Sun Jianxi Gao 21 2 0 11 Jan 2022
Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations Jiayao Zhang Hua Wang Weijie J. Su 32 7 0 11 Oct 2021
Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II Yossi Arjevani M. Field 28 18 0 21 Jul 2021
An Analytical Theory of Curriculum Learning in Teacher-Student Networks Luca Saglietti Stefano Sarao Mannelli Andrew M. Saxe 17 25 0 15 Jun 2021
Understanding Robustness in Teacher-Student Setting: A New Perspective Zhuolin Yang Zhaoxi Chen Tiffany Cai Xinyun Chen Bo-wen Li Yuandong Tian AAML 27 2 0 25 Feb 2021
Quantum field-theoretic machine learning Dimitrios Bachtis Gert Aarts B. Lucini AI4CE 17 28 0 18 Feb 2021
Align, then memorise: the dynamics of learning with feedback alignment Maria Refinetti Stéphane dÁscoli Ruben Ohana Sebastian Goldt 26 36 0 24 Nov 2020
Gradient Starvation: A Learning Proclivity in Neural Networks Mohammad Pezeshki Sekouba Kaba Yoshua Bengio Aaron Courville Doina Precup Guillaume Lajoie MLT 45 257 0 18 Nov 2020
The Gaussian equivalence of generative models for learning with shallow neural networks Sebastian Goldt Bruno Loureiro Galen Reeves Florent Krzakala M. Mézard Lenka Zdeborová BDL 38 100 0 25 Jun 2020
Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent Mehdi Abbana Bennani Thang Doan Masashi Sugiyama CLL 48 61 0 21 Jun 2020
Implicit Regularization in Deep Learning May Not Be Explainable by Norms Noam Razin Nadav Cohen 21 155 0 13 May 2020
Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation Elisa Oostwal Michiel Straat Michael Biehl MLT 56 54 0 16 Oct 2019
Modelling the influence of data structure on learning in neural networks: the hidden manifold model Sebastian Goldt M. Mézard Florent Krzakala Lenka Zdeborová BDL 21 51 0 25 Sep 2019
Norm-Based Capacity Control in Neural Networks Behnam Neyshabur Ryota Tomioka Nathan Srebro 119 577 0 27 Feb 2015