ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Models

24 May 2023

Papers citing "ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Models"

15 / 15 papers shown

Title
Emergence in non-neural models: grokking modular arithmetic via average gradient outer product Neil Rohit Mallinar Daniel Beaglehole Libin Zhu Adityanarayanan Radhakrishnan Parthe Pandit Misha Belkin 35 3 0 29 Jul 2024
Efficient adjustment for complex covariates: Gaining efficiency with DOPE Alexander Mangulad Christgau Niels Richard Hansen 29 2 0 20 Feb 2024
Feature learning as alignment: a structural property of gradient descent in non-linear neural networks Daniel Beaglehole Ioannis Mitliagkas Atish Agarwala MLT 24 2 0 07 Feb 2024
Linear Recursive Feature Machines provably recover low-rank matrices Adityanarayanan Radhakrishnan Misha Belkin D. Drusvyatskiy 51 8 0 09 Jan 2024
Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities Rahul Parhi Michael Unser 39 4 0 05 Oct 2023
Mechanism of feature learning in convolutional neural networks Daniel Beaglehole Adityanarayanan Radhakrishnan Parthe Pandit Misha Belkin FAtt MLT 24 14 0 01 Sep 2023
Autoencoders for discovering manifold dimension and coordinates in data from complex dynamical systems Kevin Zeng Carlos E. Pérez De Jesús Andrew J Fox M. Graham AI4CE 35 12 0 01 May 2023
Learning Single-Index Models with Shallow Neural Networks A. Bietti Joan Bruna Clayton Sanford M. Song 134 65 0 27 Oct 2022
Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functions Arthur Jacot 31 24 0 29 Sep 2022
Neural Networks Efficiently Learn Low-Dimensional Representations with SGD Alireza Mousavi-Hosseini Sejun Park M. Girotti Ioannis Mitliagkas Murat A. Erdogdu MLT 307 48 0 29 Sep 2022
Parallel Deep Neural Networks Have Zero Duality Gap Yifei Wang Tolga Ergen Mert Pilanci 59 10 0 13 Oct 2021
Understanding neural networks with reproducing kernel Banach spaces Francesca Bartolucci E. De Vito Lorenzo Rosasco S. Vigogna 36 48 0 20 Sep 2021
Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks Rahul Parhi Robert D. Nowak 46 37 0 18 Sep 2021
Initialization and Regularization of Factorized Neural Layers M. Khodak Neil A. Tenenholtz Lester W. Mackey Nicolò Fusi 52 56 0 03 May 2021
Norm-Based Capacity Control in Neural Networks Behnam Neyshabur Ryota Tomioka Nathan Srebro 111 551 0 27 Feb 2015