Finite Versus Infinite Neural Networks: an Empirical Study

31 July 2020

Jascha Narain Sohl-Dickstein

ArXiv PDF HTML

Papers citing "Finite Versus Infinite Neural Networks: an Empirical Study"

50 / 52 papers shown

Title
The Optimization Landscape of SGD Across the Feature Learning Strength Alexander B. Atanasov Alexandru Meterez James B. Simon C. Pehlevan 43 2 0 06 Oct 2024
Equivariant Neural Tangent Kernels Philipp Misof Pan Kessel Jan E. Gerken 50 0 0 10 Jun 2024
NTK-Guided Few-Shot Class Incremental Learning Jingren Liu Zhong Ji Yanwei Pang Yunlong Yu CLL 34 3 0 19 Mar 2024
Weak Correlations as the Underlying Principle for Linearization of Gradient-Based Learning Systems Ori Shem-Ur Yaron Oz 14 0 0 08 Jan 2024
On the Neural Tangent Kernel of Equilibrium Models Zhili Feng J. Zico Kolter 16 6 0 21 Oct 2023
Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamics Yehonatan Avidan Qianyi Li H. Sompolinsky 60 8 0 08 Sep 2023
Sparsity-depth Tradeoff in Infinitely Wide Deep Neural Networks Chanwoo Chun Daniel D. Lee BDL 30 2 0 17 May 2023
Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks Eshaan Nichani Alexandru Damian Jason D. Lee MLT 36 13 0 11 May 2023
Do deep neural networks have an inbuilt Occam's razor? Chris Mingard Henry Rees Guillermo Valle Pérez A. Louis UQCV BDL 19 15 0 13 Apr 2023
Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks Blake Bordelon C. Pehlevan MLT 38 29 0 06 Apr 2023
Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training Luís Carvalho Joao L. Costa José Mourao Gonccalo Oliveira AI4CE 13 1 0 06 Apr 2023
Dataset Distillation with Convexified Implicit Gradients Noel Loo Ramin Hasani Mathias Lechner Daniela Rus DD 29 41 0 13 Feb 2023
On a continuous time model of gradient descent dynamics and instability in deep learning Mihaela Rosca Yan Wu Chongli Qin Benoit Dherin 16 6 0 03 Feb 2023
A Simple Algorithm For Scaling Up Kernel Methods Tengyu Xu Bryan T. Kelly Semyon Malamud 11 0 0 26 Jan 2023
Dataset Distillation: A Comprehensive Review Ruonan Yu Songhua Liu Xinchao Wang DD 39 121 0 17 Jan 2023
Data Distillation: A Survey Noveen Sachdeva Julian McAuley DD 37 73 0 11 Jan 2023
Characterizing the Spectrum of the NTK via a Power Series Expansion Michael Murray Hui Jin Benjamin Bowman Guido Montúfar 30 11 0 15 Nov 2022
Evolution of Neural Tangent Kernels under Benign and Adversarial Training Noel Loo Ramin Hasani Alexander Amini Daniela Rus AAML 28 12 0 21 Oct 2022
Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernel Sungyub Kim Si-hun Park Kyungsu Kim Eunho Yang BDL 24 4 0 30 Sep 2022
Joint Embedding Self-Supervised Learning in the Kernel Regime B. Kiani Randall Balestriero Yubei Chen S. Lloyd Yann LeCun SSL 41 13 0 29 Sep 2022
Approximation results for Gradient Descent trained Shallow Neural Networks in $1d$ R. Gentile G. Welper ODL 44 6 0 17 Sep 2022
Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization Xingyou Song Sagi Perel Chansoo Lee Greg Kochanski Daniel Golovin 29 26 0 27 Jul 2022
Learning sparse features can lead to overfitting in neural networks Leonardo Petrini Francesco Cagnetta Eric Vanden-Eijnden M. Wyart MLT 29 23 0 24 Jun 2022
Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials Eshaan Nichani Yunzhi Bai Jason D. Lee 21 10 0 08 Jun 2022
Infinite Recommendation Networks: A Data-Centric Approach Noveen Sachdeva Mehak Preet Dhaliwal Carole-Jean Wu Julian McAuley DD 31 28 0 03 Jun 2022
Contrasting random and learned features in deep Bayesian linear regression Jacob A. Zavatone-Veth William L. Tong C. Pehlevan BDL MLT 28 26 0 01 Mar 2022
Investigating Power laws in Deep Representation Learning Arna Ghosh Arnab Kumar Mondal Kumar Krishna Agrawal Blake A. Richards SSL OOD 11 19 0 11 Feb 2022
Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditions Maksim Velikanov Dmitry Yarotsky 4 6 0 02 Feb 2022
A generalization gap estimation for overparameterized models via the Langevin functional variance Akifumi Okuno Keisuke Yano 22 1 0 07 Dec 2021
Depth induces scale-averaging in overparameterized linear Bayesian neural networks Jacob A. Zavatone-Veth C. Pehlevan BDL UQCV MDE 36 8 0 23 Nov 2021
Subquadratic Overparameterization for Shallow Neural Networks Chaehwan Song Ali Ramezani-Kebrya Thomas Pethick Armin Eftekhari V. Cevher 22 32 0 02 Nov 2021
Neural Networks as Kernel Learners: The Silent Alignment Effect Alexander B. Atanasov Blake Bordelon C. Pehlevan MLT 13 74 0 29 Oct 2021
Bayesian neural network unit priors and generalized Weibull-tail property M. Vladimirova Julyan Arbel Stéphane Girard BDL 54 9 0 06 Oct 2021
Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks Adityanarayanan Radhakrishnan George Stefanakis M. Belkin Caroline Uhler 30 25 0 31 Jul 2021
Dataset Distillation with Infinitely Wide Convolutional Networks Timothy Nguyen Roman Novak Lechao Xiao Jaehoon Lee DD 24 229 0 27 Jul 2021
How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective Akhilan Boopathy Ila Fiete 16 9 0 15 Jun 2021
The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective Geoff Pleiss John P. Cunningham 26 24 0 11 Jun 2021
A Neural Tangent Kernel Perspective of GANs Jean-Yves Franceschi Emmanuel de Bézenac Ibrahim Ayed Mickaël Chen Sylvain Lamprier Patrick Gallinari 29 26 0 10 Jun 2021
A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs Gadi Naveh Z. Ringel SSL MLT 20 31 0 08 Jun 2021
Relative stability toward diffeomorphisms indicates performance in deep nets Leonardo Petrini Alessandro Favero Mario Geiger M. Wyart OOD 28 15 0 06 May 2021
A Neural Pre-Conditioning Active Learning Algorithm to Reduce Label Complexity Seo Taek Kong Soomin Jeon Dongbin Na Jaewon Lee Honglak Lee Kyu-Hwan Jung 13 6 0 08 Apr 2021
Dataset Meta-Learning from Kernel Ridge-Regression Timothy Nguyen Zhourung Chen Jaehoon Lee DD 19 238 0 30 Oct 2020
Are wider nets better given the same number of parameters? A. Golubeva Behnam Neyshabur Guy Gur-Ari 13 44 0 27 Oct 2020
Stable ResNet Soufiane Hayou Eugenio Clerico Bo He George Deligiannidis Arnaud Doucet Judith Rousseau ODL SSeg 46 51 0 24 Oct 2020
Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization Neha S. Wadia Daniel Duckworth S. Schoenholz Ethan Dyer Jascha Narain Sohl-Dickstein 19 13 0 17 Aug 2020
Scalable Uncertainty for Computer Vision with Functional Variational Inference Eduardo D C Carvalho R. Clark Andrea Nicastro Paul H. J. Kelly BDL UQCV 114 22 0 06 Mar 2020
The large learning rate phase of deep learning: the catapult mechanism Aitor Lewkowycz Yasaman Bahri Ethan Dyer Jascha Narain Sohl-Dickstein Guy Gur-Ari ODL 159 234 0 04 Mar 2020
Self-Distillation Amplifies Regularization in Hilbert Space H. Mobahi Mehrdad Farajtabar Peter L. Bartlett 19 226 0 13 Feb 2020
Why bigger is not always better: on finite and infinite neural networks Laurence Aitchison 173 51 0 17 Oct 2019
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks Lechao Xiao Yasaman Bahri Jascha Narain Sohl-Dickstein S. Schoenholz Jeffrey Pennington 220 348 0 14 Jun 2018