v1v2v3 (latest)

Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation

13 February 2019

Papers citing "Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation"

50 / 211 papers shown

Title
Infinite attention: NNGP and NTK for deep attention networks Jiri Hron Yasaman Bahri Jascha Narain Sohl-Dickstein Roman Novak 135 131 0 18 Jun 2020
The Recurrent Neural Tangent Kernel Sina Alemohammad Zichao Wang Randall Balestriero Richard Baraniuk AAML 183 82 0 18 Jun 2020
The Spectrum of Fisher Information of Deep Networks Achieving Dynamical IsometryInternational Conference on Artificial Intelligence and Statistics (AISTATS), 2020 Tomohiro Hayase Ryo Karakida 268 9 0 14 Jun 2020
Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks Z. Fan Zhichao Wang 195 85 0 25 May 2020
Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels E. Weinan Stephan Wojtowytsch 225 32 0 21 May 2020
Feature Purification: How Adversarial Training Performs Robust Deep Learning Zeyuan Allen-Zhu Yuanzhi Li MLT AAML 338 167 0 20 May 2020
Adversarial Robustness Guarantees for Random Deep Neural NetworksInternational Conference on Machine Learning (ICML), 2020 Giacomo De Palma B. Kiani S. Lloyd AAML OOD 131 9 0 13 Apr 2020
On the Neural Tangent Kernel of Deep Networks with Orthogonal InitializationInternational Joint Conference on Artificial Intelligence (IJCAI), 2020 Wei Huang Weitao Du R. Xu 136 40 0 13 Apr 2020
Mehler's Formula, Branching Process, and Compositional Kernels of Deep Neural NetworksJournal of the American Statistical Association (JASA), 2020 Tengyuan Liang Hai Tran-Bach 144 11 0 09 Apr 2020
On Infinite-Width Hypernetworks Etai Littwin Tomer Galanti Lior Wolf Greg Yang 387 11 0 27 Mar 2020
Scalable Uncertainty for Computer Vision with Functional Variational InferenceComputer Vision and Pattern Recognition (CVPR), 2020 Eduardo D C Carvalho R. Clark Andrea Nicastro Paul H. J. Kelly BDL UQCV 793 22 0 06 Mar 2020
Stable behaviour of infinitely wide deep neural networksInternational Conference on Artificial Intelligence and Statistics (AISTATS), 2020 Stefano Favaro S. Fortini Stefano Peluchetti BDL 153 31 0 01 Mar 2020
Deep Randomized Neural Networks Claudio Gallicchio Simone Scardapane OOD 212 71 0 27 Feb 2020
Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite NetworksAAAI Conference on Artificial Intelligence (AAAI), 2020 Russell Tsuchida Tim Pearce Christopher van der Heide Fred Roosta M. Gallagher 250 10 0 20 Feb 2020
Robust Pruning at InitializationInternational Conference on Learning Representations (ICLR), 2020 Soufiane Hayou Jean-François Ton Arnaud Doucet Yee Whye Teh 157 49 0 19 Feb 2020
On Layer Normalization in the Transformer ArchitectureInternational Conference on Machine Learning (ICML), 2020 Ruibin Xiong Yunchang Yang Di He Kai Zheng Shuxin Zheng Chen Xing Huishuai Zhang Yanyan Lan Liwei Wang Tie-Yan Liu AI4CE 352 1,209 0 12 Feb 2020
A Deep Conditioning Treatment of Neural NetworksInternational Conference on Algorithmic Learning Theory (ALT), 2020 Naman Agarwal Pranjal Awasthi Satyen Kale AI4CE 292 18 0 04 Feb 2020
On Random Kernels of Residual Architectures Etai Littwin Tomer Galanti Lior Wolf 232 4 0 28 Jan 2020
On the infinite width limit of neural networks with a standard parameterization Jascha Narain Sohl-Dickstein Roman Novak S. Schoenholz Jaehoon Lee 288 50 0 21 Jan 2020
Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical PerspectiveNeural Computation (Neural Comput.), 2020 S. Amari 150 14 0 20 Jan 2020
Disentangling Trainability and Generalization in Deep Neural Networks Lechao Xiao Jeffrey Pennington S. Schoenholz 173 34 0 30 Dec 2019
TRADI: Tracking deep neural network weight distributions for uncertainty estimationEuropean Conference on Computer Vision (ECCV), 2019 Gianni Franchi Andrei Bursuc Emanuel Aldea Séverine Dubuisson Isabelle Bloch UQCV 342 59 0 24 Dec 2019
Mean field theory for deep dropout networks: digging up gradient backpropagation deeplyEuropean Conference on Artificial Intelligence (ECAI), 2019 Wei Huang R. Xu Weitao Du Yutian Zeng Yunce Zhao 128 6 0 19 Dec 2019
Optimization for deep learning: theory and algorithms Tian Ding ODL 283 177 0 19 Dec 2019
Analytic expressions for the output evolution of a deep neural network Anastasia Borovykh 95 0 0 18 Dec 2019
Neural Tangents: Fast and Easy Infinite Neural Networks in PythonInternational Conference on Learning Representations (ICLR), 2019 Roman Novak Lechao Xiao Jiri Hron Jaehoon Lee Alexander A. Alemi Jascha Narain Sohl-Dickstein S. Schoenholz 188 248 0 05 Dec 2019
Richer priors for infinitely wide multi-layer perceptrons Russell Tsuchida Fred Roosta M. Gallagher 115 12 0 29 Nov 2019
Enhanced Convolutional Neural Tangent Kernels Zhiyuan Li Ruosong Wang Dingli Yu S. Du Wei Hu Ruslan Salakhutdinov Sanjeev Arora 182 136 0 03 Nov 2019
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian ProcessesNeural Information Processing Systems (NeurIPS), 2019 Greg Yang 445 219 0 28 Oct 2019
The Renyi Gaussian Process: Towards Improved GeneralizationIISE Transactions (IISE Trans.), 2019 Xubo Yue Raed Al Kontar 341 3 0 15 Oct 2019
The Local Elasticity of Neural NetworksInternational Conference on Learning Representations (ICLR), 2019 Hangfeng He Weijie J. Su 274 51 0 15 Oct 2019
Pathological spectra of the Fisher information metric and its variants in deep neural networksNeural Computation (Neural Comput.), 2019 Ryo Karakida S. Akaho S. Amari 176 32 0 14 Oct 2019
Harnessing the Power of Infinitely Wide Deep Nets on Small-data TasksInternational Conference on Learning Representations (ICLR), 2019 Sanjeev Arora S. Du Zhiyuan Li Ruslan Salakhutdinov Ruosong Wang Dingli Yu AAML 225 166 0 03 Oct 2019
Neural networks are a priori biased towards Boolean functions with low entropy Chris Mingard Joar Skalse Guillermo Valle Pérez David Martínez-Rubio Vladimir Mikulik A. Louis FAtt AI4CE 265 43 0 25 Sep 2019
Asymptotics of Wide Networks from Feynman DiagramsInternational Conference on Learning Representations (ICLR), 2019 Ethan Dyer Guy Gur-Ari 189 122 0 25 Sep 2019
Dynamics of Deep Neural Networks and Neural Tangent HierarchyInternational Conference on Machine Learning (ICML), 2019 Jiaoyang Huang H. Yau 147 160 0 18 Sep 2019
Finite Depth and Width Corrections to the Neural Tangent KernelInternational Conference on Learning Representations (ICLR), 2019 Boris Hanin Mihai Nica MDE 185 162 0 13 Sep 2019
A Fine-Grained Spectral Perspective on Neural Networks Greg Yang Hadi Salman 329 117 0 24 Jul 2019
Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts Arthur Jacot Franck Gabriel François Ged Clément Hongler 125 24 0 11 Jul 2019
Disentangling feature and lazy training in deep neural networks Mario Geiger S. Spigler Arthur Jacot Matthieu Wyart 238 17 0 19 Jun 2019
Kernel and Rich Regimes in Overparametrized ModelsAnnual Conference Computational Learning Theory (COLT), 2019 Blake E. Woodworth Suriya Gunasekar Pedro H. P. Savarese E. Moroshko Itay Golan Jason D. Lee Daniel Soudry Nathan Srebro 323 390 0 13 Jun 2019
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural NetworksNeural Information Processing Systems (NeurIPS), 2019 Ryo Karakida S. Akaho S. Amari 128 43 0 07 Jun 2019
Deep ReLU Networks Have Surprisingly Few Activation PatternsNeural Information Processing Systems (NeurIPS), 2019 Boris Hanin David Rolnick 386 247 0 03 Jun 2019
Exact Convergence Rates of the Neural Tangent Kernel in the Large Depth Limit Soufiane Hayou Arnaud Doucet Judith Rousseau 360 5 0 31 May 2019
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural NetworksNeural Information Processing Systems (NeurIPS), 2019 Yuan Cao Quanquan Gu MLT AI4CE 309 413 0 30 May 2019
Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph KernelsNeural Information Processing Systems (NeurIPS), 2019 S. Du Kangcheng Hou Barnabás Póczós Ruslan Salakhutdinov Ruosong Wang Keyulu Xu 324 298 0 30 May 2019
On the Inductive Bias of Neural Tangent KernelsNeural Information Processing Systems (NeurIPS), 2019 A. Bietti Julien Mairal 276 287 0 29 May 2019
Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems Tianle Cai Ruiqi Gao Jikai Hou Siyu Chen Dong Wang Di He Zhihua Zhang Liwei Wang ODL 156 60 0 28 May 2019
Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee Wei Hu Zhiyuan Li Dingli Yu NoLa 270 12 0 27 May 2019
What Can ResNet Learn Efficiently, Going Beyond Kernels?Neural Information Processing Systems (NeurIPS), 2019 Zeyuan Allen-Zhu Yuanzhi Li 596 192 0 24 May 2019