Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning

2 October 2018

Papers citing "Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning"

26 / 126 papers shown

Title
Achieving Online Regression Performance of LSTMs with Simple RNNs Nuri Mert Vural Fatih Ilhan Selim F. Yilmaz Salih Ergüt Suleyman Serdar Kozat 6 13 0 16 May 2020
Optimization in Machine Learning: A Distribution Space Approach Yongqiang Cai Qianxiao Li Zuowei Shen 15 1 0 18 Apr 2020
The Implicit Regularization of Stochastic Gradient Flow for Least Squares Alnur Ali Edgar Dobriban R. Tibshirani 14 76 0 17 Mar 2020
On Alignment in Deep Linear Neural Networks Adityanarayanan Radhakrishnan Eshaan Nichani D. Bernstein Caroline Uhler 9 2 0 13 Mar 2020
Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data Charles H. Martin Tongsu Peng Peng Michael W. Mahoney 17 101 0 17 Feb 2020
Average-case Acceleration Through Spectral Density Estimation Fabian Pedregosa Damien Scieur 9 12 0 12 Feb 2020
Regularized Evolutionary Population-Based Training J. Liang Santiago Gonzalez H. Shahrzad Risto Miikkulainen 6 9 0 11 Feb 2020
Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters Ziming Zhang Wenchi Ma Yuanwei Wu Guanghui Wang 32 10 0 05 Jan 2020
Exact expressions for double descent and implicit regularization via surrogate random design Michal Derezinski Feynman T. Liang Michael W. Mahoney 11 77 0 10 Dec 2019
Implicit Regularization and Convergence for Weight Normalization Xiaoxia Wu Edgar Dobriban Tongzheng Ren Shanshan Wu Zhiyuan Li Suriya Gunasekar Rachel A. Ward Qiang Liu 20 21 0 18 Nov 2019
Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search Mehmet Süzen J. Cerdà C. Weber 11 1 0 10 Nov 2019
Detecting Underspecification with Local Ensembles David Madras James Atwood Alexander DÁmour 28 4 0 21 Oct 2019
Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network Taiji Suzuki Hiroshi Abe Tomoaki Nishimura AI4CE 12 43 0 25 Sep 2019
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization Thijs Vogels Sai Praneeth Karimireddy Martin Jaggi 17 316 0 31 May 2019
On Dropout and Nuclear Norm Regularization Poorya Mianjy R. Arora 19 23 0 28 May 2019
HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning O. Gencoglu M. Gils E. Guldogan Chamin Morikawa Mehmet Süzen M. Gruber J. Leinonen H. Huttunen 11 36 0 16 Apr 2019
Deep Fundamental Factor Models M. Dixon Nicholas G. Polson 26 9 0 18 Mar 2019
Inefficiency of K-FAC for Large Batch Size Training Linjian Ma Gabe Montague Jiayu Ye Z. Yao A. Gholami Kurt Keutzer Michael W. Mahoney 16 24 0 14 Mar 2019
Combining learning rate decay and weight decay with complexity gradient descent - Part I Pierre Harvey Richemond Yike Guo 17 4 0 07 Feb 2019
Cross-Entropy Loss and Low-Rank Features Have Responsibility for Adversarial Examples Kamil Nar Orhan Ocal S. Shankar Sastry K. Ramchandran AAML 11 54 0 24 Jan 2019
Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks Charles H. Martin Michael W. Mahoney 11 55 0 24 Jan 2019
Traditional and Heavy-Tailed Self Regularization in Neural Network Models Charles H. Martin Michael W. Mahoney 18 119 0 24 Jan 2019
Cleaning large correlation matrices: tools from random matrix theory J. Bun J. Bouchaud M. Potters 27 262 0 25 Oct 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 278 2,888 0 15 Sep 2016
Norm-Based Capacity Control in Neural Networks Behnam Neyshabur Ryota Tomioka Nathan Srebro 116 577 0 27 Feb 2015
The Loss Surfaces of Multilayer Networks A. Choromańska Mikael Henaff Michaël Mathieu Gerard Ben Arous Yann LeCun ODL 179 1,185 0 30 Nov 2014