Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning

2 October 2018

Papers citing "Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning"

50 / 126 papers shown

Title
Heavy-Tailed Regularization of Weight Matrices in Deep Neural Networks Xuanzhe Xiao Zengyi Li Chuanlong Xie Fengwei Zhou 21 3 0 06 Apr 2023
Per-Example Gradient Regularization Improves Learning Signals from Noisy Data Xuran Meng Yuan Cao Difan Zou 25 5 0 31 Mar 2023
Deep Learning Weight Pruning with RMT-SVD: Increasing Accuracy and Reducing Overfitting Yitzchak Shmalo Jonathan Jenkins Oleksii Krupchytskyi 22 3 0 15 Mar 2023
Permutation Equivariant Neural Functionals Allan Zhou Kaien Yang Kaylee Burns Adriano Cardace Yiding Jiang Samuel Sokota J. Zico Kolter Chelsea Finn 20 47 0 27 Feb 2023
A rate of convergence when generating stable invariant Hermitian random matrix ensembles M. Kieburg Jiyuan Zhang 19 0 0 14 Feb 2023
Greedy Ordering of Layer Weight Matrices in Transformers Improves Translation Elicia Ye 21 1 0 04 Feb 2023
Spectral Evolution and Invariance in Linear-width Neural Networks Zhichao Wang A. Engel Anand D. Sarwate Ioana Dumitriu Tony Chiang 40 14 0 11 Nov 2022
Feature Reconstruction Attacks and Countermeasures of DNN training in Vertical Federated Learning Peng Ye Zhifeng Jiang Wei Wang Bo-wen Li Baochun Li AAML FedML 32 15 0 13 Oct 2022
Exploring Low Rank Training of Deep Neural Networks Siddhartha Rao Kamalakara Acyr F. Locatelli Bharat Venkitesh Jimmy Ba Y. Gal Aidan N. Gomez 48 22 0 27 Sep 2022
The SVD of Convolutional Weights: A CNN Interpretability Framework Brenda Praggastis Davis Brown Carlos Ortiz Marrero Emilie Purvine Madelyn Shapiro Bei Wang FAtt 27 9 0 14 Aug 2022
An Empirical Study of Implicit Regularization in Deep Offline RL Çağlar Gülçehre Srivatsan Srinivasan Jakub Sygnowski Georg Ostrovski Mehrdad Farajtabar Matt Hoffman Razvan Pascanu Arnaud Doucet OffRL 14 16 0 05 Jul 2022
Studying Generalization Through Data Averaging C. Gomez-Uribe FedML 19 0 0 28 Jun 2022
Deep Partial Least Squares for Empirical Asset Pricing M. Dixon Nicholas G. Polson Kemen Goicoechea 21 2 0 20 Jun 2022
Only Tails Matter: Average-Case Universality and Robustness in the Convex Regime Leonardo A Cunha Gauthier Gidel Fabian Pedregosa Damien Scieur Courtney Paquette 16 9 0 20 Jun 2022
Rank Diminishing in Deep Neural Networks Ruili Feng Kecheng Zheng Yukun Huang Deli Zhao Michael I. Jordan Zhengjun Zha 26 28 0 13 Jun 2022
Neural Collapse: A Review on Modelling Principles and Generalization Vignesh Kothapalli 21 71 0 08 Jun 2022
Boundary between noise and information applied to filtering neural network weight matrices Max Staats M. Thamm B. Rosenow 16 3 0 08 Jun 2022
Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations Steffen Schotthöfer Emanuele Zangrando J. Kusch Gianluca Ceruti Francesco Tudisco 53 30 0 26 May 2022
An Empirical Study of the Occurrence of Heavy-Tails in Training a ReLU Gate Sayar Karmakar Anirbit Mukherjee 16 0 0 26 Apr 2022
Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization Zhengqi Gao Sucheng Ren Zihui Xue Siting Li Hang Zhao 19 3 0 05 Apr 2022
Random matrix analysis of deep neural network weight matrices M. Thamm Max Staats B. Rosenow 27 12 0 28 Mar 2022
projUNN: efficient method for training deep networks with unitary matrices B. Kiani Randall Balestriero Yann LeCun S. Lloyd 36 32 0 10 Mar 2022
ANTLER: Bayesian Nonlinear Tensor Learning and Modeler for Unstructured, Varying-Size Point Cloud Data M. Biehler Hao Yan Jianjun Shi 3DPC 11 4 0 25 Feb 2022
Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data Yaoqing Yang Ryan Theisen Liam Hodgkinson Joseph E. Gonzalez Kannan Ramchandran Charles H. Martin Michael W. Mahoney 86 17 0 06 Feb 2022
A Generalized Weighted Optimization Method for Computational Learning and Inversion Bjorn Engquist Kui Ren Yunan Yang 21 4 0 23 Jan 2022
Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case L. Pastur 19 5 0 12 Jan 2022
Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs Inbar Seroussi Gadi Naveh Z. Ringel 27 49 0 31 Dec 2021
Learning from learning machines: a new generation of AI technology to meet the needs of science L. Pion-Tonachini K. Bouchard Héctor García Martín S. Peisert W. B. Holtz ... Rick L. Stevens Mark Anderson Ken Kreutz-Delgado Michael W. Mahoney James B. Brown 17 7 0 27 Nov 2021
Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stopping Xuran Meng Jianfeng Yao 17 7 0 26 Nov 2021
Pretrained Language Models are Symbolic Mathematics Solvers too! Kimia Noorbakhsh Modar Sulaiman M. Sharifi Kallol Roy Pooyan Jamshidi LRM 20 18 0 07 Oct 2021
Random matrices in service of ML footprint: ternary random features with no performance loss Hafiz Tiomoko Ali Zhenyu Liao Romain Couillet 36 7 0 05 Oct 2021
Shift-Curvature, SGD, and Generalization Arwen V. Bradley C. Gomez-Uribe Manish Reddy Vuyyuru 27 2 0 21 Aug 2021
Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers Liam Hodgkinson Umut Simsekli Rajiv Khanna Michael W. Mahoney 15 20 0 02 Aug 2021
Taxonomizing local versus global structure in neural network loss landscapes Yaoqing Yang Liam Hodgkinson Ryan Theisen Joe Zou Joseph E. Gonzalez K. Ramchandran Michael W. Mahoney 19 36 0 23 Jul 2021
Large Scale Private Learning via Low-rank Reparametrization Da Yu Huishuai Zhang Wei Chen Jian Yin Tie-Yan Liu 13 100 0 17 Jun 2021
A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs Gadi Naveh Z. Ringel SSL MLT 20 31 0 08 Jun 2021
Neural Monge Map estimation and its applications JiaoJiao Fan Shu Liu Shaojun Ma Haomin Zhou Yongxin Chen OT 22 23 0 07 Jun 2021
Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models Courtney Paquette Elliot Paquette ODL 16 13 0 07 Jun 2021
Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics Charles H. Martin Michael W. Mahoney 13 19 0 01 Jun 2021
Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression Zhefeng Qiao Xianghao Yu Jun Zhang Khaled B. Letaief FedML 33 19 0 26 Apr 2021
Pre-interpolation loss behaviour in neural networks Arthur E. W. Venter Marthinus W. Theunissen Marelie Hattingh Davel 11 3 0 14 Mar 2021
Hessian Eigenspectra of More Realistic Nonlinear Models Zhenyu Liao Michael W. Mahoney 12 30 0 02 Mar 2021
Local Tail Statistics of Heavy-Tailed Random Matrix Ensembles with Unitary Invariance M. Kieburg A. Monteleone 22 2 0 01 Mar 2021
SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality Courtney Paquette Kiwon Lee Fabian Pedregosa Elliot Paquette 9 32 0 08 Feb 2021
On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case L. Pastur V. Slavin CML 22 12 0 20 Nov 2020
Sparse Quantized Spectral Clustering Zhenyu Liao Romain Couillet Michael W. Mahoney MQ 14 15 0 03 Oct 2020
Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra V. Papyan 6 77 0 27 Aug 2020
Shape Matters: Understanding the Implicit Bias of the Noise Covariance Jeff Z. HaoChen Colin Wei J. Lee Tengyu Ma 18 93 0 15 Jun 2020
A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent Zhenyu Liao Romain Couillet Michael W. Mahoney 11 87 0 09 Jun 2020
Halting Time is Predictable for Large Models: A Universality Property and Average-case Analysis Courtney Paquette B. V. Merrienboer Elliot Paquette Fabian Pedregosa 24 25 0 08 Jun 2020