Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning

2 October 2018

Papers citing "Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning"

50 / 126 papers shown

Title
Bielik 11B v2 Technical Report Krzysztof Ociepa Łukasz Flis Krzysztof Wróbel Adrian Gwoździej Remigiusz Kinas 29 0 0 05 May 2025
Free Random Projection for In-Context Reinforcement Learning Tomohiro Hayase B. Collins Nakamasa Inoue 14 0 0 09 Apr 2025
Interpretable Machine Learning in Physics: A Review Sebastian Johann Wetzel Seungwoong Ha Raban Iten Miriam Klopotek Ziming Liu AI4CE 75 0 0 30 Mar 2025
A Priori Generalizability Estimate for a CNN Cito Balsells Beatrice Riviere David T. Fuentes 35 0 0 24 Feb 2025
LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning Zhekai Du Yinjie Min Jingjing Li Ke Lu Changliang Zou Liuhua Peng Tingjin Chu M. Gong 153 1 0 05 Feb 2025
Using Pre-trained LLMs for Multivariate Time Series Forecasting Malcolm Wolff Shenghao Yang Kari Torkkola Michael W. Mahoney AI4TS AIFin 43 1 0 10 Jan 2025
ExpTest: Automating Learning Rate Searching and Tuning with Insights from Linearized Neural Networks Zan Chaudhry Naoko Mizuno 74 0 0 25 Nov 2024
Evaluating Loss Landscapes from a Topology Perspective Tiankai Xie Caleb Geniesse Jiaqing Chen Yaoqing Yang Dmitriy Morozov Michael W. Mahoney Ross Maciejewski Gunther H. Weber 23 1 0 14 Nov 2024
A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities Yatin Dandi Luca Pesce Hugo Cui Florent Krzakala Yue M. Lu Bruno Loureiro MLT 37 1 0 24 Oct 2024
Estimating the Spectral Moments of the Kernel Integral Operator from Finite Sample Matrices Chanwoo Chun SueYeon Chung Daniel D. Lee 24 1 0 23 Oct 2024
How Does Data Diversity Shape the Weight Landscape of Neural Networks? Yang Ba M. Mancenido Rong Pan 19 0 0 18 Oct 2024
Model Balancing Helps Low-data Training and Fine-tuning Zihang Liu Y. Hu Tianyu Pang Yefan Zhou Pu Ren Yaoqing Yang 34 2 0 16 Oct 2024
AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models Haiquan Lu Yefan Zhou Shiwei Liu Zhangyang Wang Michael W. Mahoney Yaoqing Yang 23 0 0 14 Oct 2024
AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality Peijun Qing Chongyang Gao Yefan Zhou Xingjian Diao Yaoqing Yang Soroush Vosoughi MoMe MoE 24 3 0 14 Oct 2024
Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning Siyuan Li Juanxi Tian Zedong Wang Luyuan Zhang Zicheng Liu Weiyang Jin Yang Liu Baigui Sun Stan Z. Li 31 0 0 08 Oct 2024
Mitigating Memorization In Language Models Mansi Sakarvadia Aswathy Ajith Arham Khan Nathaniel Hudson Caleb Geniesse Kyle Chard Yaoqing Yang Ian Foster Michael W. Mahoney KELM MU 50 0 0 03 Oct 2024
Diff-INR: Generative Regularization for Electrical Impedance Tomography Bowen Tong Junwu Wang Dong Liu DiffM MedIm 35 2 0 06 Sep 2024
Connectivity structure and dynamics of nonlinear recurrent neural networks David G. Clark Owen Marschall Alexander van Meegen Ashok Litwin-Kumar 18 3 0 03 Sep 2024
Approaching Deep Learning through the Spectral Dynamics of Weights David Yunis Kumar Kshitij Patel Samuel Wheeler Pedro H. P. Savarese Gal Vardi Karen Livescu Michael Maire Matthew R. Walter 47 3 0 21 Aug 2024
Many Perception Tasks are Highly Redundant Functions of their Input Data Rahul Ramesh Anthony Bisulco Ronald W. DiTullio Linran Wei Vijay Balasubramanian Kostas Daniilidis Pratik Chaudhari 38 2 0 18 Jul 2024
Improving Hyperparameter Optimization with Checkpointed Model Weights Nikhil Mehta Jonathan Lorraine Steve Masson Ramanathan Arunachalam Zaid Pervaiz Bhat James Lucas Arun George Zachariah 41 4 0 26 Jun 2024
Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees Steffen Schotthöfer M. P. Laiu FedML 32 4 0 25 Jun 2024
MD tree: a model-diagnostic tree grown on loss landscape Yefan Zhou Jianlong Chen Qinxue Cao Konstantin Schürholt Yaoqing Yang 29 2 0 24 Jun 2024
Recent and Upcoming Developments in Randomized Numerical Linear Algebra for Machine Learning Michał Dereziński Michael W. Mahoney 20 5 0 17 Jun 2024
Towards Scalable and Versatile Weight Space Learning Konstantin Schurholt Michael W. Mahoney Damian Borth 40 15 0 14 Jun 2024
Learning Continually by Spectral Regularization Alex Lewandowski Saurabh Kumar Dale Schuurmans András Gyorgy Marlos C. Machado CLL 43 5 0 10 Jun 2024
Crafting Heavy-Tails in Weight Matrix Spectrum without Gradient Noise Vignesh Kothapalli Tianyu Pang Shenyang Deng Zongmin Liu Yaoqing Yang 32 3 0 07 Jun 2024
OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning Pengxiang Li Lu Yin Xiaowei Gao Shiwei Liu 26 7 0 28 May 2024
Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization Zhe Li Bicheng Ying Zidong Liu Haibo Yang Haibo Yang FedML 59 3 0 24 May 2024
The Impact of Geometric Complexity on Neural Collapse in Transfer Learning Michael Munn Benoit Dherin Javier Gonzalvo AAML 40 1 0 24 May 2024
Spectral Adapter: Fine-Tuning in Spectral Space Fangzhao Zhang Mert Pilanci 37 8 0 22 May 2024
Transferable and Principled Efficiency for Open-Vocabulary Segmentation Jingxuan Xu Wuyang Chen Yao-Min Zhao Yunchao Wei VLM 31 2 0 11 Apr 2024
A Survey on Error-Bounded Lossy Compression for Scientific Datasets Sheng Di Jinyang Liu Kai Zhao Xin Liang Robert Underwood ... Jon C. Calhoun Guanpeng Li Kazutomo Yoshii Khalid Ayed Alharthi Franck Cappello AI4CE 30 13 0 03 Apr 2024
Asymptotics of Learning with Deep Structured (Random) Features Dominik Schröder Daniil Dmitriev Hugo Cui Bruno Loureiro 48 6 0 21 Feb 2024
Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias Emanuele Zangrando Piero Deidda Simone Brugiapaglia Nicola Guglielmi Francesco Tudisco 27 8 0 06 Feb 2024
Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective Haoyi Xiong Xuhong Li Xiaofei Zhang Jiamin Chen Xinhao Sun Yuchen Li Zeyi Sun Mengnan Du XAI 35 8 0 09 Jan 2024
FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications Grant Wilkins Sheng Di Jon C. Calhoun Zilinghan Li Kibaek Kim Robert Underwood Richard Mortier Franck Cappello FedML 37 2 0 20 Dec 2023
Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training Yefan Zhou Tianyu Pang Keqin Liu Charles H. Martin Michael W. Mahoney Yaoqing Yang 34 7 0 01 Dec 2023
GloNets: Globally Connected Neural Networks Antonio Di Cecco C. Metta M. Fantozzi F. Morandin Maurizio Parton 13 2 0 27 Nov 2023
An effective theory of collective deep learning Lluís Arola-Fernández Lucas Lacasa FedML AI4CE 18 2 0 19 Oct 2023
LASER: Linear Compression in Wireless Distributed Optimization Ashok Vardhan Makkuva Marco Bondaschi Thijs Vogels Martin Jaggi Hyeji Kim Michael C. Gastpar 74 3 0 19 Oct 2023
Enhancing Accuracy in Deep Learning Using Random Matrix Theory Leonid Berlyand Etienne Sandier Yitzchak Shmalo Lei Zhang AAML 23 0 0 04 Oct 2023
Low Tensor Rank Learning of Neural Dynamics Arthur Pellegrino Alex Cayco-Gajic Angus Chadwick 19 5 0 22 Aug 2023
Dense Sample Deep Learning Stephen Jose Hanson Vivek Yadav C. Hanson 10 2 0 20 Jul 2023
The Interpolating Information Criterion for Overparameterized Models Liam Hodgkinson Christopher van der Heide Roberto Salomone Fred Roosta Michael W. Mahoney 16 7 0 15 Jul 2023
Robust low-rank training via approximate orthonormal constraints Dayana Savostianova Emanuele Zangrando Gianluca Ceruti Francesco Tudisco 24 9 0 02 Jun 2023
A Rainbow in Deep Network Black Boxes Florentin Guth Brice Ménard G. Rochette S. Mallat 17 10 0 29 May 2023
A Three-regime Model of Network Pruning Yefan Zhou Yaoqing Yang Arin Chang Michael W. Mahoney 24 10 0 28 May 2023
Neural Functional Transformers Allan Zhou Kaien Yang Yiding Jiang Kaylee Burns Winnie Xu Samuel Sokota J. Zico Kolter Chelsea Finn 16 31 0 22 May 2023
Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspective Yuxin Dong Tieliang Gong H. Chen Chen Li 15 4 0 02 May 2023