On the Implicit Bias in Deep-Learning Algorithms

26 August 2022

Gal Vardi

Papers citing "On the Implicit Bias in Deep-Learning Algorithms"

50 / 57 papers shown

Title
How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias Ruiquan Huang Yingbin Liang Jing Yang 43 0 0 02 May 2025
Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networks Chenyang Zhang Peifeng Gao Difan Zou Yuan Cao OOD MLT 57 0 0 11 Apr 2025
Architecture independent generalization bounds for overparametrized deep ReLU networks Thomas Chen Chun-Kai Kevin Chien Patrícia Muñoz Ewald Andrew G. Moore 18 0 0 08 Apr 2025
Towards Understanding the Benefits of Neural Network Parameterizations in Geophysical Inversions: A Study With Neural Fields Anran Xu L. Heagy 39 0 0 21 Mar 2025
Evolutionary Prediction Games Eden Saig Nir Rosenfeld 44 0 0 05 Mar 2025
Low-rank bias, weight decay, and model merging in neural networks Ilja Kuzborskij Yasin Abbasi-Yadkori 42 0 0 24 Feb 2025
The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networks Sholom Schechtman Nicolas Schreuder 45 0 0 08 Feb 2025
Optimization Insights into Deep Diagonal Linear Networks Hippolyte Labarrière C. Molinari Lorenzo Rosasco S. Villa Cristian Vega 66 0 0 21 Dec 2024
Slowing Down Forgetting in Continual Learning Pascal Janetzky Tobias Schlagenhauf Stefan Feuerriegel CLL 19 0 0 11 Nov 2024
The Implicit Bias of Gradient Descent on Separable Multiclass Data Hrithik Ravi Clayton Scott Daniel Soudry Yutong Wang 25 2 0 02 Nov 2024
Rethinking generalization of classifiers in separable classes scenarios and over-parameterized regimes Julius Martinetz C. Linse Thomas Martinetz 18 0 0 22 Oct 2024
Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training Zhanpeng Zhou Mingze Wang Yuchen Mao Bingrui Li Junchi Yan AAML 55 0 0 14 Oct 2024
Fast Training of Sinusoidal Neural Fields via Scaling Initialization Taesun Yeom Sangyoon Lee Jaeho Lee 42 2 0 07 Oct 2024
Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context Spencer Frei Gal Vardi MLT 21 3 0 02 Oct 2024
Non-asymptotic Convergence of Training Transformers for Next-token Prediction Ruiquan Huang Yingbin Liang Jing Yang 16 5 0 25 Sep 2024
Probing Implicit Bias in Semi-gradient Q-learning: Visualizing the Effective Loss Landscapes via the Fokker--Planck Equation Shuyu Yin Fei Wen Peilin Liu Tao Luo 27 0 0 12 Jun 2024
The Price of Implicit Bias in Adversarially Robust Generalization Nikolaos Tsilivis Natalie Frank Nathan Srebro Julia Kempe 35 1 0 07 Jun 2024
Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation Can Yaras Peng Wang Laura Balzano Qing Qu AI4CE 27 12 0 06 Jun 2024
Improving Generalization and Convergence by Enhancing Implicit Regularization Mingze Wang Haotian He Jinbo Wang Zilin Wang Guanhua Huang Feiyu Xiong Zhiyu Li E. Weinan Lei Wu 29 6 0 31 May 2024
Information-Theoretic Generalization Bounds for Deep Neural Networks Haiyun He Christina Lee Yu 30 4 0 04 Apr 2024
Neural Redshift: Random Networks are not Random Functions Damien Teney A. Nicolicioiu Valentin Hartmann Ehsan Abbasnejad 86 18 0 04 Mar 2024
Causal hybrid modeling with double machine learning Kai-Hendrik Cohrs Gherardo Varando Nuno Carvalhais Markus Reichstein Gustau Camps-Valls 19 3 0 20 Feb 2024
Depth Separation in Norm-Bounded Infinite-Width Neural Networks Suzanna Parkinson Greg Ongie Rebecca Willett Ohad Shamir Nathan Srebro MDE 35 2 0 13 Feb 2024
How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers G. Buzaglo I. Harel Mor Shpigel Nacson Alon Brutzkus Nathan Srebro Daniel Soudry 50 3 0 09 Feb 2024
Implicit Bias and Fast Convergence Rates for Self-attention Bhavya Vasudeva Puneesh Deora Christos Thrampoulidis 24 13 0 08 Feb 2024
An extended asymmetric sigmoid with Perceptron (SIGTRON) for imbalanced linear classification Hyenkyun Woo 8 0 0 26 Dec 2023
Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescaling Mingze Wang Zeping Min Lei Wu 17 3 0 24 Nov 2023
Feature emergence via margin maximization: case studies in algebraic tasks Depen Morwani Benjamin L. Edelman Costin-Andrei Oncescu Rosie Zhao Sham Kakade 23 7 0 13 Nov 2023
Optimization dependent generalization bound for ReLU networks based on sensitivity in the tangent bundle Dániel Rácz M. Petreczky András Csertán Bálint Daróczy MLT 8 1 0 26 Oct 2023
Implicit regularization of deep residual networks towards neural ODEs P. Marion Yu-Han Wu Michael E. Sander Gérard Biau 17 14 0 03 Sep 2023
Antagonising explanation and revealing bias directly through sequencing and multimodal inference Luís Arandas Mick Grierson Miguel Carvalhais PINN DiffM 6 1 0 25 Aug 2023
The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning Nikhil Ghosh Spencer Frei Wooseok Ha Ting Yu MLT 15 3 0 06 Aug 2023
Noisy Interpolation Learning with Shallow Univariate ReLU Networks Nirmit Joshi Gal Vardi Nathan Srebro 16 8 0 28 Jul 2023
Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory Minhak Song Chulhee Yun 18 9 1 09 Jul 2023
Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs D. Chistikov Matthias Englert R. Lazic MLT 27 12 0 10 Jun 2023
Faster Margin Maximization Rates for Generic and Adversarially Robust Optimization Methods Guanghui Wang Zihao Hu Claudio Gentile Vidya Muthukumar Jacob D. Abernethy 17 0 0 27 May 2023
The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent Lei Wu Weijie J. Su MLT 17 15 0 27 May 2023
From Tempered to Benign Overfitting in ReLU Neural Networks Guy Kornowski Gilad Yehudai Ohad Shamir 18 12 0 24 May 2023
On the Implicit Bias of Linear Equivariant Steerable Networks Ziyu Chen Wei-wei Zhu 10 3 0 07 Mar 2023
Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization Spencer Frei Gal Vardi Peter L. Bartlett Nathan Srebro 24 22 0 02 Mar 2023
The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networks Spencer Frei Gal Vardi Peter L. Bartlett Nathan Srebro 19 12 0 02 Mar 2023
Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regression Mo Zhou Rong Ge 11 2 0 01 Feb 2023
Neural networks trained with SGD learn distributions of increasing complexity Maria Refinetti Alessandro Ingrosso Sebastian Goldt UQCV 17 40 0 21 Nov 2022
Do highly over-parameterized neural networks generalize since bad solutions are rare? Julius Martinetz T. Martinetz 11 1 0 07 Nov 2022
Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data Spencer Frei Gal Vardi Peter L. Bartlett Nathan Srebro Wei Hu MLT 13 38 0 13 Oct 2022
Reconstructing Training Data from Trained Neural Networks Niv Haim Gal Vardi Gilad Yehudai Ohad Shamir Michal Irani 13 130 0 15 Jun 2022
Understanding Gradient Descent on Edge of Stability in Deep Learning Sanjeev Arora Zhiyuan Li A. Panigrahi MLT 72 88 0 19 May 2022
On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias Itay Safran Gal Vardi Jason D. Lee MLT 37 23 0 18 May 2022
Gradient Methods Provably Converge to Non-Robust Networks Gal Vardi Gilad Yehudai Ohad Shamir 17 27 0 09 Feb 2022
Implicit Regularization Towards Rank Minimization in ReLU Networks Nadav Timor Gal Vardi Ohad Shamir 15 49 0 30 Jan 2022