v1v2 (latest)

On the loss landscape of a class of deep neural networks with no bad local valleys

27 September 2018

Quynh N. Nguyen

Mahesh Chandra Mukkamala

Matthias Hein

ArXiv (abs)PDF HTML

Papers citing "On the loss landscape of a class of deep neural networks with no bad local valleys"

50 / 59 papers shown

Title
Analyzing the Role of Permutation Invariance in Linear Mode Connectivity Keyao Zhan Puheng Li Lei Wu MoMe 173 0 0 13 Mar 2025
Geometrical structures of digital fluctuations in parameter space of neural networks trained with adaptive momentum optimization Igor V. Netay 107 2 0 22 Aug 2024
Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks Yichu Xu Xin-Chun Li Lan Li De-Chuan Zhan 166 2 0 21 May 2024
Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity Zhanran Lin Puheng Li Lei Wu 327 9 0 09 Apr 2024
Merging Text Transformer Models from Different Initializations Neha Verma Maha Elbayad MoMe 189 10 0 01 Mar 2024
On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm Zhanpeng Zhou Zijun Chen Yilan Chen Bo Zhang Junchi Yan MoMe 165 17 0 06 Feb 2024
Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods Akira Ito Masanori Yamada Atsutoshi Kumagai MoMe 348 7 0 06 Feb 2024
Deep Model Fusion: A Survey Weishi Li Yong Peng Miao Zhang Liang Ding Han Hu Li Shen FedML MoMe 185 73 0 27 Sep 2023
Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization Method Mohannad Elhamod Anuj Karpatne 105 3 0 26 Sep 2023
Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity Zhanpeng Zhou Yongyi Yang Xiaojiang Yang Junchi Yan Wei Hu 194 39 0 17 Jul 2023
Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning Libin Zhu Chaoyue Liu Adityanarayanan Radhakrishnan M. Belkin 198 21 0 07 Jun 2023
Subspace-Configurable Networks Dong Wang O. Saukh Xiaoxi He Lothar Thiele OOD 165 0 0 22 May 2023
Mechanistic Mode Connectivity Ekdeep Singh Lubana Eric J. Bigelow Robert P. Dick David M. Krueger Hidenori Tanaka 154 51 0 15 Nov 2022
REPAIR: REnormalizing Permuted Activations for Interpolation Repair Keller Jordan Hanie Sedghi O. Saukh R. Entezari Behnam Neyshabur MoMe 270 105 0 15 Nov 2022
When Expressivity Meets Trainability: Fewer than $n$ Neurons Can Work Jiawei Zhang Yushun Zhang Mingyi Hong Ruoyu Sun Jianfeng Yao 179 10 0 21 Oct 2022
Boosting vision transformers for image retrieval Chull Hwan Song Jooyoung Yoon Shunghyun Choi Yannis Avrithis ViT 165 35 0 21 Oct 2022
Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networks A. K. Akash Sixu Li Nicolas García Trillos 118 13 0 13 Oct 2022
On the generalization of learning algorithms that do not converge N. Chandramoorthy Andreas Loukas Khashayar Gatmiry Stefanie Jegelka MLT 176 12 0 16 Aug 2022
PRANC: Pseudo RAndom Networks for Compacting deep models Parsa Nooralinejad Ali Abbasi Soroush Abbasi Koohpayegani Kossar Pourahmadi Meibodi Rana Muhammad Shahroz Khan Soheil Kolouri Hamed Pirsiavash DD 146 0 0 16 Jun 2022
Deep learning, stochastic gradient descent and diffusion maps Carmina Fjellström Kaj Nyström DiffM 154 17 0 04 Apr 2022
On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems C. Christof Julia Kowalczyk 148 9 0 23 Feb 2022
Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer Kun Chen Dachao Lin Zhihua Zhang 92 1 0 08 Jan 2022
Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks Shaun Li AI4CE 126 1 0 03 Jan 2022
Theoretical Exploration of Flexible Transmitter Model Jin-Hui Wu Shao-Qun Zhang Yuan Jiang Zhiping Zhou 180 3 0 11 Nov 2021
What Happens after SGD Reaches Zero Loss? --A Mathematical Framework Zhiyuan Li Tianhao Wang Sanjeev Arora MLT 233 108 0 13 Oct 2021
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks R. Entezari Hanie Sedghi O. Saukh Behnam Neyshabur MoMe 216 252 0 12 Oct 2021
What training reveals about neural network complexity Andreas Loukas Marinos Poiitis Stefanie Jegelka 133 12 0 08 Jun 2021
Achieving Small Test Error in Mildly Overparameterized Neural Networks Shiyu Liang Ruoyu Sun R. Srikant 109 3 0 24 Apr 2021
A Gradient Estimator for Time-Varying Electrical Networks with Non-Linear Dissipation Jack D. Kendall 87 8 0 09 Mar 2021
Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations Bo Liu 106 8 0 25 Feb 2021
WGAN with an Infinitely Wide Generator Has No Spurious Stationary Points Albert No Taeho Yoon Sehyun Kwon Ernest K. Ryu GAN 114 2 0 15 Feb 2021
Neural Networks with Complex-Valued Weights Have No Spurious Local MinimaAnnual Conference on Information Sciences and Systems (CISS), 2025 Xingtu Liu MLT 96 0 0 31 Jan 2021
A Note on Connectivity of Sublevel Sets in Deep Learning Quynh N. Nguyen MLT 170 8 0 21 Jan 2021
Recent advances in deep learning theory Fengxiang He Dacheng Tao AI4CE 189 53 0 20 Dec 2020
Recent Theoretical Advances in Non-Convex Optimization Marina Danilova Pavel Dvurechensky Alexander Gasnikov Eduard A. Gorbunov Sergey Guminov Dmitry Kamzolov Innokentiy Shibaev 185 90 0 11 Dec 2020
Towards a Better Global Loss Landscape of GANs Ruoyu Sun Tiantian Fang Alex Schwing GAN 107 27 0 10 Nov 2020
The Global Landscape of Neural Networks: An Overview Ruoyu Sun Dawei Li Shiyu Liang Tian Ding R. Srikant 138 90 0 02 Jul 2020
Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets Haoxiang Wang Ruoyu Sun Bo Li MLT AI4CE 145 14 0 25 Jun 2020
The critical locus of overparameterized neural networks Y. Cooper UQCV 123 10 0 08 May 2020
Piecewise linear activations substantially shape the loss surfaces of neural networks Fengxiang He Bohan Wang Dacheng Tao ODL 134 31 0 27 Mar 2020
Some Geometrical and Topological Properties of DNNs' Decision Boundaries Bo Liu Mengya Shen AAML 136 3 0 07 Mar 2020
Loss landscapes and optimization in over-parameterized non-linear systems and neural networks Chaoyue Liu Libin Zhu M. Belkin ODL 192 280 0 29 Feb 2020
Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology Quynh N. Nguyen Marco Mondelli ODL AI4CE 207 75 0 18 Feb 2020
Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory Bo Liu FAtt MLT 153 1 0 12 Feb 2020
Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity Shiyu Liang Ruoyu Sun R. Srikant 120 21 0 31 Dec 2019
Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks Aleksandr Shevchenko Marco Mondelli 282 38 0 20 Dec 2019
Optimization for deep learning: theory and algorithms Ruoyu Sun ODL 171 171 0 19 Dec 2019
Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations Tian Ding Dawei Li Ruoyu Sun 189 14 0 04 Nov 2019
Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory Micah Goldblum Jonas Geiping Avi Schwarzschild Michael Moeller Tom Goldstein 271 36 0 01 Oct 2019
Convergence Rates of Variational Inference in Sparse Deep Learning Badr-Eddine Chérief-Abdellatif BDL 180 39 0 09 Aug 2019