Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

25 May 2021

Papers citing "Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances"

28 / 28 papers shown

Title
Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry Mohammed Adnan Rohan Jain Ekansh Sharma Rahul Krishnan Yani Andrew Ioannou 56 0 0 08 May 2025
Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networks Devon Jarvis Richard Klein Benjamin Rosman Andrew M. Saxe MLT 66 1 0 08 Mar 2025
Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries Chris Kolb T. Weber Bernd Bischl David Rügamer 113 0 0 04 Feb 2025
Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence Berfin Simsek Amire Bendjeddou Daniel Hsu 44 0 0 13 Nov 2024
Input Space Mode Connectivity in Deep Neural Networks Jakub Vrabel Ori Shem-Ur Yaron Oz David Krueger 56 1 0 09 Sep 2024
Remove Symmetries to Control Model Expressivity and Improve Optimization Liu Ziyin Yizhou Xu Isaac Chuang AAML 38 1 0 28 Aug 2024
Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completion Zhiwei Bai Jiajie Zhao Yaoyu Zhang AI4CE 37 0 0 22 May 2024
Merging Text Transformer Models from Different Initializations Neha Verma Maha Elbayad MoMe 59 7 0 01 Mar 2024
Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding Zhengqing Wu Berfin Simsek Francois Ged ODL 42 0 0 08 Feb 2024
Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization Leyang Zhang Yaoyu Zhang Tao Luo 20 2 0 01 Sep 2023
Layer-wise Linear Mode Connectivity Linara Adilova Maksym Andriushchenko Michael Kamp Asja Fischer Martin Jaggi FedML FAtt MoMe 33 15 0 13 Jul 2023
Proximity to Losslessly Compressible Parameters Matthew Farrugia-Roberts 30 0 0 05 Jun 2023
Improving Convergence and Generalization Using Parameter Symmetries Bo-Lu Zhao Robert Mansel Gower Robin G. Walters Rose Yu MoMe 30 13 0 22 May 2023
Understanding the Initial Condensation of Convolutional Neural Networks Zhangchen Zhou Hanxu Zhou Yuqing Li Zhi-Qin John Xu MLT AI4CE 26 5 0 17 May 2023
Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks Matthew Farrugia-Roberts 24 4 0 08 May 2023
Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent Liu Ziyin Botao Li Tomer Galanti Masakuni Ueda 37 7 0 23 Mar 2023
Equivariant Architectures for Learning in Deep Weight Spaces Aviv Navon Aviv Shamsian Idan Achituve Ethan Fetaya Gal Chechik Haggai Maron 47 63 0 30 Jan 2023
REPAIR: REnormalizing Permuted Activations for Interpolation Repair Keller Jordan Hanie Sedghi O. Saukh R. Entezari Behnam Neyshabur MoMe 46 94 0 15 Nov 2022
Symmetries, flat minima, and the conserved quantities of gradient flow Bo-Lu Zhao I. Ganev Robin G. Walters Rose Yu Nima Dehmamy 47 16 0 31 Oct 2022
Deep Double Descent via Smooth Interpolation Matteo Gamba Erik Englesson Marten Bjorkman Hossein Azizpour 63 10 0 21 Sep 2022
Random initialisations performing above chance and how to find them Frederik Benzing Simon Schug Robert Meier J. Oswald Yassir Akram Nicolas Zucchet Laurence Aitchison Angelika Steger ODL 35 24 0 15 Sep 2022
Git Re-Basin: Merging Models modulo Permutation Symmetries Samuel K. Ainsworth J. Hayase S. Srinivasa MoMe 255 314 0 11 Sep 2022
Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks Zhiwei Bai Tao Luo Z. Xu Yaoyu Zhang 31 4 0 26 May 2022
Symmetry Teleportation for Accelerated Optimization B. Zhao Nima Dehmamy Robin G. Walters Rose Yu ODL 23 20 0 21 May 2022
Improving Chest X-Ray Report Generation by Leveraging Warm Starting Aaron Nicolson Jason Dowling Bevan Koopman ViT LM&MA MedIm 30 90 0 24 Jan 2022
Embedding Principle: a hierarchical structure of loss landscape of deep neural networks Yaoyu Zhang Yuqing Li Zhongwang Zhang Tao Luo Z. Xu 26 21 0 30 Nov 2021
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks R. Entezari Hanie Sedghi O. Saukh Behnam Neyshabur MoMe 37 216 0 12 Oct 2021
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics D. Kunin Javier Sagastuy-Breña Surya Ganguli Daniel L. K. Yamins Hidenori Tanaka 107 77 0 08 Dec 2020