Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape

5 July 2019

Papers citing "Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape"

43 / 43 papers shown

Title
Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion Binchi Zhang Zaiyi Zheng Zhengzhang Chen Wenlin Yao 59 0 0 01 Feb 2025
Reinforcement Teaching Alex Lewandowski Calarina Muslimani Dale Schuurmans Matthew E. Taylor Jun Luo 81 1 0 28 Jan 2025
Weight Scope Alignment: A Frustratingly Easy Method for Model Merging Yichu Xu Xin-Chun Li Le Gan De-Chuan Zhan MoMe 40 0 0 22 Aug 2024
Approaching Deep Learning through the Spectral Dynamics of Weights David Yunis Kumar Kshitij Patel Samuel Wheeler Pedro H. P. Savarese Gal Vardi Karen Livescu Michael Maire Matthew R. Walter 52 3 0 21 Aug 2024
Coding schemes in neural networks learning classification tasks Alexander van Meegen H. Sompolinsky 36 6 0 24 Jun 2024
Continual Learning with Weight Interpolation Jkedrzej Kozal Jan Wasilewski Bartosz Krawczyk Michal Wo'zniak CLL MoMe 34 6 0 05 Apr 2024
Do Deep Neural Network Solutions Form a Star Domain? Ankit Sonthalia Alexander Rubinstein Ehsan Abbasnejad Seong Joon Oh MoMe 433 2 5 12 Mar 2024
Training-Free Pretrained Model Merging Zhenxing Xu Ke Yuan Huiqiong Wang Yong Wang Mingli Song Jie Song MoMe 32 15 0 04 Mar 2024
Training Neural Networks from Scratch with Parallel Low-Rank Adapters Minyoung Huh Brian Cheung Jeremy Bernstein Phillip Isola Pulkit Agrawal 35 10 0 26 Feb 2024
Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODE Koji Hashimoto Yuji Hirono Akiyoshi Sannai AI4CE 32 7 0 04 Feb 2024
A Compact Representation for Bayesian Neural Networks By Removing Permutation Symmetry Tim Z. Xiao Weiyang Liu Robert Bamler 31 5 0 31 Dec 2023
How to Train Neural Field Representations: A Comprehensive Study and Benchmark Samuele Papa Riccardo Valperga David M. Knigge Miltiadis Kofinas Phillip Lippe J. Sonke E. Gavves 27 7 0 16 Dec 2023
Scrap Your Schedules with PopDescent Abhinav Pomalapally B. Mabsout Renato Mansuco 15 0 0 23 Oct 2023
A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors Olivier Laurent Emanuel Aldea Gianni Franchi BDL UQCV 22 5 0 12 Oct 2023
Robot Fleet Learning via Policy Merging Lirui Wang Kaiqing Zhang Allan Zhou Max Simchowitz Russ Tedrake 42 4 0 02 Oct 2023
Deep Model Fusion: A Survey Weishi Li Yong Peng Miao Zhang Liang Ding Han Hu Li Shen FedML MoMe 30 51 0 27 Sep 2023
Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks Dominik Schnaus Jongseok Lee Daniel Cremers Rudolph Triebel UQCV BDL 38 1 0 15 Jul 2023
Hidden symmetries of ReLU networks J. E. Grigsby Kathryn A. Lindsey David Rolnick 24 21 0 09 Jun 2023
Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks F. Chen D. Kunin Atsushi Yamamura Surya Ganguli 23 26 0 07 Jun 2023
Neural Functional Transformers Allan Zhou Kaien Yang Yiding Jiang Kaylee Burns Winnie Xu Samuel Sokota J. Zico Kolter Chelsea Finn 21 31 0 22 May 2023
Exploring the Complexity of Deep Neural Networks through Functional Equivalence Guohao Shen 24 2 0 19 May 2023
Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks Matthew Farrugia-Roberts 24 4 0 08 May 2023
Permutation Equivariant Neural Functionals Allan Zhou Kaien Yang Kaylee Burns Adriano Cardace Yiding Jiang Samuel Sokota J. Zico Kolter Chelsea Finn 35 47 0 27 Feb 2023
Equivariant Architectures for Learning in Deep Weight Spaces Aviv Navon Aviv Shamsian Idan Achituve Ethan Fetaya Gal Chechik Haggai Maron 41 63 0 30 Jan 2023
Re-basin via implicit Sinkhorn differentiation F. Guerrero-Peña H. R. Medeiros Thomas Dubail Masih Aminbeidokhti Eric Granger M. Pedersoli MoMe 20 44 0 22 Dec 2022
REPAIR: REnormalizing Permuted Activations for Interpolation Repair Keller Jordan Hanie Sedghi O. Saukh R. Entezari Behnam Neyshabur MoMe 46 94 0 15 Nov 2022
Random initialisations performing above chance and how to find them Frederik Benzing Simon Schug Robert Meier J. Oswald Yassir Akram Nicolas Zucchet Laurence Aitchison Angelika Steger ODL 35 24 0 15 Sep 2022
On the detrimental effect of invariances in the likelihood for variational inference Richard Kurle R. Herbrich Tim Januschowski Bernie Wang Jan Gasthaus 19 9 0 15 Sep 2022
Git Re-Basin: Merging Models modulo Permutation Symmetries Samuel K. Ainsworth J. Hayase S. Srinivasa MoMe 252 314 0 11 Sep 2022
On the Symmetries of Deep Learning Models and their Internal Representations Charles Godfrey Davis Brown Tegan H. Emerson Henry Kvinge 22 40 0 27 May 2022
Aligned Weight Regularizers for Pruning Pretrained Neural Networks J. Ó. Neill Sourav Dutta H. Assem VLM 11 2 0 04 Apr 2022
Optimal learning rate schedules in high-dimensional non-convex optimization problems Stéphane dÁscoli Maria Refinetti Giulio Biroli 16 7 0 09 Feb 2022
Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry Fabrizio Pittorino Antonio Ferraro Gabriele Perugini Christoph Feinauer Carlo Baldassi R. Zecchina 201 24 0 07 Feb 2022
The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks R. Entezari Hanie Sedghi O. Saukh Behnam Neyshabur MoMe 37 215 0 12 Oct 2021
Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances Berfin cSimcsek François Ged Arthur Jacot Francesco Spadaro Clément Hongler W. Gerstner Johanni Brea AI4CE 28 91 0 25 May 2021
Priors in Bayesian Deep Learning: A Review Vincent Fortuin UQCV BDL 31 124 0 14 May 2021
When Are Solutions Connected in Deep Networks? Quynh N. Nguyen Pierre Bréchet Marco Mondelli 27 9 0 18 Feb 2021
GENNI: Visualising the Geometry of Equivalences for Neural Network Identifiability Daniel Lengyel Janith C. Petangoda Isak Falk Kate Highnam Michalis Lazarou A. Kolbeinsson M. Deisenroth N. Jennings 9 4 0 14 Nov 2020
Optimizing Mode Connectivity via Neuron Alignment N. Joseph Tatro Pin-Yu Chen Payel Das Igor Melnyk P. Sattigeri Rongjie Lai MoMe 223 80 0 05 Sep 2020
Data-driven effective model shows a liquid-like deep learning Wenxuan Zou Haiping Huang 24 2 0 16 Jul 2020
On the Principle of Least Symmetry Breaking in Shallow ReLU Models Yossi Arjevani M. Field 26 7 0 26 Dec 2019
Challenges in Markov chain Monte Carlo for Bayesian neural networks Theodore Papamarkou Jacob D. Hinkle M. T. Young D. Womble BDL 36 50 0 15 Oct 2019
The Loss Surfaces of Multilayer Networks A. Choromańska Mikael Henaff Michaël Mathieu Gerard Ben Arous Yann LeCun ODL 179 1,185 0 30 Nov 2014