Merging Text Transformer Models from Different Initializations

1 March 2024

Papers citing "Merging Text Transformer Models from Different Initializations"

13 / 13 papers shown

Title
Wormhole Memory: A Rubik's Cube for Cross-Dialogue Retrieval Libo Wang 105 0 0 24 Jan 2025
Merging Feed-Forward Sublayers for Compressed Transformers Neha Verma Kenton W. Murray Kevin Duh AI4CE 50 0 0 10 Jan 2025
Training-free Heterogeneous Model Merging Zhengqi Xu Han Zheng Jie Song Li Sun Mingli Song MoMe 68 1 0 03 Jan 2025
Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild Xinyu Zhao Guoheng Sun Ruisi Cai Yukun Zhou Pingzhi Li ... Binhang Yuan Hongyi Wang Ang Li Zhangyang Wang Tianlong Chen MoMe ALM 28 2 0 07 Oct 2024
Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks Edan Kinderman Itay Hubara Haggai Maron Daniel Soudry MoMe 47 0 0 02 Oct 2024
The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof Derek Lim Moe Putterman Robin Walters Haggai Maron Stefanie Jegelka 35 5 0 30 May 2024
Arcee's MergeKit: A Toolkit for Merging Large Language Models Charles Goddard Shamane Siriwardhana Malikeh Ehghaghi Luke Meyers Vladimir Karpukhin Brian Benedict Mark McQuade Jacob Solawetz MoMe KELM 82 77 0 20 Mar 2024
Git Re-Basin: Merging Models modulo Permutation Symmetries Samuel K. Ainsworth J. Hayase S. Srinivasa MoMe 243 313 0 11 Sep 2022
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency Giovanni Puccetti Anna Rogers Aleksandr Drozd F. Dell’Orletta 71 42 0 23 May 2022
Diverse Weight Averaging for Out-of-Distribution Generalization Alexandre Ramé Matthieu Kirchmeyer Thibaud Rahier A. Rakotomamonjy Patrick Gallinari Matthieu Cord OOD 191 128 0 19 May 2022
Optimizing Mode Connectivity via Neuron Alignment N. Joseph Tatro Pin-Yu Chen Payel Das Igor Melnyk P. Sattigeri Rongjie Lai MoMe 223 80 0 05 Sep 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,950 0 20 Apr 2018
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 273 2,888 0 15 Sep 2016