TIES-Merging: Resolving Interference When Merging Models

2 June 2023

Papers citing "TIES-Merging: Resolving Interference When Merging Models"

50 / 210 papers shown

Title
Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace Jinluan Yang A. Tang Didi Zhu Zhengyu Chen Li Shen Fei Wu MoMe AAML 52 3 0 17 Oct 2024
LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks Akshara Prabhakar Yuanzhi Li Karthik Narasimhan Sham Kakade Eran Malach Samy Jelassi MoMe 21 9 0 16 Oct 2024
The Non-Local Model Merging Problem: Permutation Symmetries and Variance Collapse Ekansh Sharma Daniel M. Roy Gintare Karolina Dziugaite MoMe 20 2 0 16 Oct 2024
Exploring Model Kinship for Merging Large Language Models Yedi Hu Yunzhi Yao N. Zhang Shumin Deng H. Chen MoMe 34 1 0 16 Oct 2024
Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants Rafael Ferreira David Semedo João Magalhães 31 1 0 16 Oct 2024
Agent Skill Acquisition for Large Language Models via CycleQD So Kuroki Taishi Nakamura Takuya Akiba Yujin Tang MoMe 29 0 0 16 Oct 2024
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence Shangbin Feng Zifeng Wang Yike Wang Sayna Ebrahimi Hamid Palangi ... Nathalie Rauschmayr Yejin Choi Yulia Tsvetkov Chen-Yu Lee Tomas Pfister MoMe 30 3 0 15 Oct 2024
Self-Data Distillation for Recovering Quality in Pruned Large Language Models Vithursan Thangarasa Ganesh Venkatesh Mike Lasby Nish Sinnadurai Sean Lie SyDa 33 0 0 13 Oct 2024
CollabEdit: Towards Non-destructive Collaborative Knowledge Editing Jiamu Zheng Jinghuai Zhang Tianyu Du Xuhong Zhang Jianwei Yin Tao Lin KELM 22 0 0 12 Oct 2024
How Does Vision-Language Adaptation Impact the Safety of Vision Language Models? Seongyun Lee Geewook Kim Jiyeon Kim Hyunji Lee Hoyeon Chang Sue Hyun Park Minjoon Seo 31 0 0 10 Oct 2024
Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models Zhipeng Chen Liang Song K. Zhou Wayne Xin Zhao B. Wang Weipeng Chen Ji-Rong Wen 60 0 0 10 Oct 2024
Glider: Global and Local Instruction-Driven Expert Router Pingzhi Li Prateek Yadav Jaehong Yoon Jie Peng Yi-Lin Sung Mohit Bansal Tianlong Chen MoMe MoE 25 1 0 09 Oct 2024
Decouple-Then-Merge: Finetune Diffusion Models as Multi-Task Learning Qianli Ma Xuefei Ning Dongrui Liu Li Niu Linfeng Zhang MoMe 44 0 0 09 Oct 2024
NegMerge: Consensual Weight Negation for Strong Machine Unlearning Hyoseo Kim Dongyoon Han Junsuk Choe MoMe MU 18 1 0 08 Oct 2024
Low-Rank Continual Personalization of Diffusion Models Łukasz Staniszewski Katarzyna Zaleska Kamil Deja DiffM 37 0 0 07 Oct 2024
What Matters for Model Merging at Scale? Prateek Yadav Tu Vu Jonathan Lai Alexandra Chronopoulou Manaal Faruqui Mohit Bansal Tsendsuren Munkhdalai MoMe 44 12 0 04 Oct 2024
Parameter Competition Balancing for Model Merging Guodong Du Junlin Lee Jing Li Runhua Jiang Yifei Guo ... Hanting Liu S. Goh Ho-Kin Tang Daojing He Min Zhang MoMe 19 12 0 03 Oct 2024
DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation Changdae Oh Yixuan Li Kyungwoo Song Sangdoo Yun Dongyoon Han OOD MoMe 36 4 0 03 Oct 2024
Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging Tingfeng Hui Zhenyu Zhang Shuohuan Wang Yu Sun Hua-Hong Wu Sen Su MoE 16 0 0 02 Oct 2024
Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks Edan Kinderman Itay Hubara Haggai Maron Daniel Soudry MoMe 45 0 0 02 Oct 2024
Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models Lucas Bandarkar Benjamin Muller Pritish Yuvraj Rui Hou Nayan Singhal Hongjiang Lv Bing-Quan Liu KELM LRM MoMe 33 2 0 02 Oct 2024
The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Merging Masanori Hirano Kentaro Imajo MoMe 24 1 0 30 Sep 2024
HM3: Heterogeneous Multi-Class Model Merging Stefan Hackmann MoMe 25 0 0 27 Sep 2024
HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models Yu Zhou Xingyu Wu Jibin Wu Liang Feng Kay Chen Tan MoMe 59 0 0 27 Sep 2024
Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integration Mahdi Morafah Vyacheslav Kungurtsev Hojin Chang C. L. P. Chen Bill Lin FedML 27 0 0 27 Sep 2024
Prompt Sliders for Fine-Grained Control, Editing and Erasing of Concepts in Diffusion Models Deepak Sridhar Nuno Vasconcelos DiffM 31 0 0 25 Sep 2024
Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering Ziyu Zhao Tao Shen Didi Zhu Zexi Li Jing Su Xuwu Wang Kun Kuang Fei Wu MoMe 16 5 0 24 Sep 2024
Recent Advances in Attack and Defense Approaches of Large Language Models Jing Cui Yishi Xu Zhewei Huang Shuchang Zhou Jianbin Jiao Junge Zhang PILM AAML 52 1 0 05 Sep 2024
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models Yuncheng Yang Yulei Qin Tong Wu Zihan Xu Gang Li ... Yuchen Shi Ke Li Xing Sun Jie Yang Yun Gu ALM OffRL MoE 46 0 0 28 Aug 2024
SQL-GEN: Bridging the Dialect Gap for Text-to-SQL Via Synthetic Data And Model Merging Mohammadreza Pourreza Ruoxi Sun Hailong Li Lesly Miculicich Tomas Pfister Sercan Ö. Arik MoMe 27 5 0 22 Aug 2024
MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair Meghdad Dehghan Jie JW Wu Fatemeh H. Fard Ali Ouni MoMe 35 2 0 18 Aug 2024
Activated Parameter Locating via Causal Intervention for Model Merging Fanshuang Kong Richong Zhang Ziqiao Wang MoMe 11 1 0 18 Aug 2024
Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models Vladimir Araujo Marie-Francine Moens Tinne Tuytelaars CLL MoMe 21 2 0 16 Aug 2024
FuseChat: Knowledge Fusion of Chat Models Fanqi Wan Longguang Zhong Ziyi Yang Ruijun Chen Xiaojun Quan ALM KELM MoMe 26 23 0 15 Aug 2024
BadMerging: Backdoor Attacks Against Model Merging Jinghuai Zhang Jianfeng Chi Zheng Li Kunlin Cai Yang Zhang Yuan Tian MoMe FedML AAML 32 14 0 14 Aug 2024
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning Prateek Yadav Colin Raffel Mohammed Muqeeth Lucas Page-Caccia Haokun Liu Tianlong Chen Mohit Bansal Leshem Choshen Alessandro Sordoni MoMe 38 21 0 13 Aug 2024
Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement Le Yu Bowen Yu Haiyang Yu Fei Huang Yongbin Li MoMe 27 5 0 06 Aug 2024
Efficient Pareto Manifold Learning with Low-Rank Structure Weiyu Chen James T. Kwok 23 6 0 30 Jul 2024
MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning Yupeng Chen Senmiao Wang Zhihang Lin Zhihang Lin Yushun Zhang Tian Ding Ruoyu Sun Ruoyu Sun CLL 72 1 0 30 Jul 2024
Knowledge Mechanisms in Large Language Models: A Survey and Perspective Meng Wang Yunzhi Yao Ziwen Xu Shuofei Qiao Shumin Deng ... Yong-jia Jiang Pengjun Xie Fei Huang Huajun Chen Ningyu Zhang 47 27 0 22 Jul 2024
Mitigating Catastrophic Forgetting in Language Transfer via Model Merging Anton Alexandrov Veselin Raychev Mark Niklas Muller Ce Zhang Martin Vechev Kristina Toutanova MoMe CLL KELM 33 13 0 11 Jul 2024
Foundation Model Engineering: Engineering Foundation Models Just as Engineering Software Dezhi Ran Mengzhou Wu Wei Yang Tao Xie AI4CE 24 1 0 11 Jul 2024
Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences Nikolaos Dimitriadis Pascal Frossard F. Fleuret MoE 57 6 0 10 Jul 2024
MagMax: Leveraging Model Merging for Seamless Continual Learning Daniel Marczak Bartłomiej Twardowski Tomasz Trzciñski Sebastian Cygert MoMe CLL 34 17 0 08 Jul 2024
Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models Jinliang Lu Ziliang Pang Min Xiao Yaochen Zhu Rui Xia Jiajun Zhang MoMe 29 17 0 08 Jul 2024
Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis Stefan Horoi Albert Manuel Orozco Camacho Eugene Belilovsky Guy Wolf FedML MoMe 19 9 0 07 Jul 2024
Unlocking the Potential of Model Merging for Low-Resource Languages Mingxu Tao Chen Zhang Quzhe Huang Tianyao Ma Songfang Huang Dongyan Zhao Yansong Feng CLL MoMe 20 3 0 04 Jul 2024
Knowledge Composition using Task Vectors with Learned Anisotropic Scaling Frederic Z. Zhang Paul Albert Cristian Rodriguez-Opazo Anton van den Hengel Ehsan Abbasnejad MoMe 37 7 0 03 Jul 2024
PLeaS -- Merging Models with Permutations and Least Squares Anshul Nasery J. Hayase Pang Wei Koh Sewoong Oh MoMe 38 3 0 02 Jul 2024
DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging Tzu-Han Lin Chen An Li Hung-yi Lee Yun-Nung Chen VLM ALM 26 4 0 01 Jul 2024