CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging

Model merging based on task vectors, i.e., the parameter differences between fine-tuned models and a shared base model, provides an efficient way to integrate multiple task-specific models into a multitask model without retraining. Recent works have endeavored to address the conflicts between task vectors, one of the significant challenges faced by model merging, through sparsification; however, two issues significantly limit their performance: high parameter overlap and unbalanced weight distribution. To address these issues, we propose a simple, yet effective framework called CABS (Conflict-Aware and Balanced Sparsification), consisting of Conflict-Aware Sparsification (CA) and Balanced Sparsification (BS). CA can reduce parameter overlap by applying masks during sequential pruning, ensuring that each task vector retains distinct, non-overlapping parameters. BS leverages $n$: $m$ pruning to preserve critical weights while maintaining an even distribution across layers. Our comprehensive experiments demonstrate that CABS outperforms state-of-the-art methods across diverse tasks and model sizes.

@article{yang2025_2503.01874,
  title={ CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging },
  author={ Zongzhen Yang and Binhang Qi and Hailong Sun and Wenrui Long and Ruobing Zhao and Xiang Gao },
  journal={arXiv preprint arXiv:2503.01874},
  year={ 2025 }
}