Towards Modular LLMs by Building and Reusing a Library of LoRAs

18 May 2024

Nicolas Le Roux

Papers citing "Towards Modular LLMs by Building and Reusing a Library of LoRAs"

25 / 25 papers shown

Title
Exploring How LLMs Capture and Represent Domain-Specific Knowledge Mirian Hipolito Garcia Camille Couturier Daniel Madrigal Diaz Ankur Mallick Anastasios Kyrillidis Robert Sim Victor Rühle Saravan Rajmohan 20 0 0 23 Apr 2025
FedMerge: Federated Personalization via Model Merging Shutong Chen Tianyi Zhou Guodong Long Jing Jiang Chengqi Zhang FedML MoMe 43 0 0 09 Apr 2025
MASS: MoErging through Adaptive Subspace Selection Donato Crisostomi Alessandro Zirilli Antonio Andrea Gargiulo Maria Sofia Bucarelli Simone Scardapane Fabrizio Silvestri Iacopo Masi Emanuele Rodolà MoMe 40 0 0 06 Apr 2025
Efficient Model Development through Fine-tuning Transfer Pin-Jie Lin Rishab Balasubramanian Fengyuan Liu Nikhil Kandpal Tu Vu 52 0 0 25 Mar 2025
Training Plug-n-Play Knowledge Modules with Deep Context Distillation Lucas Page-Caccia Alan Ansell E. Ponti Ivan Vulić Alessandro Sordoni SyDa 79 0 0 11 Mar 2025
LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits Zikai Zhou Qizheng Zhang Hermann Kumbong Kunle Olukotun MQ 120 0 0 12 Feb 2025
MoIN: Mixture of Introvert Experts to Upcycle an LLM Ajinkya Tejankar K. Navaneet Ujjawal Panchal Kossar Pourahmadi Hamed Pirsiavash MoE 29 0 0 13 Oct 2024
Glider: Global and Local Instruction-Driven Expert Router Pingzhi Li Prateek Yadav Jaehong Yoon Jie Peng Yi-Lin Sung Mohit Bansal Tianlong Chen MoMe MoE 25 1 0 09 Oct 2024
DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models Yuxuan Zhang Ruizhe Li MoMe 36 0 0 02 Oct 2024
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models Yuncheng Yang Yulei Qin Tong Wu Zihan Xu Gang Li ... Yuchen Shi Ke Li Xing Sun Jie Yang Yun Gu ALM OffRL MoE 40 0 0 28 Aug 2024
Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts Nikolas Gritsch Qizhen Zhang Acyr F. Locatelli Sara Hooker A. Ustun MoE 35 0 0 28 Aug 2024
On-Device Language Models: A Comprehensive Review Jiajun Xu Zhiyuan Li Wei Chen Qun Wang Xin Gao Qi Cai Ziyuan Ling 24 24 0 26 Aug 2024
SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models A. Tang Li Shen Yong Luo Shuai Xie Han Hu Lefei Zhang Bo Du Dacheng Tao MoMe 21 3 0 19 Aug 2024
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning Prateek Yadav Colin Raffel Mohammed Muqeeth Lucas Page-Caccia Haokun Liu Tianlong Chen Mohit Bansal Leshem Choshen Alessandro Sordoni MoMe 31 21 0 13 Aug 2024
MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts Lin Ning Harsh Lara Meiqi Guo Abhinav Rastogi MoMe MoE 19 1 0 02 Aug 2024
Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning Ziyu Zhao Leilei Gan Guoyin Wang Yuwei Hu Tao Shen Hongxia Yang Kun Kuang Fei Wu MoE MoMe 21 1 0 24 Jun 2024
Concept-skill Transferability-based Data Selection for Large Vision-Language Models Jaewoo Lee Boyang Li Sung Ju Hwang VLM 22 8 0 16 Jun 2024
MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models Jingwei Xu Junyu Lai Yunpeng Huang MoE MoMe 23 2 0 19 May 2024
DAM: Dynamic Adapter Merging for Continual Video QA Learning Feng Cheng Ziyang Wang Yi-Lin Sung Yan-Bo Lin Mohit Bansal Gedas Bertasius CLL MoMe 18 10 0 13 Mar 2024
Learning to Route Among Specialized Experts for Zero-Shot Generalization Mohammed Muqeeth Haokun Liu Yufan Liu Colin Raffel MoMe 20 14 0 08 Feb 2024
$π$ -Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation Chengyue Wu Teng Wang Yixiao Ge Zeyu Lu Rui-Zhi Zhou Ying Shan Ping Luo MoMe 60 35 0 27 Apr 2023
Git Re-Basin: Merging Models modulo Permutation Symmetries Samuel K. Ainsworth J. Hayase S. Srinivasa MoMe 239 313 0 11 Sep 2022
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer Tu Vu Brian Lester Noah Constant Rami Al-Rfou Daniel Matthew Cer VLM LRM 131 276 0 15 Oct 2021
Efficiently Identifying Task Groupings for Multi-Task Learning Christopher Fifty Ehsan Amid Zhe Zhao Tianhe Yu Rohan Anil Chelsea Finn 201 235 1 10 Sep 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 275 3,784 0 18 Apr 2021