Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning

18 December 2023

Jieru Mei

Papers citing "Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning"

31 / 31 papers shown

Title
OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning Xianhang Li Y. Liu Haoqin Tu Hongru Zhu Cihang Xie VLM 130 0 0 07 May 2025
VAEmo: Efficient Representation Learning for Visual-Audio Emotion with Knowledge Injection Hao Cheng Zhiwei Zhao Yichao He Zhenzhen Hu Jia Li M. Wang Richang Hong 41 0 0 05 May 2025
Segment Any Crack: Deep Semantic Segmentation Adaptation for Crack Detection Ghodsiyeh Rostami Po-Han Chen Mahdi S. Hosseini 19 0 0 19 Apr 2025
Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilities Raman Dutt Harleen Hanspal Guoxuan Xia Petru-Daniel Tudosiu Alexander Black Yongxin Yang Steven G. McDonagh Sarah Parisot MoE 38 0 0 28 Mar 2025
Decoupling Angles and Strength in Low-rank Adaptation Massimo Bini Leander Girrbach Zeynep Akata 40 0 0 23 Mar 2025
Parameter-Efficient Adaptation of Geospatial Foundation Models through Embedding Deflection Romain Thoreau Valerio Marsocci Dawa Derksen AI4CE 69 2 0 12 Mar 2025
Learning to Reason from Feedback at Test-Time Yanyang Li M. Lyu Liwei Wang LRM 31 1 0 16 Feb 2025
Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models Christopher Simic K. Riedhammer Tobias Bocklet 91 0 0 03 Feb 2025
Parameter-Efficient Fine-Tuning for Foundation Models Dan Zhang Tao Feng Lilong Xue Yuandong Wang Yuxiao Dong J. Tang 39 7 0 23 Jan 2025
A Hessian-informed hyperparameter optimization for differential learning rate Shiyun Xu Zhiqi Bu Yiliang Zhang Ian J. Barnett 39 1 0 12 Jan 2025
A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine Hanguang Xiao Feizhong Zhou X. Liu Tianqi Liu Zhipeng Li Xin Liu Xiaoxuan Huang AILaw LM&MA LRM 59 17 0 31 Dec 2024
PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuning Shenghui Li Edith C. H. Ngai Fanghua Ye Thiemo Voigt SILM 86 6 0 28 Nov 2024
ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning Zhiwei Hao Jianyuan Guo Li Shen Yong Luo Han Hu Yonggang Wen VLM 21 0 0 23 Oct 2024
AERO: Softmax-Only LLMs for Efficient Private Inference N. Jha Brandon Reagen 27 1 0 16 Oct 2024
Revisited Large Language Model for Time Series Analysis through Modality Alignment L. Zheng Chang Dong W. Zhang Lin Yue Miao Xu Olaf Maennel Weitong Chen AI4TS 120 2 0 16 Oct 2024
RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates Md. Kowsher Tara Esmaeilbeig Chun-Nam Yu Mojtaba Soltanalian Niloofar Yousefi 27 0 0 14 Oct 2024
ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models N. Jha Brandon Reagen OffRL AI4CE 28 0 0 12 Oct 2024
PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization Yao Ni Shan Zhang Piotr Koniusz 131 2 0 25 Sep 2024
Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models Divij Gupta Anubhav Bhatti Surajsinh Parmar AI4TS 26 1 0 17 Sep 2024
HRP: Human Affordances for Robotic Pre-Training M. K. Srirama Sudeep Dasari Shikhar Bahl Abhinav Gupta 26 13 0 26 Jul 2024
ASPS: Augmented Segment Anything Model for Polyp Segmentation Huiqian Li Dingwen Zhang Jieru Yao Longfei Han Zhongyu Li J. Han 46 10 0 30 Jun 2024
Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning Bingchen Zhao Yongshuo Zong Letian Zhang Timothy Hospedales VLM 28 15 0 18 Jun 2024
Effectiveness of Vision Language Models for Open-world Single Image Test Time Adaptation Manogna Sreenivas Soma Biswas VLM 43 0 0 01 Jun 2024
Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization Zhe Li Bicheng Ying Zidong Liu Haibo Yang Haibo Yang FedML 59 3 0 24 May 2024
Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models Jiawei Chen Dingkang Yang Yue Jiang Mingcheng Li Jinjie Wei Xiaolu Hou Lihua Zhang 48 6 0 25 Apr 2024
Lowering PyTorch's Memory Consumption for Selective Differentiation Samarth Bhatia Felix Dangel 16 1 0 15 Apr 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 139 308 0 21 Mar 2024
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality Qinghao Ye Haiyang Xu Guohai Xu Jiabo Ye Ming Yan ... Junfeng Tian Qiang Qi Ji Zhang Feiyan Huang Jingren Zhou VLM MLLM 206 900 0 27 Apr 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 265 4,229 0 30 Jan 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 308 11,915 0 04 Mar 2022
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts Soravit Changpinyo P. Sharma Nan Ding Radu Soricut VLM 273 1,081 0 17 Feb 2021