Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

22 December 2020

Luke Zettlemoyer

Papers citing "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning"

50 / 117 papers shown

Title
SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules Xiangyu Chen Jing Liu Ye Wang Pu Wang Matthew Brand Guanghui Wang T. Koike-Akino 51 7 0 18 Mar 2024
AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning Ruiyi Zhang Rushi Qiang Sai Ashish Somayajula Pengtao Xie 42 13 0 14 Mar 2024
STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models Linhai Zhang Jialong Wu Deyu Zhou Guoqiang Xu 30 4 0 02 Mar 2024
Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping Jianbin Zheng Minghui Hu Zhongyi Fan Chaoyue Wang Changxing Ding Dacheng Tao Tat-Jen Cham 43 27 0 29 Feb 2024
SIBO: A Simple Booster for Parameter-Efficient Fine-Tuning Zhihao Wen Jie Zhang Yuan Fang MoE 34 3 0 19 Feb 2024
Uncertainty quantification in fine-tuned LLMs using LoRA ensembles Oleksandr Balabanov Hampus Linander UQCV 36 14 0 19 Feb 2024
Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation Zixian Guo Yuxiang Wei Ming-Yu Liu Zhilong Ji Jinfeng Bai Yiwen Guo Wangmeng Zuo VLM 36 8 0 26 Dec 2023
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs Max Zimmer Megi Andoni Christoph Spiegel Sebastian Pokutta VLM 52 10 0 23 Dec 2023
Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes Zhen Qin Daoyuan Chen Bingchen Qian Bolin Ding Yaliang Li Shuiguang Deng FedML 42 32 0 11 Dec 2023
Efficient Stitchable Task Adaptation Haoyu He Zizheng Pan Jing Liu Jianfei Cai Bohan Zhuang 34 3 0 29 Nov 2023
A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA Damjan Kalajdzievski ALM 33 80 0 28 Nov 2023
Bridging Information-Theoretic and Geometric Compression in Language Models Emily Cheng Corentin Kervadec Marco Baroni 36 17 0 20 Oct 2023
Matrix Compression via Randomized Low Rank and Low Precision Factorization R. Saha Varun Srivastava Mert Pilanci 26 19 0 17 Oct 2023
Decomposed Prompt Tuning via Low-Rank Reparameterization Yao Xiao Lu Xu Jiaxi Li Wei Lu Xiaoli Li VLM 25 6 0 16 Oct 2023
NOLA: Compressing LoRA using Linear Combination of Random Basis Soroush Abbasi Koohpayegani K. Navaneet Parsa Nooralinejad Soheil Kolouri Hamed Pirsiavash 40 12 0 04 Oct 2023
AdaptNet: Policy Adaptation for Physics-Based Character Control Pei Xu Kaixiang Xie Sheldon Andrews P. Kry Michael Neff Morgan McGuire Ioannis Karamouzas Victor Zordan TTA 39 17 0 30 Sep 2023
Prompting or Fine-tuning? A Comparative Study of Large Language Models for Taxonomy Construction Boqi Chen Fandi Yi Dániel Varró 37 16 0 04 Sep 2023
Adaptive Low Rank Adaptation of Segment Anything to Salient Object Detection Rui-Qing Cui Siyuan He Shi Qiu VLM 18 5 0 10 Aug 2023
Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? T. Kajitsuka Issei Sato 31 16 0 26 Jul 2023
Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation Zunnan Xu Zhihong Chen Yong Zhang Yibing Song Xiang Wan Guanbin Li VLM 35 47 0 21 Jul 2023
Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions Dongshuo Yin Xueting Han Bin Li Hao Feng Jinghua Bai VPVLM 36 18 0 16 Jun 2023
Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation Minghui Hu Jianbin Zheng Daqing Liu Chuanxia Zheng Chaoyue Wang Dacheng Tao Tat-Jen Cham DiffM 37 9 0 01 Jun 2023
Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning Baohao Liao Shaomu Tan Christof Monz KELM 23 29 0 01 Jun 2023
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning Umang Gupta Aram Galstyan Greg Ver Steeg 11 2 0 30 May 2023
Fine-Tuning Language Models with Just Forward Passes Sadhika Malladi Tianyu Gao Eshaan Nichani Alexandru Damian Jason D. Lee Danqi Chen Sanjeev Arora 41 180 0 27 May 2023
COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models Jinqi Xiao Miao Yin Yu Gong Xiao Zang Jian Ren Bo Yuan VLM ViT 45 9 0 26 May 2023
Parameter-Efficient Fine-Tuning without Introducing New Latency Baohao Liao Yan Meng Christof Monz 24 49 0 26 May 2023
Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity Raman Dutt Linus Ericsson Pedro Sanchez Sotirios A. Tsaftaris Timothy M. Hospedales MedIm 35 50 0 14 May 2023
Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models Yaohua Zha Jinpeng Wang Tao Dai Bin Chen Zhi Wang Shutao Xia VLM 53 45 0 14 Apr 2023
Gradient-Free Textual Inversion Zhengcong Fei Mingyuan Fan Junshi Huang DiffM 33 31 0 12 Apr 2023
Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning Zhen Wang Yikang Shen Leonid Karlinsky Rogerio Feris Huan Sun Yoon Kim VLM VPVLM 44 107 0 06 Mar 2023
Modular Deep Learning Jonas Pfeiffer Sebastian Ruder Ivan Vulić Edoardo Ponti MoMe OOD 32 73 0 22 Feb 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 31 47 0 02 Feb 2023
The geometry of hidden representations of large transformer models L. Valeriani Diego Doimo F. Cuturello Alessandro Laio A. Ansuini Alberto Cazzaniga MILM 29 50 0 01 Feb 2023
When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods Zhuo Zhang Yuanhang Yang Yong Dai Lizhen Qu Zenglin Xu FedML 46 66 0 20 Dec 2022
BARTSmiles: Generative Masked Language Models for Molecular Representations Gayane Chilingaryan Hovhannes Tamoyan Ani Tevosyan N. Babayan L. Khondkaryan Karen Hambardzumyan Zaven Navoyan Hrant Khachatrian Armen Aghajanyan SSL 35 25 0 29 Nov 2022
Rethinking the Number of Shots in Robust Model-Agnostic Meta-Learning Xiaoyue Duan Guoliang Kang Runqi Wang Shumin Han Shenjun Xue Tian Wang Baochang Zhang 29 2 0 28 Nov 2022
NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension Xin He Jiangchao Yao Yuxin Wang Zhenheng Tang Ka Chu Cheung Simon See Bo Han Xiaowen Chu 19 9 0 23 Nov 2022
Cross-Reality Re-Rendering: Manipulating between Digital and Physical Realities Siddhartha Datta 33 0 0 15 Nov 2022
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning Yaqing Wang Sahaj Agarwal Subhabrata Mukherjee Xiaodong Liu Jing Gao Ahmed Hassan Awadallah Jianfeng Gao MoE 22 118 0 31 Oct 2022
Parameter-Efficient Tuning Makes a Good Classification Head Zhuoyi Yang Ming Ding Yanhui Guo Qingsong Lv Jie Tang VLM 58 14 0 30 Oct 2022
Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Delta Tuning Jing Yi Weize Chen Yujia Qin Yankai Lin Ning Ding Xu Han Zhiyuan Liu Maosong Sun Jie Zhou 20 2 0 24 Oct 2022
Parameter-Efficient Tuning with Special Token Adaptation Xiaoocong Yang James Y. Huang Wenxuan Zhou Muhao Chen 34 12 0 10 Oct 2022
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 33 109 0 31 Aug 2022
Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation Zejiang Hou Julian Salazar George Polovets 30 14 0 07 Jul 2022
When Does Differentially Private Learning Not Suffer in High Dimensions? Xuechen Li Daogao Liu Tatsunori Hashimoto Huseyin A. Inan Janardhan Kulkarni Y. Lee Abhradeep Thakurta 34 58 0 01 Jul 2022
Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models Yang Shu Zhangjie Cao Ziyang Zhang Jianmin Wang Mingsheng Long 22 4 0 08 Jun 2022
Parameter-Efficient and Student-Friendly Knowledge Distillation Jun Rao Xv Meng Liang Ding Shuhan Qi Dacheng Tao 37 46 0 28 May 2022
Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning Mozhdeh Gheini Xuezhe Ma Jonathan May 46 5 0 25 May 2022
BBTv2: Towards a Gradient-Free Future with Large Language Models Tianxiang Sun Zhengfu He Hong Qian Yunhua Zhou Xuanjing Huang Xipeng Qiu 108 53 0 23 May 2022