Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

22 December 2020

Luke Zettlemoyer

Papers citing "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning"

50 / 117 papers shown

Title
Transformers for Learning on Noisy and Task-Level Manifolds: Approximation and Generalization Insights Zhaiming Shen Alex Havrilla Rongjie Lai A. Cloninger Wenjing Liao 39 0 0 06 May 2025
Integration of Large Language Models and Traditional Deep Learning for Social Determinants of Health Prediction Paul Landes Jimeng Sun Adam Cross 36 0 0 06 May 2025
Communication-Efficient Wireless Federated Fine-Tuning for Large-Scale AI Models Bumjun Kim Wan Choi 34 0 0 01 May 2025
FISH-Tuning: Enhancing PEFT Methods with Fisher Information Kang Xue Ming Dong Xinhui Tu Tingting He 41 0 0 05 Apr 2025
Liger: Linearizing Large Language Models to Gated Recurrent Structures Disen Lan Weigao Sun Jiaxi Hu Jusen Du Yu-Xi Cheng 69 0 0 03 Mar 2025
PaCA: Partial Connection Adaptation for Efficient Fine-Tuning Sunghyeon Woo Sol Namkung Sunwoo Lee Inho Jeong Beomseok Kim Dongsuk Jeon 39 0 0 28 Feb 2025
Low Tensor-Rank Adaptation of Kolmogorov--Arnold Networks Yihang Gao Michael K. Ng Vincent Y. F. Tan 77 0 0 17 Feb 2025
TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems Si-Yang Liu Han-Jia Ye 70 7 0 04 Feb 2025
Explaining Context Length Scaling and Bounds for Language Models Jingzhe Shi Qinwei Ma Hongyi Liu Hang Zhao Jeng-Neng Hwang Lei Li LRM 81 3 0 03 Feb 2025
PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment Zequan Liu Yi Zhao Ming Tan Wei Zhu Aaron Xuxiang Tian 43 0 0 03 Feb 2025
RandLoRA: Full-rank parameter-efficient fine-tuning of large models Paul Albert Frederic Z. Zhang Hemanth Saratchandran Cristian Rodriguez-Opazo Anton van den Hengel Ehsan Abbasnejad 111 2 0 03 Feb 2025
Memory-Efficient Fine-Tuning of Transformers via Token Selection Antoine Simoulin Namyong Park Xiaoyi Liu Grey Yang 115 0 0 31 Jan 2025
MADation: Face Morphing Attack Detection with Foundation Models Eduarda Caldeira Guray Ozgur Tahar Chettaoui Marija Ivanovska Peter Peer Fadi Boutros Vitomir Štruc Naser Damer CVBM 47 2 1 28 Jan 2025
DAGPrompT: Pushing the Limits of Graph Prompting with a Distribution-aware Graph Prompt Tuning Approach Qin Chen Liang Wang Bo Zheng Guojie Song 44 2 0 28 Jan 2025
BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models Yibin Wang Haizhou Shi Ligong Han Dimitris N. Metaxas Hao Wang BDL UQLM 116 8 0 28 Jan 2025
ABXI: Invariant Interest Adaptation for Task-Guided Cross-Domain Sequential Recommendation Qingtian Bian Marcus Vinícius de Carvalho Tieying Li Jiaxing Xu Hui Fang Yiping Ke 48 0 0 25 Jan 2025
EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition Hamid Nasiri Peter Garraghan 41 1 0 21 Jan 2025
Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards Omar Erak Nouf Alabbasi Omar Alhussein Ismail Lotfi Amr Hussein Sami Muhaidat Merouane Debbah RALM 84 5 0 17 Jan 2025
Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning Hanwen Zhong Jiaxin Chen Yutong Zhang Di Huang Yunhong Wang MoE 44 0 0 12 Jan 2025
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs Akhiad Bercovich Tomer Ronen Talor Abramovich Nir Ailon Nave Assaf ... Ido Shahaf Oren Tropp Omer Ullman Argov Ran Zilberstein Ran El-Yaniv 88 1 0 28 Nov 2024
FRoundation: Are Foundation Models Ready for Face Recognition? Tahar Chettaoui Naser Damer Fadi Boutros CVBM 41 5 0 31 Oct 2024
Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies Liwen Wang Sheng Chen Linnan Jiang Shu Pan Runze Cai Sen Yang Fei Yang 49 3 0 24 Oct 2024
Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations M. Germán-Morales A. J. Rivera-Rivas M. J. del Jesus Díaz C. J. Carmona AI4TS AI4CE 56 0 0 15 Oct 2024
Parameter Efficient Fine-tuning via Explained Variance Adaptation Fabian Paischer Lukas Hauzenberger Thomas Schmied Benedikt Alkin Marc Peter Deisenroth Sepp Hochreiter 42 4 0 09 Oct 2024
Collaborative and Efficient Personalization with Mixtures of Adaptors Abdulla Jasem Almansoori Samuel Horváth Martin Takáč FedML 46 2 0 04 Oct 2024
Large Language Models as Markov Chains Oussama Zekri Ambroise Odonnat Abdelhakim Benechehab Linus Bleistein Nicolas Boullé I. Redko 48 10 0 03 Oct 2024
Attention layers provably solve single-location regression Pierre Marion Raphael Berthier Gérard Biau Claire Boyer 194 3 0 02 Oct 2024
Geometric Signatures of Compositionality Across a Language Model's Lifetime Jin Hwa Lee Thomas Jiralerspong Lei Yu Yoshua Bengio Emily Cheng CoGe 87 1 0 02 Oct 2024
Selective Aggregation for Low-Rank Adaptation in Federated Learning Pengxin Guo Shuang Zeng Y. Wang Huijie Fan Feifei Wang Liangqiong Qu FedML 47 11 0 02 Oct 2024
Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopy Bojian Li Bo Liu Jinghua Yue F. Zhou Fugen Zhou MedIm MDE 53 2 0 12 Sep 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 75 17 0 25 Jul 2024
Impact of Model Size on Fine-tuned LLM Performance in Data-to-Text Generation: A State-of-the-Art Investigation Joy Mahapatra Utpal Garain 45 8 0 19 Jul 2024
Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences Nikolaos Dimitriadis Pascal Frossard F. Fleuret MoE 67 6 0 10 Jul 2024
SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning Bac Nguyen Stefan Uhlich Fabien Cardinaux Lukas Mauch Marzieh Edraki Aaron Courville OODD CLL VLM 57 3 0 03 Jul 2024
Reasoning in Large Language Models: A Geometric Perspective Romain Cosentino Sarath Shekkizhar LRM 46 2 0 02 Jul 2024
ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts Samar Khanna Medhanie Irgau David B. Lobell Stefano Ermon VLM 32 4 0 16 Jun 2024
Talking Heads: Understanding Inter-layer Communication in Transformer Language Models Jack Merullo Carsten Eickhoff Ellie Pavlick 58 13 0 13 Jun 2024
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning Hanqing Wang Zeguan Xiao Shuo Wang Guanhua Chen Guanhua Chen 44 19 0 13 Jun 2024
Enhancing Domain Adaptation through Prompt Gradient Alignment Hoang Phan Lam C. Tran Quyen Tran Trung Le 52 0 0 13 Jun 2024
Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning Chung-Ming Chien Andros Tjandra Apoorv Vyas Matt Le Bowen Shi Wei-Ning Hsu 32 0 0 10 Jun 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou Shuaiwen Leon Song Jianlong Wu Liqiang Nie Guohao Li 45 6 0 07 Jun 2024
Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation Can Yaras Peng Wang Laura Balzano Qing Qu AI4CE 37 13 0 06 Jun 2024
Low-Rank Adaption on Transformer-based Oriented Object Detector for Satellite Onboard Processing of Remote Sensing Images Xinyang Pu Feng Xu 32 3 0 04 Jun 2024
An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updates Albin Soutif--Cormerais Simone Magistri Joost van de Weijer Andew D. Bagdanov 40 1 0 28 May 2024
Emergence of a High-Dimensional Abstraction Phase in Language Transformers Emily Cheng Diego Doimo Corentin Kervadec Iuri Macocco Jade Yu Alessandro Laio Marco Baroni 112 11 0 24 May 2024
Leveraging Large Language Models for Multimodal Search Oriol Barbany Michael Huang Xinliang Zhu Arnab Dhua 31 9 0 24 Apr 2024
FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering Siqi Ping Yuzhu Mao Yang Liu Xiao-Ping Zhang Wenbo Ding FedML 34 4 0 23 Apr 2024
Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation Xinyu Ma Xu Chu Zhibang Yang Yang Lin Xin Gao Junfeng Zhao 46 7 0 05 Apr 2024
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Fanxu Meng Zhaohui Wang Muhan Zhang VLM 64 76 0 03 Apr 2024
MEDDAP: Medical Dataset Enhancement via Diversified Augmentation Pipeline Yasamin Medghalchi N. Zakariaei Arman Rahmim I. Hacihaliloglu MedIm 46 2 0 25 Mar 2024