Measuring the Intrinsic Dimension of Objective Landscapes

24 April 2018

Papers citing "Measuring the Intrinsic Dimension of Objective Landscapes"

50 / 88 papers shown

Title
Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning Anh Tong Thanh Nguyen-Tang Dongeun Lee Duc Nguyen Toan M. Tran David Hall Cheongwoong Kang Jaesik Choi 35 0 0 03 Mar 2025
BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models Yibin Wang H. Shi Ligong Han Dimitris N. Metaxas Hao Wang BDL UQLM 113 6 0 28 Jan 2025
LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics Thomas Robert M. Safaryan Ionut-Vlad Modoranu Dan Alistarh ODL 33 2 0 21 Oct 2024
Collaborative and Efficient Personalization with Mixtures of Adaptors Abdulla Jasem Almansoori Samuel Horváth Martin Takáč FedML 44 2 0 04 Oct 2024
Geometric Signatures of Compositionality Across a Language Model's Lifetime Jin Hwa Lee Thomas Jiralerspong Lei Yu Yoshua Bengio Emily Cheng CoGe 84 0 0 02 Oct 2024
Selective Aggregation for Low-Rank Adaptation in Federated Learning Pengxin Guo Shuang Zeng Y. Wang Huijie Fan Feifei Wang Liangqiong Qu FedML 44 8 0 02 Oct 2024
LoRA-Pro: Are Low-Rank Adapters Properly Optimized? Zhengbo Wang Jian Liang Ran He Zilei Wang Tieniu Tan 50 15 0 25 Jul 2024
SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning Bac Nguyen Stefan Uhlich Fabien Cardinaux Lukas Mauch Marzieh Edraki Aaron Courville OODD CLL VLM 54 3 0 03 Jul 2024
Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead Rickard Brüel-Gabrielsson Jiacheng Zhu Onkar Bhardwaj Leshem Choshen Kristjan Greenewald Mikhail Yurochkin Justin Solomon 43 5 0 17 Jun 2024
ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts Samar Khanna Medhanie Irgau David B. Lobell Stefano Ermon VLM 32 4 0 16 Jun 2024
MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning Hanqing Wang Zeguan Xiao Shuo Wang Guanhua Chen Guanhua Chen 44 19 0 13 Jun 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou S. Song Jianlong Wu Liqiang Nie Bernard Ghanem 45 6 0 07 Jun 2024
Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation Can Yaras Peng Wang Laura Balzano Qing Qu AI4CE 37 12 0 06 Jun 2024
Does SGD really happen in tiny subspaces? Minhak Song Kwangjun Ahn Chulhee Yun 69 4 1 25 May 2024
Emergence of a High-Dimensional Abstraction Phase in Language Transformers Emily Cheng Diego Doimo Corentin Kervadec Iuri Macocco Jade Yu A. Laio Marco Baroni 112 11 0 24 May 2024
Machine Unlearning via Null Space Calibration Huiqiang Chen Tianqing Zhu Xin Yu Wanlei Zhou 41 6 0 21 Apr 2024
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models Fanxu Meng Zhaohui Wang Muhan Zhang VLM 64 70 0 03 Apr 2024
Introducing Routing Functions to Vision-Language Parameter-Efficient Fine-Tuning with Low-Rank Bottlenecks Tingyu Qu Tinne Tuytelaars Marie-Francine Moens MoE 43 2 0 14 Mar 2024
Sparse is Enough in Fine-tuning Pre-trained Large Language Models Weixi Song Z. Li Lefei Zhang Hai Zhao Bo Du VLM 19 7 0 19 Dec 2023
Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes Zhen Qin Daoyuan Chen Bingchen Qian Bolin Ding Yaliang Li Shuiguang Deng FedML 32 32 0 11 Dec 2023
PCDP-SGD: Improving the Convergence of Differentially Private SGD via Projection in Advance Haichao Sha Ruixuan Liu Yi-xiao Liu Hong Chen 52 1 0 06 Dec 2023
A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA Damjan Kalajdzievski ALM 22 77 0 28 Nov 2023
The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models Anton Razzhigaev Matvey Mikhalchuk Elizaveta Goncharova Ivan V. Oseledets Denis Dimitrov Andrey Kuznetsov 27 7 0 10 Nov 2023
Efficient Cross-Task Prompt Tuning for Few-Shot Conversational Emotion Recognition Yige Xu Zhiwei Zeng Zhiqi Shen VLM 25 3 0 23 Oct 2023
Bridging Information-Theoretic and Geometric Compression in Language Models Emily Cheng Corentin Kervadec Marco Baroni 34 16 0 20 Oct 2023
Decomposed Prompt Tuning via Low-Rank Reparameterization Yao Xiao Lu Xu Jiaxi Li Wei Lu Xiaoli Li VLM 17 6 0 16 Oct 2023
NOLA: Compressing LoRA using Linear Combination of Random Basis Soroush Abbasi Koohpayegani K. Navaneet Parsa Nooralinejad Soheil Kolouri Hamed Pirsiavash 40 12 0 04 Oct 2023
AdaptNet: Policy Adaptation for Physics-Based Character Control Pei Xu Kaixiang Xie Sheldon Andrews P. Kry Michael Neff Morgan McGuire Ioannis Karamouzas Victor Zordan TTA 37 16 0 30 Sep 2023
Prompting or Fine-tuning? A Comparative Study of Large Language Models for Taxonomy Construction Boqi Chen Fandi Yi Dániel Varró 29 16 0 04 Sep 2023
Quantifying lottery tickets under label noise: accuracy, calibration, and complexity V. Arora Daniele Irto Sebastian Goldt G. Sanguinetti 36 2 0 21 Jun 2023
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning Umang Gupta Aram Galstyan Greg Ver Steeg 6 2 0 30 May 2023
Probing the Purview of Neural Networks via Gradient Analysis Jinsol Lee Charles Lehman M. Prabhushankar Ghassan AlRegib 29 7 0 06 Apr 2023
A survey of deep learning optimizers -- first and second order methods Rohan Kashyap ODL 31 6 0 28 Nov 2022
PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization Sanae Lotfi Marc Finzi Sanyam Kapoor Andres Potapczynski Micah Goldblum A. Wilson BDL MLT AI4CE 29 51 0 24 Nov 2022
Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States Ziqiao Wang Yongyi Mao 27 10 0 19 Nov 2022
Parameter-Efficient Tuning Makes a Good Classification Head Zhuoyi Yang Ming Ding Yanhui Guo Qingsong Lv Jie Tang VLM 37 14 0 30 Oct 2022
Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Delta Tuning Jing Yi Weize Chen Yujia Qin Yankai Lin Ning Ding Xu Han Zhiyuan Liu Maosong Sun Jie Zhou 15 2 0 24 Oct 2022
Tunable Complexity Benchmarks for Evaluating Physics-Informed Neural Networks on Coupled Ordinary Differential Equations Alexander New B. Eng A. Timm A. Gearhart 17 4 0 14 Oct 2022
Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask? Mansheej Paul F. Chen Brett W. Larsen Jonathan Frankle Surya Ganguli Gintare Karolina Dziugaite UQCV 25 38 0 06 Oct 2022
Extremely Simple Activation Shaping for Out-of-Distribution Detection Andrija Djurisic Nebojsa Bozanic Arjun Ashok Rosanne Liu OODD 169 150 0 20 Sep 2022
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 28 109 0 31 Aug 2022
LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinity Martin Gubri Maxime Cordy Mike Papadakis Yves Le Traon Koushik Sen AAML 27 51 0 26 Jul 2022
Towards Semantic Communication Protocols: A Probabilistic Logic Perspective Sejin Seo Jihong Park Seung-Woo Ko Jinho D. Choi M. Bennis Seong-Lyun Kim 27 22 0 08 Jul 2022
When Does Differentially Private Learning Not Suffer in High Dimensions? Xuechen Li Daogao Liu Tatsunori Hashimoto Huseyin A. Inan Janardhan Kulkarni Y. Lee Abhradeep Thakurta 25 58 0 01 Jul 2022
LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood Piotr Tempczyk Rafał Michaluk Łukasz Garncarek Przemysław Spurek Jacek Tabor Adam Goliñski 35 26 0 29 Jun 2022
Few-Shot Learning by Dimensionality Reduction in Gradient Space M. Gauch M. Beck Thomas Adler D. Kotsur Stefan Fiel ... Markus Holzleitner Werner Zellinger D. Klotz Sepp Hochreiter Sebastian Lehner 35 9 0 07 Jun 2022
PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models Rabeeh Karimi Mahabadi Luke Zettlemoyer James Henderson Marzieh Saeidi Lambert Mathias Ves Stoyanov Majid Yazdani VLM 31 69 0 03 Apr 2022
APG: Adaptive Parameter Generation Network for Click-Through Rate Prediction Bencheng Yan Pengjie Wang Kai Zhang Feng Li Hongbo Deng Jian Xu Bo Zheng 19 20 0 30 Mar 2022
Parameter-efficient Model Adaptation for Vision Transformers Xuehai He Chunyuan Li Pengchuan Zhang Jianwei Yang X. Wang 28 84 0 29 Mar 2022
Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models Ning Ding Yujia Qin Guang Yang Fu Wei Zonghan Yang ... Jianfei Chen Yang Liu Jie Tang Juan Li Maosong Sun 20 196 0 14 Mar 2022