Compressible Dynamics in Deep Overparameterized Low-Rank Learning &
Adaptation

Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation

6 June 2024

Peng Wang

Qing Qu

Papers citing "Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation"

13 / 13 papers shown

Title
An Overview of Low-Rank Structures in the Training and Adaptation of Large Models Laura Balzano Tianjiao Ding B. Haeffele Soo Min Kwon Qing Qu Peng Wang Z. Wang Can Yaras OffRL AI4CE 50 0 0 25 Mar 2025
SubTrack your Grad: Gradient Subspace Tracking for Memory and Time Efficient Full-Parameter LLM Training Sahar Rajabi Nayeema Nonta Sirisha Rambhatla 80 0 0 03 Feb 2025
Understanding How Nonlinear Layers Create Linearly Separable Features for Low-Dimensional Data Alec S. Xu Can Yaras Peng Wang Q. Qu 23 0 0 04 Jan 2025
BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference Changwoo Lee Soo Min Kwon Qing Qu Hun-Seok Kim 20 0 0 28 Oct 2024
LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization Jui-Nan Yen Si Si Zhao Meng Felix X. Yu Sai Surya Duvvuri Inderjit Dhillon Cho-Jui Hsieh Sanjiv Kumar 22 1 0 27 Oct 2024
On the Crucial Role of Initialization for Matrix Factorization Bingcong Li Liang Zhang Aryan Mokhtari Niao He 26 1 0 24 Oct 2024
Large Language Models as Markov Chains Oussama Zekri Ambroise Odonnat Abdelhakim Benechehab Linus Bleistein Nicolas Boullé I. Redko 34 9 0 03 Oct 2024
Does SGD really happen in tiny subspaces? Minhak Song Kwangjun Ahn Chulhee Yun 44 4 1 25 May 2024
Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compression Joseph Shenouda Rahul Parhi Kangwook Lee Robert D. Nowak 19 12 0 25 May 2023
Rank Overspecified Robust Matrix Recovery: Subgradient Method and Exact Recovery Lijun Ding Liwei Jiang Yudong Chen Qing Qu Zhihui Zhu 13 28 0 23 Sep 2021
Initialization and Regularization of Factorized Neural Layers M. Khodak Neil A. Tenenholtz Lester W. Mackey Nicolò Fusi 63 56 0 03 May 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 220 3,054 0 23 Jan 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,927 0 20 Apr 2018