Improving Neural Network Training in Low Dimensional Random Bases

Improving Neural Network Training in Low Dimensional Random Bases

9 November 2020

Frithjof Gressmann

Zach Eaton-Rosen

Carlo Luschi

Papers citing "Improving Neural Network Training in Low Dimensional Random Bases"

18 / 18 papers shown

Title
Preconditioned Subspace Langevin Monte Carlo Tyler Maunu Jiayi Yao 90 0 0 18 Dec 2024
Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis Zhijie Chen Qiaobo Li A. Banerjee FedML 35 0 0 11 Nov 2024
PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning Qibin Wang Xiaolin Hu Weikai Xu Wei Liu Jian Luan Bin Wang 28 1 0 25 Sep 2024
Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategy Tao Li Weisen Jiang Fanghui Liu X. Huang James T. Kwok MoMe 53 1 0 04 Jul 2024
Interpretability of Language Models via Task Spaces Lucas Weber Jaap Jumelet Elia Bruni Dieuwke Hupkes 35 4 0 10 Jun 2024
Does SGD really happen in tiny subspaces? Minhak Song Kwangjun Ahn Chulhee Yun 61 4 1 25 May 2024
Towards Green AI: Current status and future research Christian Clemm Lutz Stobbe Kishan Wimalawarne Jan Druschke 45 2 0 01 May 2024
Training-time Neuron Alignment through Permutation Subspace for Improving Linear Mode Connectivity and Model Fusion Zexi Li Zhiqi Li Jie Lin Tao Shen Tao Lin Chao Wu 28 4 0 02 Feb 2024
Identifying Policy Gradient Subspaces Jan Schneider-Barnes Pierre Schumacher Simon Guist Le Chen D. Haeufle Bernhard Scholkopf Dieter Buchler 31 5 0 12 Jan 2024
Enhancing Neural Training via a Correlated Dynamics Model Jonathan Brokman Roy Betser Rotem Turjeman Tom Berkov I. Cohen Guy Gilboa 24 3 0 20 Dec 2023
Deep Model Fusion: A Survey Weishi Li Yong Peng Miao Zhang Liang Ding Han Hu Li Shen FedML MoMe 28 51 0 27 Sep 2023
Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models Zhong Zhang Bang Liu Junming Shao 23 6 0 27 May 2023
PGrad: Learning Principal Gradients For Domain Generalization Zhe Wang J. E. Grigsby Yanjun Qi OOD 29 10 0 02 May 2023
Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimization Yueqi Xie Weizhong Zhang Renjie Pi Fangzhao Wu Qifeng Chen Xing Xie Sunghun Kim FedML 18 7 0 10 Nov 2022
Trainable Weight Averaging: Accelerating Training and Improving Generalization Tao Li Zhehao Huang Yingwen Wu Zhengbao He Qinghua Tao X. Huang Chih-Jen Lin MoMe 50 0 0 26 May 2022
Kernel Modulation: A Parameter-Efficient Method for Training Convolutional Neural Networks Yuhuang Hu Shih-Chii Liu 15 0 0 29 Mar 2022
Subspace Adversarial Training Tao Li Yingwen Wu Sizhe Chen Kun Fang Xiaolin Huang AAML OOD 36 56 0 24 Nov 2021
Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces Tao Li Lei Tan Qinghua Tao Yipeng Liu Xiaolin Huang 29 10 0 20 Mar 2021