Enabling Lightweight Fine-tuning for Pre-trained Language Model
Compression based on Matrix Product Operators

Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

4 June 2021

ArXiv (abs)PDF HTML

Papers citing "Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators"

19 / 19 papers shown

Title
Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations Zican Dong Han Peng Peiyu Liu Wayne Xin Zhao Dong Wu Feng Xiao Ziyi Wang MoE 86 2 0 09 Apr 2025
Parameter-Efficient Fine-Tuning for Foundation Models Dan Zhang Tao Feng Lilong Xue Yuandong Wang Yuxiao Dong J. Tang 234 12 0 23 Jan 2025
Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation Yu-Liang Zhan Zhong-Yi Lu Hao Sun Ze-Feng Gao 82 0 0 10 Nov 2024
Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models Mingxue Xu Sadia Sharmin Danilo Mandic 72 2 0 03 Oct 2024
Propulsion: Steering LLM with Tiny Fine-Tuning Md. Kowsher Nusrat Jahan Prottasha Prakash Bhat 93 7 0 17 Sep 2024
Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models Yuyan Chen Qiang Fu Ge Fan Lun Du Jian-Guang Lou Shi Han Dongmei Zhang Zhixu Li Yanghua Xiao MoE 81 17 0 04 Jul 2024
QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation Zhuo Chen Rumen Dangovski Charlotte Loh Owen Dugan Di Luo Marin Soljacic MQ 93 9 0 31 May 2024
Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression Peiyu Liu Zeming Gao Wayne Xin Zhao Yipeng Ma Tao Wang Ji-Rong Wen MQ 145 5 0 21 May 2024
Application of Tensorized Neural Networks for Cloud Classification Alifu Xiafukaiti Devanshu Garg Aruto Hosaka Koichi Yanagisawa Yuichiro Minato Tsuyoshi Yoshida 45 0 0 21 Mar 2024
LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models Yifan Yang Jiajun Zhou Ngai Wong Zheng Zhang 75 8 0 18 Feb 2024
Sparse Low-rank Adaptation of Pre-trained Language Models Ning Ding Xingtai Lv Qiaosen Wang Yulin Chen Bowen Zhou Zhiyuan Liu Maosong Sun 80 70 0 20 Nov 2023
Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study Peiyu Liu Zikang Liu Ze-Feng Gao Dawei Gao Wayne Xin Zhao Yaliang Li Bolin Ding Ji-Rong Wen MQ LRM 94 35 0 16 Jul 2023
On the Impact of Knowledge Distillation for Model Interpretability Hyeongrok Han Siwon Kim Hyun-Soo Choi Sungroh Yoon 64 5 0 25 May 2023
Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture Peiyu Liu Ze-Feng Gao Yushuo Chen Wayne Xin Zhao Ji-Rong Wen MoE 68 0 0 27 Mar 2023
Tensor Networks Meet Neural Networks: A Survey and Future Perspectives Maolin Wang Yu Pan Zenglin Xu Xiangli Yang Guangxi Li A. Cichocki Andrzej Cichocki 208 22 0 22 Jan 2023
An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning Shaoshuai Shi Qing Yang Yang Xiang Shuhan Qi Xinyu Wang 75 1 0 30 Nov 2022
Exploring Extreme Parameter Compression for Pre-trained Language Models Yuxin Ren Benyou Wang Lifeng Shang Xin Jiang Qun Liu 82 19 0 20 May 2022
Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attention Yifan Chen Devamanyu Hazarika Mahdi Namazifar Yang Liu Di Jin Dilek Z. Hakkani-Tür 57 9 0 07 May 2022
Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models Ze-Feng Gao Peiyu Liu Wayne Xin Zhao Zhong-Yi Lu Ji-Rong Wen MoE 73 27 0 02 Mar 2022