Kronecker Decomposition for GPT Compression

Kronecker Decomposition for GPT Compression

15 October 2021

Marzieh S. Tahaei

Mehdi Rezagholizadeh

Papers citing "Kronecker Decomposition for GPT Compression"

9 / 9 papers shown

Title
Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models Mingxue Xu Sadia Sharmin Danilo P. Mandic 14 2 0 03 Oct 2024
TQCompressor: improving tensor decomposition methods in neural networks via permutations V. Abronin A. Naumov D. Mazur D. Bystrov K. Tsarova Ar. Melnikov Ivan V. Oseledets S. Dolgov R. Brasher M. Perelshtein 25 6 0 29 Jan 2024
What Matters In The Structured Pruning of Generative Language Models? Michael Santacroce Zixin Wen Yelong Shen Yuan-Fang Li 12 32 0 07 Feb 2023
Parameter-efficient Model Adaptation for Vision Transformers Xuehai He Chunyuan Li Pengchuan Zhang Jianwei Yang X. Wang 20 80 0 29 Mar 2022
Compression of Generative Pre-trained Language Models via Quantization Chaofan Tao Lu Hou Wei Zhang Lifeng Shang Xin Jiang Qun Liu Ping Luo Ngai Wong MQ 22 103 0 21 Mar 2022
A Short Study on Compressing Decoder-Based Language Models Tianda Li Yassir El Mesbahi I. Kobyzev Ahmad Rashid A. Mahmud Nithin Anchuri Habib Hajimolahoseini Yang Liu Mehdi Rezagholizadeh 73 25 0 16 Oct 2021
KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation Marzieh S. Tahaei Ella Charlaix V. Nia A. Ghodsi Mehdi Rezagholizadeh 41 22 0 13 Sep 2021
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 243 1,791 0 17 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 294 6,927 0 20 Apr 2018