Leveraging redundancy in attention with Reuse Transformers

13 October 2021

Srinadh Bhojanapalli

Sanjiv Kumar

Papers citing "Leveraging redundancy in attention with Reuse Transformers"

21 / 21 papers shown

Title
Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps Jiashun Cheng Aochuan Chen Nuo Chen Ziqi Gao Yuhan Li Jia Li Fugee Tsung 80 0 0 20 Jun 2025
AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs Di He Ajay Jaiswal Songjun Tu Li Shen Ganzhao Yuan Shiwei Liu L. Yin 130 0 0 17 Jun 2025
Is Attention Required for Transformer Inference? Explore Function-preserving Attention Replacement Yuxin Ren Maxwell D Collins Miao Hu Huanrui Yang 140 0 0 24 May 2025
MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning Jiancheng Zhao Xingda Yu Zhen Yang MoE 134 3 0 27 Mar 2025
AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models Haiquan Lu Yefan Zhou Shiwei Liu Zhangyang Wang Michael W. Mahoney Yaoqing Yang 84 13 0 14 Oct 2024
Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression Jingcun Wang Yu-Guang Chen Ing-Chao Lin Bing Li Grace Li Zhang 117 9 0 02 Oct 2024
EchoAtt: Attend, Copy, then Adjust for More Efficient Large Language Models Hossein Rajabzadeh A. Jafari Aman Sharma Benyamin Jami Hyock Ju Kwon Ali Ghodsi Boxing Chen Mehdi Rezagholizadeh 73 0 0 22 Sep 2024
EfficientASR: Speech Recognition Network Compression via Attention Redundancy and Chunk-Level FFN Optimization Jianzong Wang Ziqi Liang Xulong Zhang Ning Cheng Jing Xiao 112 0 0 30 Apr 2024
MLP Can Be A Good Transformer Learner Sihao Lin Pumeng Lyu Dongrui Liu Tao Tang Xiaodan Liang Andy Song Xiaojun Chang ViT 125 14 0 08 Apr 2024
Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers Yiwei Lu Yaoliang Yu Xinlin Li Vahid Partovi Nia MQ 121 5 0 27 Feb 2024
MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT Omkar Thawakar Ashmal Vayani Salman Khan Hisham Cholakal Rao M. Anwer Michael Felsberg Timothy Baldwin Eric P. Xing Fahad Shahbaz Khan 155 42 0 26 Feb 2024
Head-wise Shareable Attention for Large Language Models Zouying Cao Yifei Yang Hai Zhao 100 4 0 19 Feb 2024
Dynamic Layer Tying for Parameter-Efficient Transformers Tamir David Hay Lior Wolf 98 5 0 23 Jan 2024
Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models Rosco Hunter Łukasz Dudziak Mohamed S. Abdelfattah Abhinav Mehrotra Sourav Bhattacharya Hongkai Wen 211 1 0 13 Dec 2023
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin You Wu Zhenyu Zhang Cheng-Yu Hsieh Yaqing Wang ... Mykola Pechenizkiy Yi Liang Michael Bendersky Zhangyang Wang Shiwei Liu 225 115 0 08 Oct 2023
Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation Kangwook Jang Sungnyun Kim Se-Young Yun Hoi-Rim Kim 136 7 0 19 May 2023
AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems Yuan Feng Hyeran Jeon F. Blagojevic Cyril Guyot Qing Li Dong Li GNN 91 3 0 23 Jan 2023
Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition Ruiqi Wang Wonse Jo Dezhong Zhao Weizheng Wang B. Yang Guohua Chen Byung-Cheol Min HAI 116 33 0 30 Sep 2022
On The Computational Complexity of Self-Attention Feyza Duman Keles Pruthuvi Maheshakya Wijewardena Chinmay Hegde 187 164 0 11 Sep 2022
Visualizing and Understanding Patch Interactions in Vision Transformer Jie Ma Yalong Bai Bineng Zhong Wei Zhang Ting Yao Tao Mei ViT 76 43 0 11 Mar 2022
Q-ViT: Fully Differentiable Quantization for Vision Transformer Zhexin Li Tong Yang Peisong Wang Jian Cheng ViT MQ 115 44 0 19 Jan 2022