Low-Rank Bottleneck in Multi-head Attention Models

17 February 2020

Srinadh Bhojanapalli

Sanjiv Kumar

Papers citing "Low-Rank Bottleneck in Multi-head Attention Models"

20 / 20 papers shown

Title
Approximation Rate of the Transformer Architecture for Sequence Modeling Hao Jiang Qianxiao Li 46 9 0 03 Jan 2025
Hadamard Attention Recurrent Transformer: A Strong Baseline for Stereo Matching Transformer Ziyang Chen Yongjun Zhang Wenting Li Bingshu Wang Yabo Wu Yong Zhao C. L. P. Chen 40 0 0 02 Jan 2025
Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Pingyi Chen Zhongyi Shui Chenglu Zhu Lin Yang MedIm 32 4 0 18 Oct 2024
Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models Dennis Wu Jerry Yao-Chieh Hu Teng-Yun Hsiao Han Liu 40 28 0 04 Apr 2024
Data-free Weight Compress and Denoise for Large Language Models Runyu Peng Yunhua Zhou Qipeng Guo Yang Gao Hang Yan Xipeng Qiu Dahua Lin 36 1 0 26 Feb 2024
When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges Wang Chao Jiaxuan Zhao Licheng Jiao Lingling Li Fang Liu Shuyuan Yang 61 13 0 19 Jan 2024
Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? T. Kajitsuka Issei Sato 29 16 0 26 Jul 2023
Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion Isha Rawal Alexander Matyasko Shantanu Jaiswal Basura Fernando Cheston Tan 21 1 0 15 Jun 2023
Memorization Capacity of Multi-Head Attention in Transformers Sadegh Mahdavi Renjie Liao Christos Thrampoulidis 24 22 0 03 Jun 2023
EIT: Enhanced Interactive Transformer Tong Zheng Bei Li Huiwen Bao Tong Xiao Jingbo Zhu 24 2 0 20 Dec 2022
FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration Yangjun Wu Kebin Fang Yao Zhao Hao Zhang Lifeng Shi Mengqi Zhang 34 0 0 09 Nov 2022
Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost Sungjun Cho Seonwoo Min Jinwoo Kim Moontae Lee Honglak Lee Seunghoon Hong 30 3 0 27 Oct 2022
Pure Transformers are Powerful Graph Learners Jinwoo Kim Tien Dat Nguyen Seonwoo Min Sungjun Cho Moontae Lee Honglak Lee Seunghoon Hong 32 187 0 06 Jul 2022
Attention Mechanism in Neural Networks: Where it Comes and Where it Goes Derya Soydaner 3DV 36 149 0 27 Apr 2022
Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention Tong Yu Ruslan Khalitov Lei Cheng Zhirong Yang MoE 21 10 0 22 Apr 2022
Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practice Andreas Grivas Nikolay Bogoychev Adam Lopez 9 9 0 12 Mar 2022
Can Vision Transformers Perform Convolution? Shanda Li Xiangning Chen Di He Cho-Jui Hsieh ViT 24 19 0 02 Nov 2021
A Survey of Transformers Tianyang Lin Yuxin Wang Xiangyang Liu Xipeng Qiu ViT 27 1,084 0 08 Jun 2021
Which transformer architecture fits my data? A vocabulary bottleneck in self-attention Noam Wies Yoav Levine Daniel Jannai Amnon Shashua 40 20 0 09 May 2021
The Depth-to-Width Interplay in Self-Attention Yoav Levine Noam Wies Or Sharir Hofit Bata Amnon Shashua 8 45 0 22 Jun 2020