Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 648 papers shown

Title
FlexAttention for Efficient High-Resolution Vision-Language Models Junyan Li Delin Chen Tianle Cai Peihao Chen Yining Hong Zhenfang Chen Yikang Shen Chuang Gan VLM 67 4 0 29 Jul 2024
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption Shi Luohe Hongyi Zhang Yao Yao Z. Li Zhao Hai 31 31 0 25 Jul 2024
Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners Yifei Gao Jie Ou Lei Wang Fanhua Shang Jaji Wu MQ 45 0 0 22 Jul 2024
Longhorn: State Space Models are Amortized Online Learners Bo Liu Rui Wang Lemeng Wu Yihao Feng Peter Stone Qian Liu 46 10 0 19 Jul 2024
TorchGT: A Holistic System for Large-scale Graph Transformer Training Mengdie Zhang Jie Sun Qi Hu Peng Sun Zeke Wang Yonggang Wen Tianwei Zhang GNN 39 2 0 19 Jul 2024
Attention in SRAM on Tenstorrent Grayskull Moritz Thüning 22 3 0 18 Jul 2024
Linear-Complexity Self-Supervised Learning for Speech Processing Shucong Zhang Titouan Parcollet Rogier van Dalen Sourav Bhattacharya 33 1 0 18 Jul 2024
MaskVD: Region Masking for Efficient Video Object Detection Sreetama Sarkar Gourav Datta Souvik Kundu Kai Zheng Chirayata Bhattacharyya P. Beerel 25 3 0 16 Jul 2024
Hierarchical Multi-modal Transformer for Cross-modal Long Document Classification Tengfei Liu Yongli Hu Junbin Gao Yanfeng Sun Baocai Yin 26 0 0 14 Jul 2024
Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers Sukjun Hwang Aakash Lahoti Tri Dao Albert Gu Mamba 62 12 0 13 Jul 2024
AI Safety in Generative AI Large Language Models: A Survey Jaymari Chua Yun Yvonna Li Shiyi Yang Chen Wang Lina Yao LM&MA 34 12 0 06 Jul 2024
DGR-MIL: Exploring Diverse Global Representation in Multiple Instance Learning for Whole Slide Image Classification Wenhui Zhu Xiwen Chen Peijie Qiu Aristeidis Sotiras Abolfazl Razi Yalin Wang 32 5 0 04 Jul 2024
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 31 0 0 03 Jul 2024
When big data actually are low-rank, or entrywise approximation of certain function-generated matrices Stanislav Budzinskiy 62 2 0 03 Jul 2024
On the Anatomy of Attention Nikhil Khatri Tuomas Laakkonen Jonathon Liu Vincent Wang-Ma'scianica 3DV 46 1 0 02 Jul 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 34 17 0 01 Jul 2024
Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning Haobo Song Hao Zhao Soumajit Majumder Tao Lin 23 3 0 01 Jul 2024
InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management Wonbeom Lee Jungi Lee Junghwan Seo Jaewoong Sim RALM 26 75 0 28 Jun 2024
Fibottention: Inceptive Visual Representation Learning with Diverse Attention Across Heads Ali Khaleghi Rahimian Manish Kumar Govind Subhajit Maity Dominick Reilly Christian Kummerle Srijan Das A. Dutta 38 1 0 27 Jun 2024
From Efficient Multimodal Models to World Models: A Survey Xinji Mai Zeng Tao Junxiong Lin Haoran Wang Yang Chang Yanlan Kang Yan Wang Wenqiang Zhang 32 5 0 27 Jun 2024
Unveiling and Controlling Anomalous Attention Distribution in Transformers Ruiqing Yan Xingbo Du Haoyu Deng Linghan Zheng Qiuzhuang Sun Jifang Hu Yuhang Shao Penghao Jiang Jinrong Jiang Lian Zhao 36 1 0 26 Jun 2024
Learning Neural Networks with Sparse Activations Pranjal Awasthi Nishanth Dikkala Pritish Kamath Raghu Meka 30 2 0 26 Jun 2024
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers Chao Lou Zixia Jia Zilong Zheng Kewei Tu ODL 31 18 0 24 Jun 2024
Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers Xiuying Wei Skander Moalla Razvan Pascanu Çağlar Gülçehre 22 0 0 24 Jun 2024
Fine-grained Attention in Hierarchical Transformers for Tabular Time-series Raphaël Azorin Zied Ben-Houidi Massimo Gallo A. Finamore Pietro Michiardi AI4TS LMTD 55 0 0 21 Jun 2024
MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression Tianyu Fu Haofeng Huang Xuefei Ning Genghan Zhang Boju Chen ... Shiyao Li Shengen Yan Guohao Dai Huazhong Yang Yu Wang MQ 44 17 0 21 Jun 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 67 13 0 20 Jun 2024
A Primal-Dual Framework for Transformers and Neural Networks Tan M. Nguyen Tam Nguyen Nhat Ho Andrea L. Bertozzi Richard G. Baraniuk Stanley J. Osher ViT 21 13 0 19 Jun 2024
Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis R. Teo Tan M. Nguyen 43 4 0 19 Jun 2024
Language Modeling with Editable External Knowledge Belinda Z. Li Emmy Liu Alexis Ross Abbas Zeitoun Graham Neubig Jacob Andreas KELM 30 4 0 17 Jun 2024
SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention Qianchao Zhu Jiangfei Duan Chang Chen Siran Liu Xiuhong Li ... Huanqi Cao Xiao Chuanfu Xingcheng Zhang Dahua Lin Chao Yang 30 15 0 17 Jun 2024
Vertical LoRA: Dense Expectation-Maximization Interpretation of Transformers Zhuolin Fu 37 0 0 13 Jun 2024
Adaptively Bypassing Vision Transformer Blocks for Efficient Visual Tracking Xiangyang Yang Dan Zeng Xucheng Wang You Wu Hengzhou Ye Qijun Zhao Shuiwang Li 53 3 0 12 Jun 2024
QuickLLaMA: Query-aware Inference Acceleration for Large Language Models Jingyao Li Han Shi Xin Jiang Zhenguo Li Hong Xu Jiaya Jia LRM 33 2 0 11 Jun 2024
ReduceFormer: Attention with Tensor Reduction by Summation John Yang Le An Su Inn Park 26 0 0 11 Jun 2024
What Can We Learn from State Space Models for Machine Learning on Graphs? Yinan Huang Siqi Miao Pan Li 39 7 0 09 Jun 2024
Parameter-Inverted Image Pyramid Networks Xizhou Zhu Xue Yang Zhaokai Wang Hao Li Wenhan Dou Junqi Ge Lewei Lu Yu Qiao Jifeng Dai 47 0 0 06 Jun 2024
Loki: Low-Rank Keys for Efficient Sparse Attention Prajwal Singhania Siddharth Singh Shwai He S. Feizi A. Bhatele 32 13 0 04 Jun 2024
Block Transformer: Global-to-Local Language Modeling for Fast Inference Namgyu Ho Sangmin Bae Taehyeon Kim Hyunjik Jo Yireun Kim Tal Schuster Adam Fisch James Thorne Se-Young Yun 45 8 0 04 Jun 2024
Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations Sarthak Yadav Z. Tan Mamba 29 10 0 04 Jun 2024
SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM Quandong Wang Yuxuan Yuan Xiaoyu Yang Ruike Zhang Kang Zhao Wei Liu Jian Luan Daniel Povey Bin Wang 41 0 0 03 Jun 2024
Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification Jungmin Yun Mihyeon Kim Youngbin Kim 69 9 0 03 Jun 2024
Automatic Channel Pruning for Multi-Head Attention Eunho Lee Youngbae Hwang ViT 32 1 0 31 May 2024
Optimizing Foundation Model Inference on a Many-tiny-core Open-source RISC-V Platform Viviane Potocnik Luca Colagrande Tim Fischer L. Bertaccini Daniele Jahier Pagliari Alessio Burrello Luca Benini 23 3 0 29 May 2024
SFANet: Spatial-Frequency Attention Network for Weather Forecasting Jiaze Wang Hao Chen Hongcan Xu Jinpeng Li Bo-Lan Wang Kun Shao Furui Liu Huaxi Chen Guangyong Chen Pheng-Ann Heng 56 0 0 29 May 2024
Wavelet-Based Image Tokenizer for Vision Transformers Zhenhai Zhu Radu Soricut ViT 37 3 0 28 May 2024
Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass Ethan Shen Alan Fan Sarah M Pratt Jae Sung Park Matthew Wallingford Sham Kakade Ari Holtzman Ranjay Krishna Ali Farhadi Aditya Kusupati 35 2 0 28 May 2024
Matryoshka Multimodal Models Mu Cai Jianwei Yang Jianfeng Gao Yong Jae Lee VLM 39 25 0 27 May 2024
SMR: State Memory Replay for Long Sequence Modeling Biqing Qi Junqi Gao Kaiyan Zhang Dong Li Jianxing Liu Ligang Wu Bowen Zhou 23 5 0 27 May 2024
SWAT: Scalable and Efficient Window Attention-based Transformers Acceleration on FPGAs Zhenyu Bai Pranav Dangi Huize Li Tulika Mitra 29 5 0 27 May 2024