Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 648 papers shown

Title
FIT: Far-reaching Interleaved Transformers Ting-Li Chen Lala Li 19 12 0 22 May 2023
CageViT: Convolutional Activation Guided Efficient Vision Transformer Hao Zheng Jinbao Wang Xiantong Zhen H. Chen Jingkuan Song Feng Zheng ViT 10 0 0 17 May 2023
The emergence of clusters in self-attention dynamics Borjan Geshkovski Cyril Letrouit Yury Polyanskiy Philippe Rigollet 22 46 0 09 May 2023
The Role of Global and Local Context in Named Entity Recognition Arthur Amalvy Vincent Labatut Richard Dufour 38 4 0 04 May 2023
Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? Sonal Sannigrahi Josef van Genabith C. España-Bonet AILaw 34 4 0 28 Apr 2023
SCM: Enhancing Large Language Model with Self-Controlled Memory Framework Bin Wang Xinnian Liang Jian Yang Huijia Huang Shuangzhi Wu Peihao Wu Lu Lu Zejun Ma Zhoujun Li LLMAG KELM RALM 94 25 0 26 Apr 2023
Improving Autoregressive NLP Tasks via Modular Linearized Attention Victor Agostinelli Lizhong Chen 22 1 0 17 Apr 2023
SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers Alberto Marchisio David Durà Maurizio Capra Maurizio Martina Guido Masera Muhammad Shafique 21 17 0 08 Apr 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 24 39 0 07 Apr 2023
DIR-AS: Decoupling Individual Identification and Temporal Reasoning for Action Segmentation Peiyao Wang Haibin Ling 15 2 0 04 Apr 2023
Dialogue-Contextualized Re-ranking for Medical History-Taking Jian Zhu Ilya Valmianski Anitha Kannan 19 1 0 04 Apr 2023
Dual-Attention Neural Transducers for Efficient Wake Word Spotting in Speech Recognition Saumya Yashmohini Sahai Jing Liu Thejaswi Muniyappa Kanthashree Mysore Sathyendra Anastasios Alexandridis ... Ross McGowan Ariya Rastrow Feng-Ju Chang Athanasios Mouchtaris Siegfried Kunzmann 31 5 0 03 Apr 2023
SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications Abdelrahman M. Shaker Muhammad Maaz H. Rasheed Salman Khan Ming Yang F. Khan ViT 35 83 0 27 Mar 2023
You Only Segment Once: Towards Real-Time Panoptic Segmentation Jie Hu Linyan Huang Tianhe Ren Shengchuan Zhang Rongrong Ji Liujuan Cao SSeg 44 54 0 26 Mar 2023
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization Pavan Kumar Anasosalu Vasu J. Gabriel Jeff J. Zhu Oncel Tuzel Anurag Ranjan ViT 26 149 0 24 Mar 2023
EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms Shikhar Tuli N. Jha 34 3 0 24 Mar 2023
The Multiscale Surface Vision Transformer Simon Dahan Logan Z. J. Williams Daniel Rueckert E. C. Robinson MedIm ViT 10 2 0 21 Mar 2023
Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers Jaehoon Yoo Semin Kim Doyup Lee Chiheon Kim Seunghoon Hong 21 3 0 20 Mar 2023
HDformer: A Higher Dimensional Transformer for Diabetes Detection Utilizing Long Range Vascular Signals Ella Lan MedIm 20 1 0 17 Mar 2023
Efficiency 360: Efficient Vision Transformers Badri N. Patro Vijay Srinivas Agneeswaran 21 6 0 16 Feb 2023
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment Jared Fernandez Jacob Kahn Clara Na Yonatan Bisk Emma Strubell FedML 25 10 0 13 Feb 2023
Efficient Attention via Control Variates Lin Zheng Jianbo Yuan Chong-Jun Wang Lingpeng Kong 24 18 0 09 Feb 2023
Efficient Joint Learning for Clinical Named Entity Recognition and Relation Extraction Using Fourier Networks: A Use Case in Adverse Drug Events A. Yazdani D. Proios H. Rouhizadeh Douglas Teodoro 19 7 0 08 Feb 2023
Single Cells Are Spatial Tokens: Transformers for Spatial Transcriptomic Data Imputation Haifang Wen Wenzhuo Tang Wei Jin Jiayuan Ding Renming Liu Xinnan Dai Feng Shi Lulu Shang Jiliang Tang Yuying Xie 27 8 0 06 Feb 2023
Mnemosyne: Learning to Train Transformers with Transformers Deepali Jain K. Choromanski Kumar Avinava Dubey Sumeet Singh Vikas Sindhwani Tingnan Zhang Jie Tan OffRL 31 9 0 02 Feb 2023
Exploring Attention Map Reuse for Efficient Transformer Neural Networks Kyuhong Shim Jungwook Choi Wonyong Sung ViT 17 3 0 29 Jan 2023
On the Connection Between MPNN and Graph Transformer Chen Cai Truong Son-Hy Rose Yu Yusu Wang 28 50 0 27 Jan 2023
Effective End-to-End Vision Language Pretraining with Semantic Visual Loss Xiaofeng Yang Fayao Liu Guosheng Lin VLM 19 7 0 18 Jan 2023
Dynamic Grained Encoder for Vision Transformers Lin Song Songyang Zhang Songtao Liu Zeming Li Xuming He Hongbin Sun Jian-jun Sun Nanning Zheng ViT 21 34 0 10 Jan 2023
Does compressing activations help model parallel training? S. Bian Dacheng Li Hongyi Wang Eric P. Xing Shivaram Venkataraman 19 4 0 06 Jan 2023
Multi-Stage Spatio-Temporal Aggregation Transformer for Video Person Re-identification Ziyi Tang Ruimao Zhang Zhanglin Peng Jinrui Chen Liang Lin 12 18 0 02 Jan 2023
Hungry Hungry Hippos: Towards Language Modeling with State Space Models Daniel Y. Fu Tri Dao Khaled Kamal Saab A. Thomas Atri Rudra Christopher Ré 53 368 0 28 Dec 2022
A Length-Extrapolatable Transformer Yutao Sun Li Dong Barun Patra Shuming Ma Shaohan Huang Alon Benhaim Vishrav Chaudhary Xia Song Furu Wei 24 115 0 20 Dec 2022
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 120 36 0 15 Dec 2022
UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentation Abdelrahman M. Shaker Muhammad Maaz H. Rasheed Salman Khan Ming Yang F. Khan MedIm 24 129 0 08 Dec 2022
Deep neural network techniques for monaural speech enhancement: state of the art analysis P. Ochieng 28 21 0 01 Dec 2022
Lightweight Structure-Aware Attention for Visual Understanding Heeseung Kwon F. M. Castro M. Marín-Jiménez N. Guil Alahari Karteek 26 2 0 29 Nov 2022
Dynamic Feature Pruning and Consolidation for Occluded Person Re-Identification Yuteng Ye Hang Zhou Jiale Cai Chenxing Gao Youjia Zhang Junle Wang Qiang Hu Junqing Yu Wei Yang 23 6 0 27 Nov 2022
TetraDiffusion: Tetrahedral Diffusion Models for 3D Shape Generation Nikolai Kalischek T. Peters Jan Dirk Wegner Konrad Schindler DiffM 19 12 0 23 Nov 2022
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention Zineng Tang Jaemin Cho Jie Lei Mohit Bansal VLM 16 9 0 21 Nov 2022
Efficient Transformers with Dynamic Token Pooling Piotr Nawrot J. Chorowski Adrian Lañcucki E. Ponti 6 42 0 17 Nov 2022
Learning to Kindle the Starlight Yu Yuan Jiaqi Wu Lindong Wang Zhongliang Jing H. Leung Shuyuan Zhu Han Pan DiffM 19 3 0 16 Nov 2022
Token Turing Machines Michael S. Ryoo K. Gopalakrishnan Kumara Kahatapitiya Ted Xiao Kanishka Rao Austin Stone Yao Lu Julian Ibarz Anurag Arnab 27 21 0 16 Nov 2022
ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention Jyotikrishna Dass Shang Wu Huihong Shi Chaojian Li Zhifan Ye Zhongfeng Wang Yingyan Lin 17 49 0 09 Nov 2022
Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost Sungjun Cho Seonwoo Min Jinwoo Kim Moontae Lee Honglak Lee Seunghoon Hong 30 3 0 27 Oct 2022
A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives Carlos Hernandez-Olivan Javier Hernandez-Olivan J. R. Beltrán MGen 32 6 0 25 Oct 2022
Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation Botao Yu Peiling Lu Rui Wang Wei Hu Xu Tan Wei Ye Shikun Zhang Tao Qin Tie-Yan Liu MGen 25 54 0 19 Oct 2022
The Devil in Linear Transformer Zhen Qin Xiaodong Han Weixuan Sun Dongxu Li Lingpeng Kong Nick Barnes Yiran Zhong 34 70 0 19 Oct 2022
Token Merging: Your ViT But Faster Daniel Bolya Cheng-Yang Fu Xiaoliang Dai Peizhao Zhang Christoph Feichtenhofer Judy Hoffman MoMe 23 417 0 17 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 39 9 0 14 Oct 2022