Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 648 papers shown

Title
Fast Quantum Algorithm for Attention Computation Yeqi Gao Zhao-quan Song Xin Yang Ruizhe Zhang LRM 23 19 0 16 Jul 2023
A Survey of Techniques for Optimizing Transformer Inference Krishna Teja Chitty-Venkata Sparsh Mittal M. Emani V. Vishwanath Arun Somani 31 62 0 16 Jul 2023
Video-FocalNets: Spatio-Temporal Focal Modulation for Video Action Recognition Syed Talal Wasim Muhammad Uzair Khattak Muzammal Naseer Salman Khan M. Shah F. Khan ViT 46 19 0 13 Jul 2023
SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understanding Titouan Parcollet Rogier van Dalen Shucong Zhang S. Bhattacharya 16 6 0 12 Jul 2023
ReLoRA: High-Rank Training Through Low-Rank Updates Vladislav Lialin Namrata Shivagunde Sherin Muckatira Anna Rumshisky BDL 29 93 0 11 Jul 2023
Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni Percy Liang RALM 27 1,389 0 06 Jul 2023
Scaling In-Context Demonstrations with Structured Attention Tianle Cai Kaixuan Huang Jason D. Lee Mengdi Wang LRM 31 8 0 05 Jul 2023
LongNet: Scaling Transformers to 1,000,000,000 Tokens Jiayu Ding Shuming Ma Li Dong Xingxing Zhang Shaohan Huang Wenhui Wang Nanning Zheng Furu Wei CLL 35 151 0 05 Jul 2023
MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers Jakob Drachmann Havtorn Amelie Royer Tijmen Blankevoort B. Bejnordi 25 8 0 05 Jul 2023
Sumformer: Universal Approximation for Efficient Transformers Silas Alberti Niclas Dern L. Thesing Gitta Kutyniok 19 16 0 05 Jul 2023
Learning Feature Matching via Matchable Keypoint-Assisted Graph Neural Network Zizhuo Li Jiayi Ma 27 2 0 04 Jul 2023
ContextSpeech: Expressive and Efficient Text-to-Speech for Paragraph Reading Yujia Xiao Shaofei Zhang Xi Wang Xuejiao Tan Lei He Sheng Zhao Frank Soong Tan Lee 17 5 0 03 Jul 2023
Extending Context Window of Large Language Models via Positional Interpolation Shouyuan Chen Sherman Wong Liangjian Chen Yuandong Tian 12 494 0 27 Jun 2023
LongCoder: A Long-Range Pre-trained Language Model for Code Completion Daya Guo Canwen Xu Nan Duan Jian Yin Julian McAuley 13 77 0 26 Jun 2023
A Multilingual Translator to SQL with Database Schema Pruning to Improve Self-Attention M. A. José Fabio Gagliardi Cozman 13 3 0 25 Jun 2023
LightGlue: Local Feature Matching at Light Speed Philipp Lindenberger Paul-Edouard Sarlin Marc Pollefeys 3DV VLM 14 394 0 23 Jun 2023
Efficient Online Processing with Deep Neural Networks Lukas Hedegaard 18 0 0 23 Jun 2023
Constant Memory Attention Block Leo Feng Frederick Tung Hossein Hajimirsadeghi Yoshua Bengio Mohamed Osama Ahmed 12 0 0 21 Jun 2023
Investigating Pre-trained Language Models on Cross-Domain Datasets, a Step Closer to General AI Mohamad Ballout U. Krumnack Gunther Heidemann Kai-Uwe Kühnberger 18 3 0 21 Jun 2023
Sparse Modular Activation for Efficient Sequence Modeling Liliang Ren Yang Liu Shuohang Wang Yichong Xu Chenguang Zhu Chengxiang Zhai 43 13 0 19 Jun 2023
Block-State Transformers Mahan Fathi Jonathan Pilault Orhan Firat C. Pal Pierre-Luc Bacon Ross Goroshin 23 17 0 15 Jun 2023
GCformer: An Efficient Framework for Accurate and Scalable Long-Term Multivariate Time Series Forecasting Yanjun Zhao Ziqing Ma Tian Zhou Liang Sun M. Ye Yi Qian AI4TS 30 19 0 14 Jun 2023
SqueezeLLM: Dense-and-Sparse Quantization Sehoon Kim Coleman Hooper A. Gholami Zhen Dong Xiuyu Li Sheng Shen Michael W. Mahoney Kurt Keutzer MQ 24 166 0 13 Jun 2023
Augmenting Language Models with Long-Term Memory Weizhi Wang Li Dong Hao Cheng Xiaodong Liu Xifeng Yan Jianfeng Gao Furu Wei KELM RALM 28 83 0 12 Jun 2023
Revisiting Token Pruning for Object Detection and Instance Segmentation Yifei Liu Mathias Gehrig Nico Messikommer Marco Cannici Davide Scaramuzza ViT VLM 37 24 0 12 Jun 2023
$E(2)$ -Equivariant Vision Transformer Renjun Xu Kaifan Yang Ke Liu Fengxiang He ViT MDE 16 9 0 11 Jun 2023
ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer Haoran You Huihong Shi Yipin Guo Yingyan Lin Lin 26 16 0 10 Jun 2023
Multi-level Multiple Instance Learning with Transformer for Whole Slide Image Classification Rui-qi Zhang Qiaozheng Zhang Yingzhuang Liu Hao Xin Y. Liu Xinggang Wang ViT MedIm 27 8 0 08 Jun 2023
Recovering Simultaneously Structured Data via Non-Convex Iteratively Reweighted Least Squares C. Kümmerle J. Maly 17 1 0 08 Jun 2023
InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding Junda Wu Tong Yu Rui Wang Zhao-quan Song Ruiyi Zhang Handong Zhao Chaochao Lu Shuai Li Ricardo Henao VLM 29 22 0 08 Jun 2023
An Efficient Transformer for Simultaneous Learning of BEV and Lane Representations in 3D Lane Detection Ziye Chen K. Smith‐Miles Bo Du G. Qian Mingming Gong ViT 18 8 0 08 Jun 2023
GAT-GAN : A Graph-Attention-based Time-Series Generative Adversarial Network Srikrishna Iyer Teck-Hou Teng AI4TS 11 1 0 03 Jun 2023
RITA: Group Attention is All You Need for Timeseries Analytics Jiaming Liang Lei Cao Samuel Madden Z. Ives Guoliang Li AI4TS 16 0 0 02 Jun 2023
The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles Md Shamim Hussain Mohammed J. Zaki D. Subramanian 37 2 0 02 Jun 2023
Faster Causal Attention Over Large Sequences Through Sparse Flash Attention Matteo Pagliardini Daniele Paliotta Martin Jaggi Franccois Fleuret LRM 15 22 0 01 Jun 2023
Coneheads: Hierarchy Aware Attention Albert Tseng Tao Yu Toni J.B. Liu Chris De Sa 3DPC 9 5 0 01 Jun 2023
Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation Yingyi Chen Qinghua Tao F. Tonin Johan A. K. Suykens 26 19 0 31 May 2023
Recasting Self-Attention with Holographic Reduced Representations Mohammad Mahmudul Alam Edward Raff Stella Biderman Tim Oates James Holt 4 8 0 31 May 2023
Blockwise Parallel Transformer for Large Context Models Hao Liu Pieter Abbeel 41 11 0 30 May 2023
Networked Time Series Imputation via Position-aware Graph Enhanced Variational Autoencoders Dingsu Wang Yuchen Yan Ruizhong Qiu Yada Zhu Kaiyu Guan A. Margenot Hanghang Tong AI4TS 33 27 0 29 May 2023
HyperConformer: Multi-head HyperMixer for Efficient Speech Recognition Florian Mai Juan Pablo Zuluaga Titouan Parcollet P. Motlícek 21 10 0 29 May 2023
COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models Jinqi Xiao Miao Yin Yu Gong Xiao Zang Jian Ren Bo Yuan VLM ViT 30 9 0 26 May 2023
Do We Really Need a Large Number of Visual Prompts? Youngeun Kim Yuhang Li Abhishek Moitra Ruokai Yin Priyadarshini Panda VLM VPVLM 34 5 0 26 May 2023
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time Zichang Liu Aditya Desai Fangshuo Liao Weitao Wang Victor Xie Zhaozhuo Xu Anastasios Kyrillidis Anshumali Shrivastava 23 201 0 26 May 2023
TranSFormer: Slow-Fast Transformer for Machine Translation Bei Li Yi Jing Xu Tan Zhen Xing Tong Xiao Jingbo Zhu 41 7 0 26 May 2023
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers Sotiris Anagnostidis Dario Pavllo Luca Biggio Lorenzo Noci Aurélien Lucchi Thomas Hofmann 32 53 0 25 May 2023
Focus Your Attention (with Adaptive IIR Filters) Shahar Lutati Itamar Zimerman Lior Wolf 32 9 0 24 May 2023
TACR: A Table-alignment-based Cell-selection and Reasoning Model for Hybrid Question-Answering Jian Wu Yicheng Xu Yan Gao Jian-Guang Lou Börje F. Karlsson Manabu Okumura LMTD 13 3 0 24 May 2023
RWKV: Reinventing RNNs for the Transformer Era Bo Peng Eric Alcaide Quentin G. Anthony Alon Albalak Samuel Arcadinho ... Qihang Zhao P. Zhou Qinghua Zhou Jian Zhu Rui-Jie Zhu 72 556 0 22 May 2023
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model Xiao Wang Wei Zhou Qi Zhang Jie Zhou Songyang Gao Junzhe Wang Menghan Zhang Xiang Gao Yunwen Chen Tao Gui 34 7 0 22 May 2023