Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 648 papers shown

Title
ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound Yan-Bo Lin Jie Lei Mohit Bansal Gedas Bertasius 31 39 0 06 Apr 2022
TubeDETR: Spatio-Temporal Video Grounding with Transformers Antoine Yang Antoine Miech Josef Sivic Ivan Laptev Cordelia Schmid ViT 28 94 0 30 Mar 2022
Fine-tuning Image Transformers using Learnable Memory Mark Sandler A. Zhmoginov Max Vladymyrov Andrew Jackson ViT 15 47 0 29 Mar 2022
FS6D: Few-Shot 6D Pose Estimation of Novel Objects Yisheng He Yao Wang Haoqiang Fan Jian-jun Sun Qifeng Chen 10 83 0 28 Mar 2022
Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection Xin Huang A. Khetan Rene Bidart Zohar S. Karnin 17 14 0 27 Mar 2022
Diagonal State Spaces are as Effective as Structured State Spaces Ankit Gupta Albert Gu Jonathan Berant 34 290 0 27 Mar 2022
Error Correction Code Transformer Yoni Choukroun Lior Wolf 19 47 0 27 Mar 2022
A Survey on Aspect-Based Sentiment Classification Gianni Brauwers Flavius Frasincar LLMAG 24 109 0 27 Mar 2022
A General Survey on Attention Mechanisms in Deep Learning Gianni Brauwers Flavius Frasincar 23 295 0 27 Mar 2022
Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustness Giulio Lovisotto Nicole Finnie Mauricio Muñoz Chaithanya Kumar Mummadi J. H. Metzen AAML ViT 17 32 0 25 Mar 2022
Linearizing Transformer with Key-Value Memory Yizhe Zhang Deng Cai 20 5 0 23 Mar 2022
Open-Vocabulary DETR with Conditional Matching Yuhang Zang Wei Li Kaiyang Zhou Chen Huang Chen Change Loy ObjD VLM 20 197 0 22 Mar 2022
Local-Global Context Aware Transformer for Language-Guided Video Segmentation Chen Liang Wenguan Wang Tianfei Zhou Jiaxu Miao Yawei Luo Yi Yang VOS 24 74 0 18 Mar 2022
Block-Recurrent Transformers DeLesley S. Hutchins Imanol Schlag Yuhuai Wu Ethan Dyer Behnam Neyshabur 16 94 0 11 Mar 2022
Towards Self-Supervised Category-Level Object Pose and Size Estimation Yisheng He Haoqiang Fan Haibin Huang Qifeng Chen Jian-jun Sun 11 17 0 06 Mar 2022
FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours Shenggan Cheng Xuanlei Zhao Guangyang Lu Bin-Rui Li Zhongming Yu Tian Zheng R. Wu Xiwen Zhang Jian Peng Yang You AI4CE 17 30 0 02 Mar 2022
A Data-scalable Transformer for Medical Image Segmentation: Architecture, Model Efficiency, and Benchmark Yunhe Gao Mu Zhou Ding Liu Zhennan Yan Shaoting Zhang Dimitris N. Metaxas ViT MedIm 18 68 0 28 Feb 2022
Dynamic N:M Fine-grained Structured Sparse Attention Mechanism Zhaodong Chen Yuying Quan Zheng Qu L. Liu Yufei Ding Yuan Xie 22 21 0 28 Feb 2022
NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better Chuhan Wu Fangzhao Wu Tao Qi Yongfeng Huang Xing Xie 25 58 0 24 Feb 2022
Transformer Quality in Linear Time Weizhe Hua Zihang Dai Hanxiao Liu Quoc V. Le 71 222 0 21 Feb 2022
cosFormer: Rethinking Softmax in Attention Zhen Qin Weixuan Sun Huicai Deng Dongxu Li Yunshen Wei Baohong Lv Junjie Yan Lingpeng Kong Yiran Zhong 21 211 0 17 Feb 2022
ActionFormer: Localizing Moments of Actions with Transformers Chen-Da Liu-Zhang Jianxin Wu Yin Li ViT 23 328 0 16 Feb 2022
General-purpose, long-context autoregressive modeling with Perceiver AR Curtis Hawthorne Andrew Jaegle Cătălina Cangea Sebastian Borgeaud C. Nash ... Hannah R. Sheahan Neil Zeghidour Jean-Baptiste Alayrac João Carreira Jesse Engel 35 65 0 15 Feb 2022
CATs++: Boosting Cost Aggregation with Convolutions and Transformers Seokju Cho Sunghwan Hong Seung Wook Kim ViT 19 34 0 14 Feb 2022
Exploiting Spatial Sparsity for Event Cameras with Visual Transformers Zuowen Wang Yuhuang Hu Shih-Chii Liu ViT 20 33 0 10 Feb 2022
Structure-Aware Transformer for Graph Representation Learning Dexiong Chen Leslie O’Bray Karsten M. Borgwardt 26 236 0 07 Feb 2022
Improving Sample Efficiency of Value Based Models Using Attention and Vision Transformers Amir Ardalan Kalantari Mohammad Amini Sarath Chandar Doina Precup 44 4 0 01 Feb 2022
Continual Transformers: Redundancy-Free Attention for Online Inference Lukas Hedegaard Arian Bakhtiarnia Alexandros Iosifidis CLL 14 11 0 17 Jan 2022
QuadTree Attention for Vision Transformers Shitao Tang Jiahui Zhang Siyu Zhu Ping Tan ViT 157 156 0 08 Jan 2022
Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networks Lei Cheng Ruslan Khalitov Tong Yu Zhirong Yang 20 32 0 06 Jan 2022
Learning Operators with Coupled Attention Georgios Kissas Jacob H. Seidman Leonardo Ferreira Guilhoto V. Preciado George J. Pappas P. Perdikaris 24 107 0 04 Jan 2022
Full Transformer Framework for Robust Point Cloud Registration with Deep Information Interaction Guang-Sheng Chen Meiling Wang Yufeng Yue Qingxiang Zhang Li-xin Yuan ViT 32 17 0 17 Dec 2021
Neural Architectures for Biological Inter-Sentence Relation Extraction Enrique Noriega-Atala Peter Lovett Clayton T. Morrison Mihai Surdeanu NAI 25 3 0 17 Dec 2021
LongT5: Efficient Text-To-Text Transformer for Long Sequences Mandy Guo Joshua Ainslie David C. Uthus Santiago Ontanon Jianmo Ni Yun-hsuan Sung Yinfei Yang VLM 31 307 0 15 Dec 2021
Discourse-Aware Soft Prompting for Text Generation Marjan Ghazvininejad Vladimir Karpukhin Vera Gor Asli Celikyilmaz 23 6 0 10 Dec 2021
Self-attention Does Not Need $O(n^2)$ Memory M. Rabe Charles Staats LRM 18 139 0 10 Dec 2021
FaceFormer: Speech-Driven 3D Facial Animation with Transformers Yingruo Fan Zhaojiang Lin Jun Saito Wenping Wang Taku Komura CVBM 38 194 0 10 Dec 2021
Attention-Based Model and Deep Reinforcement Learning for Distribution of Event Processing Tasks A. Mazayev F. Al-Tam N. Correia 21 5 0 07 Dec 2021
STJLA: A Multi-Context Aware Spatio-Temporal Joint Linear Attention Network for Traffic Forecasting Yuchen Fang Yanjun Qin Haiyong Luo Fang Zhao Chenxing Wang GNN AI4TS 19 1 0 04 Dec 2021
Linear algebra with transformers Franccois Charton AIMat 27 56 0 03 Dec 2021
OCR-free Document Understanding Transformer Geewook Kim Teakgyu Hong Moonbin Yim Jeongyeon Nam Jinyoung Park Jinyeong Yim Wonseok Hwang Sangdoo Yun Dongyoon Han Seunghyun Park ViT 44 262 0 30 Nov 2021
Comparative Study of Long Document Classification Vedangi Wagh Snehal Khandve Isha Joshi Apurva Wani Geetanjali Kale Raviraj Joshi 22 25 0 01 Nov 2021
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training Yongbin Li Hongxin Liu Zhengda Bian Boxiang Wang Haichen Huang Fan Cui Chuan-Qing Wang Yang You GNN 11 143 0 28 Oct 2021
VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization Mucong Ding Kezhi Kong Jingling Li Chen Zhu John P. Dickerson Furong Huang Tom Goldstein GNN MQ 25 47 0 27 Oct 2021
SOFT: Softmax-free Transformer with Linear Complexity Jiachen Lu Jinghan Yao Junge Zhang Martin Danelljan Hang Xu Weiguo Gao Chunjing Xu Thomas B. Schon Li Zhang 16 161 0 22 Oct 2021
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models Qinyuan Ye Madian Khabsa M. Lewis Sinong Wang Xiang Ren Aaron Jaech 29 5 0 16 Oct 2021
Speech Summarization using Restricted Self-Attention Roshan S. Sharma Shruti Palaskar A. Black Florian Metze 17 33 0 12 Oct 2021
Token Pooling in Vision Transformers D. Marin Jen-Hao Rick Chang Anurag Ranjan Anish K. Prabhu Mohammad Rastegari Oncel Tuzel ViT 65 66 0 08 Oct 2021
Efficient and Private Federated Learning with Partially Trainable Networks Hakim Sidahmed Zheng Xu Ankush Garg Yuan Cao Mingqing Chen FedML 47 13 0 06 Oct 2021
Ripple Attention for Visual Perception with Sub-quadratic Complexity Lin Zheng Huijie Pan Lingpeng Kong 21 3 0 06 Oct 2021