Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 648 papers shown

Title
Hierarchical Skip Decoding for Efficient Autoregressive Text Generation Yunqi Zhu Xuebing Yang Yuanyuan Wu Wensheng Zhang 26 3 0 22 Mar 2024
vid-TLDR: Training Free Token merging for Light-weight Video Transformer Joonmyung Choi Sanghyeok Lee Jaewon Chu Minhyuk Choi Hyunwoo J. Kim MoMe ViT 42 12 0 20 Mar 2024
TexTile: A Differentiable Metric for Texture Tileability Carlos Rodriguez-Pardo Dan Casas Elena Garces Jorge López-Moreno DiffM 28 4 0 19 Mar 2024
MELTing point: Mobile Evaluation of Language Transformers Stefanos Laskaridis Kleomenis Katevas Lorenzo Minto Hamed Haddadi 27 21 0 19 Mar 2024
HCPM: Hierarchical Candidates Pruning for Efficient Detector-Free Matching Ying Chen Yong-Jin Liu Kai Wu Qiang Nie Shang Xu Huifang Ma Bing Wang Chengjie Wang VLM 40 1 0 19 Mar 2024
NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens Cunxiang Wang Ruoxi Ning Boqi Pan Tonghui Wu Qipeng Guo ... Guangsheng Bao Xiangkun Hu Zheng Zhang Qian Wang Yue Zhang RALM 95 3 0 18 Mar 2024
Unifying Feature and Cost Aggregation with Transformers for Semantic and Visual Correspondence Sung‐Jin Hong Seokju Cho Seungryong Kim Stephen Lin ViT 51 5 0 17 Mar 2024
EfficientMorph: Parameter-Efficient Transformer-Based Architecture for 3D Image Registration Abu Zahid Bin Aziz Mokshagna Sai Teja Karanam Tushar Kataria Shireen Elhabian ViT MedIm 23 1 0 16 Mar 2024
Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers Sanghyeok Lee Joonmyung Choi Hyunwoo J. Kim ViT 36 7 0 15 Mar 2024
Implicit Regularization of Gradient Flow on One-Layer Softmax Attention Heejune Sheen Siyu Chen Tianhao Wang Harrison H. Zhou MLT 33 10 0 13 Mar 2024
StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses Jia-Nan Li Quan Tu Cunli Mao Zhengtao Yu Ji-Rong Wen Rui Yan OffRL 24 3 0 13 Mar 2024
TrafficGPT: Breaking the Token Barrier for Efficient Long Traffic Analysis and Generation Jian Qu Xiaobo Ma Jianfeng Li AI4TS 28 10 0 09 Mar 2024
Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed Yifan Wang Xingyi He He Sida Peng Dongli Tan Xiaowei Zhou 3DV 28 41 0 07 Mar 2024
PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation Junsong Chen Chongjian Ge Enze Xie Yue Wu Lewei Yao Xiaozhe Ren Zhongdao Wang Ping Luo Huchuan Lu Zhenguo Li 128 86 0 07 Mar 2024
Mastering Memory Tasks with World Models Mohammad Reza Samsami Artem Zholus Janarthanan Rajendran Sarath Chandar CLL OffRL 29 21 0 07 Mar 2024
TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax Tobias Christian Nauen Sebastián M. Palacio Andreas Dengel 51 3 0 05 Mar 2024
NiNformer: A Network in Network Transformer with Token Mixing Generated Gating Function Abdullah Nazhat Abdullah Tarkan Aydin 31 0 0 04 Mar 2024
Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures Yuchen Duan Weiyun Wang Zhe Chen Xizhou Zhu Lewei Lu Tong Lu Yu Qiao Hongsheng Li Jifeng Dai Wenhai Wang ViT 38 44 0 04 Mar 2024
ATP: Enabling Fast LLM Serving via Attention on Top Principal Keys Yue Niu Saurav Prakash Salman Avestimehr 21 1 0 01 Mar 2024
RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval Kaiyue Wen Xingyu Dang Kaifeng Lyu 44 24 0 28 Feb 2024
Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling Mahdi Karami Ali Ghodsi VLM 36 6 0 28 Feb 2024
Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study Adrien Bazoge Emmanuel Morin B. Daille P. Gourraud 14 2 0 26 Feb 2024
Trajectory Prediction for Autonomous Driving Using a Transformer Network Zhenning Li Hao Yu 23 0 0 26 Feb 2024
Multimodal Transformer With a Low-Computational-Cost Guarantee Sungjin Park Edward Choi 44 1 0 23 Feb 2024
Linear Transformers are Versatile In-Context Learners Max Vladymyrov J. Oswald Mark Sandler Rong Ge 24 13 0 21 Feb 2024
OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification Yifan Peng Yui Sudo Muhammad Shakeel Shinji Watanabe VLM 35 17 0 20 Feb 2024
`Keep it Together': Enforcing Cohesion in Extractive Summaries by Simulating Human Memory Ronald Cardenas Matthias Shen Shay B. Cohen 8 0 0 16 Feb 2024
Model Compression and Efficient Inference for Large Language Models: A Survey Wenxiao Wang Wei Chen Yicong Luo Yongliu Long Zhengkai Lin Liye Zhang Binbin Lin Deng Cai Xiaofei He MQ 36 46 0 15 Feb 2024
Stochastic Spiking Attention: Accelerating Attention with Stochastic Computing in Spiking Networks Zihang Song Prabodh Katti Osvaldo Simeone Bipin Rajendran 16 2 0 14 Feb 2024
FAST: Factorizable Attention for Speeding up Transformers Armin Gerami Monte Hoover P. S. Dulepet R. Duraiswami 27 0 0 12 Feb 2024
On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference Siyu Ren Kenny Q. Zhu 18 27 0 09 Feb 2024
InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory Chaojun Xiao Pengle Zhang Xu Han Guangxuan Xiao Yankai Lin Zhengyan Zhang Zhiyuan Liu Maosong Sun LLMAG 39 33 0 07 Feb 2024
CAST: Clustering Self-Attention using Surrogate Tokens for Efficient Transformers Adjorn van Engelenhoven Nicola Strisciuglio Estefanía Talavera 21 1 0 06 Feb 2024
Is Mamba Capable of In-Context Learning? Riccardo Grazzi Julien N. Siems Simon Schrodi Thomas Brox Frank Hutter 24 40 0 05 Feb 2024
Arithmetic Feature Interaction Is Necessary for Deep Tabular Learning Yi Cheng Renjun Hu Haochao Ying Xing Shi Jian Wu Wei Lin LMTD 37 8 0 04 Feb 2024
Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models Xindi Wang Mahsa Salmani Parsa Omidi Xiangyu Ren Mehdi Rezagholizadeh A. Eshaghi LRM 29 35 0 03 Feb 2024
Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes Yingyi Chen Qinghua Tao F. Tonin Johan A. K. Suykens 22 1 0 02 Feb 2024
Sequence Shortening for Context-Aware Machine Translation Paweł Mąka Yusuf Can Semerci Jan Scholtes Gerasimos Spanakis 17 2 0 02 Feb 2024
A Manifold Representation of the Key in Vision Transformers Li Meng Morten Goodwin Anis Yazidi P. Engelstad 21 0 0 01 Feb 2024
Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition Lei Liu Li Liu Haizhou Li 21 6 0 31 Jan 2024
SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design Seokju Yun Youngmin Ro ViT 36 29 0 29 Jan 2024
A Comprehensive Survey of Compression Algorithms for Language Models Seungcheol Park Jaehyeon Choi Sojin Lee U. Kang MQ 24 11 0 27 Jan 2024
CascadedGaze: Efficiency in Global Context Extraction for Image Restoration Amirhosein Ghasemabadi Muhammad Kamran Janjua Mohammad Salameh Chunhua Zhou Fengyu Sun Di Niu 24 11 0 26 Jan 2024
Do deep neural networks utilize the weight space efficiently? Onur Can Koyun B. U. Toreyin 16 0 0 26 Jan 2024
SGTR+: End-to-end Scene Graph Generation with Transformer Rongjie Li Songyang Zhang Xuming He ViT 19 2 0 23 Jan 2024
OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning Chu Myaet Thwal Minh N. H. Nguyen Ye Lin Tun Seongjin Kim My T. Thai Choong Seon Hong 49 5 0 22 Jan 2024
With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation Y. Wang D. Ma D. Cai RALM 36 17 0 21 Jan 2024
LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units Zeyu Liu Gourav Datta Anni Li P. Beerel 27 8 0 20 Jan 2024
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model Lianghui Zhu Bencheng Liao Qian Zhang Xinlong Wang Wenyu Liu Xinggang Wang Mamba 32 703 0 17 Jan 2024
The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey Saurav Pawar S.M. Towhidul Islam Tonmoy S. M. M. Zaman Vinija Jain Aman Chadha Amitava Das 29 27 0 15 Jan 2024