Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 645 papers shown

Title
TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text Songshuo Lu Hua Wang Yutian Rong Zhi Chen Yaohua Tang VLM 31 12 0 10 Oct 2024
Unlocking Real-Time Fluorescence Lifetime Imaging: Multi-Pixel Parallelism for FPGA-Accelerated Processing Ismail Erbas Aporva Amarnath Vikas Pandey Karthik Swaminathan Naigang Wang Xavier Intes 26 1 0 09 Oct 2024
InAttention: Linear Context Scaling for Transformers Joseph Eisner 21 0 0 09 Oct 2024
Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity Mutian He Philip N. Garner 80 0 0 09 Oct 2024
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 31 7 0 08 Oct 2024
EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment Yifei Xing Xiangyuan Lan Ruiping Wang D. Jiang Wenjun Huang Qingfang Zheng Yaowei Wang Mamba 33 0 0 08 Oct 2024
LevAttention: Time, Space, and Streaming Efficient Algorithm for Heavy Attentions R. Kannan Chiranjib Bhattacharyya Praneeth Kacham David P. Woodruff 23 0 0 07 Oct 2024
NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping Yamin Li Ange Lou Ziyuan Xu Shengchao Zhang Shiyu Wang Dario J. Englot Soheil Kolouri Daniel Moyer Roza G. Bayrak Catie Chang 20 4 0 07 Oct 2024
Fundamental Limitations on Subquadratic Alternatives to Transformers Josh Alman Hantao Yu 23 1 0 05 Oct 2024
S7: Selective and Simplified State Space Layers for Sequence Modeling Taylan Soydan Nikola Zubić Nico Messikommer Siddhartha Mishra Davide Scaramuzza 35 4 0 04 Oct 2024
ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI Ahmad Elawady Gunjan Chhablani Ram Ramrakhya Karmesh Yadav Dhruv Batra Z. Kira Andrew Szot OffRL 28 0 0 03 Oct 2024
UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation Zixuan Li Jing Xiong Fanghua Ye Chuanyang Zheng Xun Wu ... Xiaodan Liang Chengming Li Zhenan Sun Lingpeng Kong Ngai Wong RALM UQLM 27 2 0 03 Oct 2024
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration Jintao Zhang Jia wei Pengle Zhang Jun-Jie Zhu Jun Zhu Jianfei Chen VLM MQ 82 18 0 03 Oct 2024
On The Adaptation of Unlimiformer for Decoder-Only Transformers Kian Ahrabian Alon Benhaim Barun Patra Jay Pujara Saksham Singhal Xia Song 38 0 0 02 Oct 2024
STGformer: Efficient Spatiotemporal Graph Transformer for Traffic Forecasting Hongjun Wang Jiyuan Chen Tong Pan Zheng Dong Lingyu Zhang Renhe Jiang Xuan Song AI4TS GNN 37 0 0 01 Oct 2024
GLMHA A Guided Low-rank Multi-Head Self-Attention for Efficient Image Restoration and Spectral Reconstruction Zaid Ilyas Naveed Akhtar David Suter Syed Zulqarnain Gilani 15 0 0 01 Oct 2024
Cottention: Linear Transformers With Cosine Attention Gabriel Mongaras Trevor Dohm Eric C. Larson 24 0 0 27 Sep 2024
Treating Brain-inspired Memories as Priors for Diffusion Model to Forecast Multivariate Time Series Muyao Wang Wenchao Chen Zhibin Duan Bo Chen AI4TS DiffM 34 0 0 27 Sep 2024
dnaGrinder: a lightweight and high-capacity genomic foundation model Qihang Zhao Chi Zhang Weixiong Zhang 26 0 0 24 Sep 2024
CSPS: A Communication-Efficient Sequence-Parallelism based Serving System for Transformer based Models with Long Prompts Zeyu Zhang Haiying Shen VLM 24 0 0 23 Sep 2024
Efficiently Dispatching Flash Attention For Partially Filled Attention Masks Agniv Sharma Jonas Geiping 22 0 0 23 Sep 2024
CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMs Junlin Lv Yuan Feng Xike Xie Xin Jia Qirong Peng Guiming Xie 21 3 0 19 Sep 2024
Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillation Rui Yu Runkai Zhao Jiagen Li Qingsong Zhao Songhao Zhu HuaiCheng Yan Meng Wang Mamba 29 3 0 17 Sep 2024
Mamba-ST: State Space Model for Efficient Style Transfer Filippo Botti Alex Ergasti Leonardo Rossi Tomaso Fontanini Claudio Ferrari Massimo Bertozzi Andrea Prati Mamba 42 2 0 16 Sep 2024
A framework for measuring the training efficiency of a neural architecture Eduardo Cueto-Mendoza John D. Kelleher 38 0 0 12 Sep 2024
DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models Maryam Akhavan Aghdam Hongpeng Jin Yanzhao Wu MoE 18 3 0 10 Sep 2024
AttentionX: Exploiting Consensus Discrepancy In Attention from A Distributed Optimization Perspective Guoqiang Zhang Richard Heusdens 29 0 0 06 Sep 2024
MVTN: A Multiscale Video Transformer Network for Hand Gesture Recognition Mallika Garg Debashis Ghosh P. M. Pradhan ViT 28 1 0 05 Sep 2024
LowFormer: Hardware Efficient Design for Convolutional Transformer Backbones Moritz Nottebaum Matteo Dunnhofer C. Micheloni ViT 29 1 0 05 Sep 2024
An Analysis of Linear Complexity Attention Substitutes with BEST-RQ Ryan Whetten Titouan Parcollet Adel Moumen Marco Dinarelli Yannick Esteve 22 0 0 04 Sep 2024
On the design space between molecular mechanics and machine learning force fields Yuanqing Wang Kenichiro Takaba Michael S. Chen Marcus Wieder Yuzhi Xu ... Kyunghyun Cho Joe G. Greener Peter K. Eastman Stefano Martiniani M. Tuckerman AI4CE 37 4 0 03 Sep 2024
MemLong: Memory-Augmented Retrieval for Long Text Modeling Weijie Liu Zecheng Tang Juntao Li Kehai Chen Min Zhang RALM 17 2 0 30 Aug 2024
Autoregressive model path dependence near Ising criticality Yi Hong Teoh R. Melko AI4CE 25 1 0 28 Aug 2024
An alternative formulation of attention pooling function in translation Eddie Conti 26 0 0 23 Aug 2024
Multi-modal Intermediate Feature Interaction AutoEncoder for Overall Survival Prediction of Esophageal Squamous Cell Cancer Chengyu Wu Yatao Zhang Yaqi Wang Qifeng Wang Shuai Wang 11 0 0 23 Aug 2024
Macformer: Transformer with Random Maclaurin Feature Attention Yuhan Guo Lizhong Ding Ye Yuan Guoren Wang 46 0 0 21 Aug 2024
OccMamba: Semantic Occupancy Prediction with State Space Models Heng Li Yuenan Hou Xiaohan Xing Xiao Sun Xiao Sun Yanyong Zhang Mamba 48 3 0 19 Aug 2024
HySem: A context length optimized LLM pipeline for unstructured tabular extraction Narayanan PP A. P. N. Iyer 36 0 0 18 Aug 2024
ELASTIC: Efficient Linear Attention for Sequential Interest Compression Jiaxin Deng Shiyao Wang Song Lu Yinfeng Li Xinchen Luo Yuanjun Liu Peixing Xu Guorui Zhou 39 0 0 18 Aug 2024
Linear Attention is Enough in Spatial-Temporal Forecasting Xinyu Ning AI4TS 27 0 0 17 Aug 2024
Ex3: Automatic Novel Writing by Extracting, Excelsior and Expanding Lei Huang Jiaming Guo Guanhua He Xishan Zhang Rui Zhang Shaohui Peng Shaoli Liu Tianshi Chen 26 2 0 16 Aug 2024
Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discovery Yue Yu Ning Liu Fei Lu Tian Gao S. Jafarzadeh Stewart Silling AI4CE 41 7 0 14 Aug 2024
DyG-Mamba: Continuous State Space Modeling on Dynamic Graphs Dongyuan Li Shiyin Tan Ying Zhang Ming Jin Shirui Pan Manabu Okumura Renhe Jiang Mamba 18 2 0 13 Aug 2024
MetMamba: Regional Weather Forecasting with Spatial-Temporal Mamba Model Haoyu Qin Yungang Chen Qianchuan Jiang Pengchao Sun Xiancai Ye Chao Lin Mamba AI4CE 31 1 0 12 Aug 2024
Post-Training Sparse Attention with Double Sparsity Shuo Yang Ying Sheng Joseph E. Gonzalez Ion Stoica Lianmin Zheng 28 7 0 11 Aug 2024
Sampling Foundational Transformer: A Theoretical Perspective Viet Anh Nguyen Minh Lenhat Khoa Nguyen Duong Duc Hieu Dao Huu Hung Truong Son-Hy 42 0 0 11 Aug 2024
SAMSA: Efficient Transformer for Many Data Modalities Minh Lenhat Viet Anh Nguyen Khoa Nguyen Duong Duc Hieu Dao Huu Hung Truong Son-Hy 46 0 0 10 Aug 2024
DeMansia: Mamba Never Forgets Any Tokens Ricky Fang Mamba 19 0 0 04 Aug 2024
Cross-layer Attention Sharing for Large Language Models Yongyu Mu Yuzhang Wu Yuchun Fan Chenglong Wang Hengyu Li Qiaozhi He Murun Yang Tong Xiao Jingbo Zhu 36 5 0 04 Aug 2024
ThinK: Thinner Key Cache by Query-Driven Pruning Yuhui Xu Zhanming Jie Hanze Dong Lei Wang Xudong Lu Aojun Zhou Amrita Saha Caiming Xiong Doyen Sahoo 67 14 0 30 Jul 2024