Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 635 papers shown

Title
Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain Hyowon Wi Jeongwhan Choi Noseong Park 18 0 0 13 May 2025
Hierarchical Sparse Attention Framework for Computationally Efficient Classification of Biological Cells Elad Yoshai Dana Yagoda-Aharoni Eden Dotan N. Shaked 13 0 0 12 May 2025
Graph Laplacian Wavelet Transformer via Learnable Spectral Decomposition Andrew Kiruluta Eric Lundy Priscilla Burity 24 0 0 09 May 2025
Accurate and Efficient Multivariate Time Series Forecasting via Offline Clustering Yiming Niu Jinliang Deng L. Zhang Zimu Zhou Yongxin Tong AI4TS 21 0 0 09 May 2025
T-T: Table Transformer for Tagging-based Aspect Sentiment Triplet Extraction Kun Peng Chaodong Tong Cong Cao Hao Peng Q. Li Guanlin Wu Lei Jiang Yanbing Liu Philip S. Yu LMTD 46 0 0 08 May 2025
Image Recognition with Online Lightweight Vision Transformer: A Survey Zherui Zhang Rongtao Xu Jie Zhou Changwei Wang Xingtian Pei ... Jiguang Zhang Li Guo Longxiang Gao W. Xu Shibiao Xu ViT 98 0 0 06 May 2025
From Attention to Atoms: Spectral Dictionary Learning for Fast, Interpretable Language Models Andrew Kiruluta 24 0 0 29 Apr 2025
DYNAMAX: Dynamic computing for Transformers and Mamba based architectures Miguel Nogales Matteo Gambella Manuel Roveri 56 0 0 29 Apr 2025
An Empirical Study on Prompt Compression for Large Language Models Z. Zhang Jinyi Li Yihuai Lan X. Wang Hao Wang MQ 42 0 0 24 Apr 2025
Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light Ali Hassani Fengzhe Zhou Aditya Kane Jiannan Huang Chieh-Yun Chen ... Bing Xu Haicheng Wu Wen-mei W. Hwu Ming-Yu Liu Humphrey Shi 26 0 0 23 Apr 2025
Landmark-Free Preoperative-to-Intraoperative Registration in Laparoscopic Liver Resection Jun Zhou Bingchen Gao Kai Wang Jialun Pei Pheng-Ann Heng Jing Qin MedIm 32 0 0 21 Apr 2025
ECViT: Efficient Convolutional Vision Transformer with Local-Attention and Multi-scale Stages Zhoujie Qian ViT 24 0 0 21 Apr 2025
CacheFormer: High Attention-Based Segment Caching Sushant Singh A. Mahmood 31 0 0 18 Apr 2025
Packing Input Frame Context in Next-Frame Prediction Models for Video Generation Lvmin Zhang Maneesh Agrawala DiffM VGen 70 0 0 17 Apr 2025
MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models Junyang Zhang Tianyi Zhu Cheng Luo A. Anandkumar RALM 42 0 0 16 Apr 2025
A Review of YOLOv12: Attention-Based Enhancements vs. Previous Versions Rahima Khanam Muhammad Hussain 31 0 0 16 Apr 2025
Millions of States: Designing a Scalable MoE Architecture with RWKV-7 Meta-learner Liu Xiao Li Zhiyuan Lin Yueyu 31 0 0 11 Apr 2025
Learnable Multi-Scale Wavelet Transformer: A Novel Alternative to Self-Attention Andrew Kiruluta Priscilla Burity Samantha Williams 25 3 0 08 Apr 2025
Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision Xiaofeng Han Shunpeng Chen Zenghuang Fu Zhe Feng Lue Fan ... Li Guo Weiliang Meng Xiaopeng Zhang Rongtao Xu Shibiao Xu 63 1 0 03 Apr 2025
FT-Transformer: Resilient and Reliable Transformer with End-to-End Fault Tolerant Attention Huangliang Dai Shixun Wu Hairui Zhao Jiajun Huang Zizhe Jian Yue Zhu Haiyang Hu Zizhong Chen 46 0 0 03 Apr 2025
TransMamba: Flexibly Switching between Transformer and Mamba Yixing Li Ruobing Xie Zhen Yang X. Sun Shuaipeng Li ... Zhanhui Kang Yu Cheng C. Xu Di Wang Jie Jiang Mamba 65 1 0 31 Mar 2025
ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Models Guoyizhe Wei Rama Chellappa 31 0 0 30 Mar 2025
Function Fitting Based on Kolmogorov-Arnold Theorem and Kernel Functions Jianpeng Liu Qizhi Pan 30 0 0 29 Mar 2025
Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendation Jiakai Tang Sunhao Dai Teng Shi Jun Xu X. Chen Wen Chen Wu Jian Yuning Jiang LRM 63 5 0 28 Mar 2025
Burst Image Super-Resolution with Mamba Ozan Unal Steven Marty Dengxin Dai Mamba 43 0 0 25 Mar 2025
Selecting and Pruning: A Differentiable Causal Sequentialized State-Space Model for Two-View Correspondence Learning Xiang Fang S. Zhang Hao Zhang Tao Lu Huabing Zhou Jiayi Ma Mamba 75 0 0 23 Mar 2025
Fractal-IR: A Unified Framework for Efficient and Scalable Image Restoration Yawei Li Bin Ren Jingyun Liang Rakesh Ranjan Mengyuan Liu N. Sebe Ming-Hsuan Yang Luca Benini 56 0 0 22 Mar 2025
From S4 to Mamba: A Comprehensive Survey on Structured State Space Models Shriyank Somvanshi Md Monzurul Islam Mahmuda Sultana Mimi Sazzad Bin Bashar Polock Gaurab Chhetri Subasish Das Mamba AI4TS 40 0 0 22 Mar 2025
ATTENTION2D: Communication Efficient Distributed Self-Attention Mechanism Venmugil Elango 48 0 0 20 Mar 2025
PSA-MIL: A Probabilistic Spatial Attention-Based Multiple Instance Learning for Whole Slide Image Classification Sharon Peled Y. Maruvka Moti Freiman 38 0 0 20 Mar 2025
InhibiDistilbert: Knowledge Distillation for a ReLU and Addition-based Transformer Tony Zhang Rickard Brännvall 36 0 0 20 Mar 2025
The Power of Context: How Multimodality Improves Image Super-Resolution Kangfu Mei Hossein Talebi Mojtaba Ardakani Vishal M. Patel P. Milanfar M. Delbracio DiffM 77 1 0 18 Mar 2025
Is Discretization Fusion All You Need for Collaborative Perception? Kang Yang Tianci Bu L. Li Chunxu Li Y. Wang Deying Li 63 0 0 18 Mar 2025
Learning Shape-Independent Transformation via Spherical Representations for Category-Level Object Pose Estimation Huan Ren Wenfei Yang Xiang Liu Shifeng Zhang Tianzhu Zhang 67 2 0 18 Mar 2025
CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences Ziran Qin Yuchen Cao Mingbao Lin Wen Hu Shixuan Fan Ke Cheng Weiyao Lin Jianguo Li 69 3 0 16 Mar 2025
TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Models Xudong Tan Peng Ye Chongjun Tu Jianjian Cao Yaoxin Yang Lin Zhang Dongzhan Zhou Tao Chen VLM 56 0 0 13 Mar 2025
STEAD: Spatio-Temporal Efficient Anomaly Detection for Time and Compute Sensitive Applications Andrew Gao Jun Liu AI4TS 58 0 0 11 Mar 2025
MIRAM: Masked Image Reconstruction Across Multiple Scales for Breast Lesion Risk Prediction H. Q. Vo Pengyu Yuan Zheng Yin Kelvin K. Wong Chika F. Ezeana S. Ly Stephen T. C. Wong H. Nguyen 39 0 0 10 Mar 2025
Similarity-Guided Layer-Adaptive Vision Transformer for UAV Tracking Chaocan Xue Bineng Zhong Qihua Liang Yaozong Zheng Ning Li Yuanliang Xue Shuxiang Song 36 0 0 09 Mar 2025
SED2AM: Solving Multi-Trip Time-Dependent Vehicle Routing Problem using Deep Reinforcement Learning Arash Mozhdehi Y. Wang Sun Sun Xin Eric Wang AI4TS 58 0 0 06 Mar 2025
ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task Vittorio Pippi Matthieu Guillaumin S. Cascianelli Rita Cucchiara M. Jaritz Loris Bazzani 62 0 0 06 Mar 2025
Partial Convolution Meets Visual Attention Haiduo Huang Fuwei Yang D. Li Ji Liu Lu Tian Jinzhang Peng Pengju Ren E. Barsoum 3DH 136 0 0 05 Mar 2025
Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer Yujiao Yang Jing Lian Linhui Li MoE 77 0 0 04 Mar 2025
Walking the Web of Concept-Class Relationships in Incrementally Trained Interpretable Models Susmit Agrawal Deepika Vemuri S. Paul Vineeth N. Balasubramanian CLL 62 0 0 27 Feb 2025
Minds on the Move: Decoding Trajectory Prediction in Autonomous Driving with Cognitive Insights Haicheng Liao Chengyue Wang Kaiqun Zhu Yilong Ren Bolin Gao Shengbo Eben Li Chengzhong Xu Z. Li 59 2 0 27 Feb 2025
Self-Adjust Softmax Chuanyang Zheng Yihang Gao Guoxuan Chen Han Shi Jing Xiong Xiaozhe Ren Chao Huang Xin Jiang Z. Li Yu-Hu Li 38 0 0 25 Feb 2025
Attention Eclipse: Manipulating Attention to Bypass LLM Safety-Alignment Pedram Zaree Md Abdullah Al Mamun Quazi Mishkatul Alam Yue Dong Ihsen Alouani Nael B. Abu-Ghazaleh AAML 41 0 0 24 Feb 2025
Single-Channel EEG Tokenization Through Time-Frequency Modeling Jathurshan Pradeepkumar Xihao Piao Zheng Chen Jimeng Sun 40 1 0 22 Feb 2025
Compression Barriers for Autoregressive Transformers Themistoklis Haris Krzysztof Onak 37 1 0 21 Feb 2025
Surface Vision Mamba: Leveraging Bidirectional State Space Model for Efficient Spherical Manifold Representation Rongzhao He Weihao Zheng Leilei Zhao Ying Wang Dalin Zhu Dan Wu Bin Hu Mamba 89 0 0 21 Feb 2025