Linformer: Self-Attention with Linear Complexity

8 June 2020

Sinong Wang

Belinda Z. Li

Madian Khabsa

Han Fang

Hao Ma

ArXiv PDF HTML

Papers citing "Linformer: Self-Attention with Linear Complexity"

50 / 648 papers shown

Title
MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation Sihyun Yu Meera Hahn Dan Kondratyuk Jinwoo Shin Agrim Gupta José Lezama Irfan Essa David A. Ross Jonathan Huang DiffM VGen 72 0 0 18 Feb 2025
Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis Amir Hosein Fadaei M. Dehaqani 40 0 0 11 Feb 2025
ZETA: Leveraging Z-order Curves for Efficient Top-k Attention Qiuhao Zeng Jerry Huang Peng Lu Gezheng Xu Boxing Chen Charles X. Ling Boyu Wang 47 1 0 24 Jan 2025
5G LDPC Linear Transformer for Channel Decoding Mario Hernandez Fernando Pinero 31 0 0 23 Jan 2025
Parallel Sequence Modeling via Generalized Spatial Propagation Network Hongjun Wang Wonmin Byeon Jiarui Xu Jinwei Gu Ka Chun Cheung Xiaolong Wang Kai Han Jan Kautz Sifei Liu 114 0 0 21 Jan 2025
Episodic Memories Generation and Evaluation Benchmark for Large Language Models Alexis Huet Zied Ben-Houidi Dario Rossi LLMAG 54 0 0 21 Jan 2025
ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models Thibaut Thonet Jos Rozen Laurent Besacier RALM 132 2 0 20 Jan 2025
FutureDepth: Learning to Predict the Future Improves Video Depth Estimation R. Yasarla Manish Kumar Singh Hong Cai Yunxiao Shi Jisoo Jeong Yinhao Zhu Shizhong Han Risheek Garrepalli Fatih Porikli MDE 88 6 0 17 Jan 2025
MAMo: Leveraging Memory and Attention for Monocular Video Depth Estimation R. Yasarla H. Cai Jisoo Jeong Y. Shi Risheek Garrepalli Fatih Porikli MDE 63 16 0 17 Jan 2025
Hadamard Attention Recurrent Transformer: A Strong Baseline for Stereo Matching Transformer Ziyang Chen Yongjun Zhang Wenting Li Bingshu Wang Yabo Wu Yong Zhao C. L. P. Chen 38 0 0 02 Jan 2025
A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames Pinelopi Papalampidi Skanda Koppula Shreya Pathak Justin T Chiu Joseph Heyward Viorica Patraucean Jiajun Shen Antoine Miech Andrew Zisserman Aida Nematzdeh VLM 58 24 0 31 Dec 2024
VMamba: Visual State Space Model Yue Liu Yunjie Tian Yuzhong Zhao Hongtian Yu Lingxi Xie Yaowei Wang Qixiang Ye Jianbin Jiao Yunfan Liu Mamba 111 609 0 31 Dec 2024
Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking You Wu Yongxin Li Mengyuan Liu Xucheng Wang Xiangyang Yang Hengzhou Ye Dan Zeng Qijun Zhao Shuiwang Li 93 0 0 28 Dec 2024
V"Mean"ba: Visual State Space Models only need 1 hidden dimension Tien-Yu Chi Hung-Yueh Chiang Chi-Chih Chang N. Huang Kai-Chiang Wu 88 0 0 21 Dec 2024
ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition Seungdong Yoa Seungjun Lee Hyeseung Cho Bumsoo Kim Woohyung Lim ViT 67 0 0 21 Dec 2024
BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models Patrick Haller Jonas Golde A. Akbik 72 0 0 20 Dec 2024
LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning Yansheng Mao Jiaqi Li Fanxu Meng Jing Xiong Zilong Zheng Muhan Zhang LLMAG RALM 92 1 0 18 Dec 2024
Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models Elvis Nunez L. Zancato Benjamin Bowman Aditya Golatkar W. Xia Stefano Soatto 73 2 0 17 Dec 2024
Advances in Transformers for Robotic Applications: A Review Nikunj Sanghai Nik Bear Brown AI4CE 75 0 0 13 Dec 2024
A Decade of Deep Learning: A Survey on The Magnificent Seven Dilshod Azizov Muhammad Arslan Manzoor Velibor Bojkovic Yingxu Wang Z. Wang ... Liang Li Siwei Liu Yu Zhong Wei Liu Shangsong Liang OOD AI4TS MedIm 116 0 0 13 Dec 2024
CubeFormer: A Simple yet Effective Baseline for Lightweight Image Super-Resolution Jikai Wang Huan Zheng Jianbing Shen SupR 68 0 0 03 Dec 2024
Phaseformer: Phase-based Attention Mechanism for Underwater Image Restoration and Beyond MD Raqib Khan Anshul Negi Ashutosh Kulkarni Shruti S. Phutke Santosh Kumar Vipparthi Subrahmanyam Murala 70 1 0 02 Dec 2024
State-Space Large Audio Language Models Saurabhchand Bhati Yuan Gong Leonid Karlinsky Hilde Kuehne Rogerio Feris James Glass 92 0 0 24 Nov 2024
Best of Both Worlds: Advantages of Hybrid Graph Sequence Models Ali Behrouz Ali Parviz Mahdi Karami Clayton Sanford Bryan Perozzi Vahab Mirrokni 79 2 0 23 Nov 2024
Nd-BiMamba2: A Unified Bidirectional Architecture for Multi-Dimensional Data Processing Hao Liu Mamba AI4CE 77 1 0 22 Nov 2024
Financial Risk Assessment via Long-term Payment Behavior Sequence Folding Yiran Qiao Yateng Tang Xiang Ao Qi Yuan Ziming Liu Chen Shen Xuehao Zheng 72 0 0 22 Nov 2024
EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality Sanghyeok Lee Joonmyung Choi Hyunwoo J. Kim 110 3 0 22 Nov 2024
A Theory for Compressibility of Graph Transformers for Transductive Learning Hamed Shirzad Honghao Lin A. Velingker B. Venkatachalam David P. Woodruff Danica J. Sutherland 75 1 0 20 Nov 2024
MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers Ning Ding Yehui Tang Haochen Qin Zhenli Zhou Chao Xu Lin Li Kai Han Heng Liao Yunhe Wang 59 0 0 20 Nov 2024
ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization Weibo Zhao Yubin Shi Xinyu Lyu Wanchen Sui Shen Li Yong Li MQ 47 1 0 12 Nov 2024
AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation Anil Kag Huseyin Coskun Jierun Chen Junli Cao Willi Menapace Aliaksandr Siarohin Sergey Tulyakov Jian Ren 46 3 0 07 Nov 2024
A Mamba Foundation Model for Time Series Forecasting Haoyu Ma Yushu Chen Wenlai Zhao Jinzhe Yang Yingsheng Ji Xinghua Xu Xiaozhu Liu Hao Jing Shengzhuo Liu Guangwen Yang AI4TS Mamba 41 2 0 05 Nov 2024
The Evolution of RWKV: Advancements in Efficient Language Modeling Akul Datta VLM 36 1 0 05 Nov 2024
TransUNext: towards a more advanced U-shaped framework for automatic vessel segmentation in the fundus image Xiang Li Mingsi Liu Lixin Duan ViT MedIm 29 0 0 05 Nov 2024
Context Parallelism for Scalable Million-Token Inference Amy Yang Jingyi Yang Aya Ibrahim Xinfeng Xie Bangsheng Tang Grigory Sizov Jeremy Reizenstein Jongsoo Park Jianyu Huang MoE LRM 60 5 0 04 Nov 2024
NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMs Nursena Köprücü Destiny Okpekpe Antonio Orvieto Mamba 33 1 0 31 Oct 2024
RAM: Replace Attention with MLP for Efficient Multivariate Time Series Forecasting Suhan Guo Jiahong Deng Yi Wei Hui Dou F. Shen Jian Zhao AI4TS 109 0 0 31 Oct 2024
Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learning Aosong Feng Rex Ying Leandros Tassiulas 27 2 0 28 Oct 2024
PODTILE: Facilitating Podcast Episode Browsing with Auto-generated Chapters Azin Ghazimatin Ekaterina Garmash Gustavo Penha Kristen Sheets Martin Achenbach ... Ben Carterette Ann Clifton Paul N. Bennett C. Hauff M. Lalmas 29 2 0 21 Oct 2024
Taming Mambas for Voxel Level 3D Medical Image Segmentation Luca Lumetti Vittorio Pipoli Kevin Marchesini Elisa Ficarra C. Grana Federico Bolelli MedIm Mamba 29 0 0 20 Oct 2024
Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Pingyi Chen Zhongyi Shui Chenglu Zhu Lin Yang MedIm 32 4 0 18 Oct 2024
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 114 0 0 17 Oct 2024
ChuLo: Chunk-Level Key Information Representation for Long Document Processing Yan Li Soyeon Caren Han Yue Dai Feiqi Cao 23 0 0 14 Oct 2024
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers Enze Xie Junsong Chen Junyu Chen Han Cai Haotian Tang ... Zhekai Zhang Muyang Li Ligeng Zhu Y. Lu Song Han VLM 31 49 0 14 Oct 2024
SLaNC: Static LayerNorm Calibration Mahsa Salmani Nikita Trukhanov I. Soloveychik MQ 26 0 0 14 Oct 2024
Towards Better Multi-head Attention via Channel-wise Sample Permutation Shen Yuan Hongteng Xu 17 1 0 14 Oct 2024
Arrhythmia Classification Using Graph Neural Networks Based on Correlation Matrix Seungwoo Han 18 0 0 14 Oct 2024
DeBiFormer: Vision Transformer with Deformable Agent Bi-level Routing Attention Nguyen Huu Bao Long Chenyu Zhang Yuzhi Shi Tsubasa Hirakawa Takayoshi Yamashita Tohgoroh Matsui H. Fujiyoshi 26 2 0 11 Oct 2024
A Framework to Enable Algorithmic Design Choice Exploration in DNNs Timothy L. Cronin IV Sanmukh Kuppannagari 43 0 0 10 Oct 2024
SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation Hang Yin Xiuwei Xu Zhenyu Wu Jie Zhou Jiwen Lu 29 13 0 10 Oct 2024