Random Feature Attention

3 March 2021

Hao Peng

Lingpeng Kong

Papers citing "Random Feature Attention"

50 / 55 papers shown

Title
Conformal Transformations for Symmetric Power Transformers Saurabh Kumar Jacob Buckman Carles Gelada Sean Zhang 65 0 0 05 Mar 2025
Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis Amir Hosein Fadaei M. Dehaqani 40 0 0 11 Feb 2025
Parallel Sequence Modeling via Generalized Spatial Propagation Network Hongjun Wang Wonmin Byeon Jiarui Xu Jinwei Gu Ka Chun Cheung Xiaolong Wang Kai Han Jan Kautz Sifei Liu 81 0 0 21 Jan 2025
Training Hybrid Neural Networks with Multimode Optical Nonlinearities Using Digital Twins Ilker Oguz Louis J. E. Suter J. Hsieh Mustafa Yildirim Niyazi Ulaş Dinç Christophe Moser D. Psaltis 48 2 0 14 Jan 2025
Key-value memory in the brain Samuel J. Gershman Ila Fiete Kazuki Irie 29 7 0 06 Jan 2025
Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass Tong Chen Hao Fang Patrick Xia Xiaodong Liu Benjamin Van Durme Luke Zettlemoyer Jianfeng Gao Hao Cheng KELM 40 2 0 08 Nov 2024
Estimating the Spectral Moments of the Kernel Integral Operator from Finite Sample Matrices Chanwoo Chun SueYeon Chung Daniel D. Lee 19 1 0 23 Oct 2024
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 66 0 0 17 Oct 2024
Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity Mutian He Philip N. Garner 80 0 0 09 Oct 2024
Optimal Kernel Quantile Learning with Random Features Caixing Wang Xingdong Feng 32 0 0 24 Aug 2024
Linear Attention Sequence Parallelism Weigao Sun Zhen Qin Dong Li Xuyang Shen Yu Qiao Yiran Zhong 68 2 0 03 Apr 2024
DiJiang: Efficient Large Language Models through Compact Kernelization Hanting Chen Zhicheng Liu Xutao Wang Yuchuan Tian Yunhe Wang VLM 24 5 0 29 Mar 2024
Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling Mahdi Karami Ali Ghodsi VLM 31 6 0 28 Feb 2024
Long-MIL: Scaling Long Contextual Multiple Instance Learning for Histopathology Whole Slide Image Analysis Honglin Li Yunlong Zhang Chenglu Zhu Jiatong Cai Sunyi Zheng Lin Yang VLM 25 4 0 21 Nov 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 24 15 0 28 Sep 2023
Spike-driven Transformer Man Yao Jiakui Hu Zhaokun Zhou Liuliang Yuan Yonghong Tian Boxing Xu Guoqi Li 21 109 0 04 Jul 2023
Improving Autoregressive NLP Tasks via Modular Linearized Attention Victor Agostinelli Lizhong Chen 17 1 0 17 Apr 2023
On Efficient Training of Large-Scale Deep Learning Models: A Literature Review Li Shen Yan Sun Zhiyuan Yu Liang Ding Xinmei Tian Dacheng Tao VLM 24 39 0 07 Apr 2023
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers K. Choromanski Shanda Li Valerii Likhosherstov Kumar Avinava Dubey Shengjie Luo Di He Yiming Yang Tamás Sarlós Thomas Weingarten Adrian Weller 17 8 0 03 Feb 2023
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 114 36 0 15 Dec 2022
Token Turing Machines Michael S. Ryoo K. Gopalakrishnan Kumara Kahatapitiya Ted Xiao Kanishka Rao Austin Stone Yao Lu Julian Ibarz Anurag Arnab 27 21 0 16 Nov 2022
Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation Botao Yu Peiling Lu Rui Wang Wei Hu Xu Tan Wei Ye Shikun Zhang Tao Qin Tie-Yan Liu MGen 16 54 0 19 Oct 2022
The Devil in Linear Transformer Zhen Qin Xiaodong Han Weixuan Sun Dongxu Li Lingpeng Kong Nick Barnes Yiran Zhong 29 69 0 19 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 39 9 0 14 Oct 2022
LARF: Two-level Attention-based Random Forests with a Mixture of Contamination Models A. Konstantinov Lev V. Utkin 25 0 0 11 Oct 2022
Adapting Pretrained Text-to-Text Models for Long Text Sequences Wenhan Xiong Anchit Gupta Shubham Toshniwal Yashar Mehdad Wen-tau Yih RALM VLM 49 30 0 21 Sep 2022
Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization T. Nguyen Richard G. Baraniuk Robert M. Kirby Stanley J. Osher Bao Wang 21 9 0 01 Aug 2022
Long Range Language Modeling via Gated State Spaces Harsh Mehta Ankit Gupta Ashok Cutkosky Behnam Neyshabur Mamba 26 229 0 27 Jun 2022
Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning Cheng Tan Zhangyang Gao Lirong Wu Yongjie Xu Jun-Xiong Xia Siyuan Li Stan Z. Li 25 102 0 24 Jun 2022
SimA: Simple Softmax-free Attention for Vision Transformers Soroush Abbasi Koohpayegani Hamed Pirsiavash 14 24 0 17 Jun 2022
Leveraging Locality in Abstractive Text Summarization Yixin Liu Ansong Ni Linyong Nan Budhaditya Deb Chenguang Zhu Ahmed Hassan Awadallah Dragomir R. Radev 21 18 0 25 May 2022
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation Ta-Chung Chi Ting-Han Fan Peter J. Ramadge Alexander I. Rudnicky 39 65 0 20 May 2022
Twist Decoding: Diverse Generators Guide Each Other Jungo Kasai Keisuke Sakaguchi Ronan Le Bras Hao Peng Ximing Lu Dragomir R. Radev Yejin Choi Noah A. Smith SyDa 19 4 0 19 May 2022
Attention Mechanism in Neural Networks: Where it Comes and Where it Goes Derya Soydaner 3DV 28 149 0 27 Apr 2022
Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention Tong Yu Ruslan Khalitov Lei Cheng Zhirong Yang MoE 16 10 0 22 Apr 2022
A Call for Clarity in Beam Search: How It Works and When It Stops Jungo Kasai Keisuke Sakaguchi Ronan Le Bras Dragomir R. Radev Yejin Choi Noah A. Smith 19 6 0 11 Apr 2022
Block-Recurrent Transformers DeLesley S. Hutchins Imanol Schlag Yuhuai Wu Ethan Dyer Behnam Neyshabur 13 94 0 11 Mar 2022
cosFormer: Rethinking Softmax in Attention Zhen Qin Weixuan Sun Huicai Deng Dongxu Li Yunshen Wei Baohong Lv Junjie Yan Lingpeng Kong Yiran Zhong 21 210 0 17 Feb 2022
General-purpose, long-context autoregressive modeling with Perceiver AR Curtis Hawthorne Andrew Jaegle Cătălina Cangea Sebastian Borgeaud C. Nash ... Hannah R. Sheahan Neil Zeghidour Jean-Baptiste Alayrac João Carreira Jesse Engel 30 65 0 15 Feb 2022
Low-Rank Constraints for Fast Inference in Structured Models Justin T. Chiu Yuntian Deng Alexander M. Rush BDL 11 13 0 08 Jan 2022
Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networks Lei Cheng Ruslan Khalitov Tong Yu Zhirong Yang 20 32 0 06 Jan 2022
Token Pooling in Vision Transformers D. Marin Jen-Hao Rick Chang Anurag Ranjan Anish K. Prabhu Mohammad Rastegari Oncel Tuzel ViT 65 65 0 08 Oct 2021
Ripple Attention for Visual Perception with Sub-quadratic Complexity Lin Zheng Huijie Pan Lingpeng Kong 19 3 0 06 Oct 2021
Spike2Vec: An Efficient and Scalable Embedding Approach for COVID-19 Spike Sequences Sarwan Ali M. Patterson 13 46 0 12 Sep 2021
Pre-Trained Models: Past, Present and Future Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu ... Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao Jun Zhu AIFin MQ AI4MH 19 807 0 14 Jun 2021
Scalable Vision Transformers with Hierarchical Pooling Zizheng Pan Bohan Zhuang Jing Liu Haoyu He Jianfei Cai ViT 17 126 0 19 Mar 2021
CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation J. Clark Dan Garrette Iulia Turc John Wieting 11 210 0 11 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 48 970 0 04 Mar 2021
Formal Language Theory Meets Modern NLP William Merrill AI4CE NAI 14 12 0 19 Feb 2021
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 57 1,097 0 14 Sep 2020