Learning a Fourier Transform for Linear Relative Positional Encodings in
Transformers

Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers

3 February 2023

Valerii Likhosherstov

Kumar Avinava Dubey

Thomas Weingarten

Papers citing "Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers"

17 / 17 papers shown

Title
LOOPE: Learnable Optimal Patch Order in Positional Embeddings for Vision Transformers M. Chowdhury Md Rifat Ur Rahman Akil Ahmad Taki 22 0 0 19 Apr 2025
Structure-informed Positional Encoding for Music Generation Manvi Agarwal Changhong Wang Gaël Richard 19 2 0 20 Feb 2024
Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physics Siqi Miao Zhiyuan Lu Mia Liu Javier Duarte Pan Li 29 4 0 19 Feb 2024
SeTformer is What You Need for Vision and Language Pourya Shamsolmoali Masoumeh Zareapoor Eric Granger Michael Felsberg 20 4 0 07 Jan 2024
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers Tobias Christian Nauen Sebastián M. Palacio Federico Raue Andreas Dengel 35 3 0 18 Aug 2023
Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigation Xuesu Xiao Tingnan Zhang K. Choromanski Edward J. Lee Anthony G. Francis ... Leila Takayama Roy Frostig Jie Tan Carolina Parada Vikas Sindhwani 63 54 0 22 Sep 2022
Your Transformer May Not be as Powerful as You Expect Shengjie Luo Shanda Li Shuxin Zheng Tie-Yan Liu Liwei Wang Di He 52 50 0 26 May 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 303 11,730 0 04 Mar 2022
On Learning the Transformer Kernel Sankalan Pal Chowdhury Adamos Solomou Kumar Avinava Dubey Mrinmaya Sachan ViT 31 14 0 15 Oct 2021
Is Attention Better Than Matrix Decomposition? Zhengyang Geng Meng-Hao Guo Hongxu Chen Xia Li Ke Wei Zhouchen Lin 51 134 0 09 Sep 2021
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 242 690 0 27 Aug 2021
Deep Autoregressive Models with Spectral Attention Fernando Moreno-Pino Pablo Martínez Olmos Antonio Artés-Rodríguez AI4TS 21 17 0 13 Jul 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 253 4,735 0 24 Feb 2021
LambdaNetworks: Modeling Long-Range Interactions Without Attention Irwan Bello 260 178 0 17 Feb 2021
The Open Catalyst 2020 (OC20) Dataset and Community Challenges L. Chanussot Abhishek Das Siddharth Goyal Thibaut Lavril Muhammed Shuaibi ... Brandon M. Wood Junwoong Yoon Devi Parikh C. L. Zitnick Zachary W. Ulissi 218 498 0 20 Oct 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 249 1,982 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 234 578 0 12 Mar 2020