FNet: Mixing Tokens with Fourier Transforms

9 May 2021

James Lee-Thorp

Joshua Ainslie

Ilya Eckstein

Santiago Ontanon

ArXiv PDF HTML

Papers citing "FNet: Mixing Tokens with Fourier Transforms"

50 / 251 papers shown

Title
ExpNet: A unified network for Expert-Level Classification Junde Wu Huihui Fang Yehui Yang Yu Zhang Haoyi Xiong Huazhu Fu Yanwu Xu 27 0 0 29 Nov 2022
DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attention Bosheng Qin Juncheng Li Siliang Tang Yueting Zhuang 25 2 0 24 Nov 2022
Convexifying Transformers: Improving optimization and understanding of transformer networks Tolga Ergen Behnam Neyshabur Harsh Mehta MLT 44 15 0 20 Nov 2022
Efficient Transformers with Dynamic Token Pooling Piotr Nawrot J. Chorowski Adrian Lañcucki E. Ponti 20 42 0 17 Nov 2022
An FNet based Auto Encoder for Long Sequence News Story Generation Paul K. Mandal Rakeshkumar V. Mahto 24 0 0 15 Nov 2022
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers Michael Hassid Hao Peng Daniel Rotem Jungo Kasai Ivan Montero Noah A. Smith Roy Schwartz 32 24 0 07 Nov 2022
QNet: A Quantum-native Sequence Encoder Architecture Wei-Yen Day Hao-Sheng Chen Min Sun 21 0 0 31 Oct 2022
Contextual Learning in Fourier Complex Field for VHR Remote Sensing Images Yan Zhang Xiyuan Gao Qingyan Duan Jiaxu Leng Xiao Pu Xinbo Gao ViT 16 1 0 28 Oct 2022
Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data Aditya Dutt Alina Zare P. Gader 16 7 0 25 Oct 2022
MetaFormer Baselines for Vision Weihao Yu Chenyang Si Pan Zhou Mi Luo Yichen Zhou Jiashi Feng Shuicheng Yan Xinchao Wang MoE 40 156 0 24 Oct 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 43 9 0 14 Oct 2022
Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities Brian Bartoldson B. Kailkhura Davis W. Blalock 31 47 0 13 Oct 2022
FCT-GAN: Enhancing Table Synthesis via Fourier Transform Zilong Zhao Robert Birke L. Chen 27 7 0 12 Oct 2022
Are All Vision Models Created Equal? A Study of the Open-Loop to Closed-Loop Causality Gap Mathias Lechner Ramin Hasani Alexander Amini Tsun-Hsuan Wang T. Henzinger Daniela Rus CML OOD 26 7 0 09 Oct 2022
Digital Asset Valuation: A Study on Domain Names, Email Addresses, and NFTs Kai Sun 11 2 0 06 Oct 2022
WavSpA: Wavelet Space Attention for Boosting Transformers' Long Sequence Learning Ability Yufan Zhuang Zihan Wang Fangbo Tao Jingbo Shang ViT AI4TS 35 3 0 05 Oct 2022
Liquid Structural State-Space Models Ramin Hasani Mathias Lechner Tsun-Hsuan Wang Makram Chahine Alexander Amini Daniela Rus AI4TS 107 96 0 26 Sep 2022
Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers Nurullah Sevim Ege Ozan Özyedek Furkan Şahinuç Aykut Koç 35 11 0 26 Sep 2022
Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design Hongxiang Fan Thomas C. P. Chau Stylianos I. Venieris Royson Lee Alexandros Kouris Wayne Luk Nicholas D. Lane Mohamed S. Abdelfattah 37 57 0 20 Sep 2022
Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition D. Kothandaraman Ming-Shun Lin Tianyi Zhou 25 6 0 15 Sep 2022
Public Wisdom Matters! Discourse-Aware Hyperbolic Fourier Co-Attention for Social-Text Classification Karish Grover S. Angara Md. Shad Akhtar Tanmoy Chakraborty 16 12 0 15 Sep 2022
Pre-Training a Graph Recurrent Network for Language Representation Yile Wang Linyi Yang Zhiyang Teng M. Zhou Yue Zhang GNN 38 1 0 08 Sep 2022
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 30 109 0 31 Aug 2022
MRL: Learning to Mix with Attention and Convolutions Shlok Mohta Hisahiro Suganuma Yoshiki Tanaka 28 2 0 30 Aug 2022
Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning J. Hu Roberto Cavicchioli Alessandro Capotondi 29 21 0 13 Aug 2022
Multi Resolution Analysis (MRA) for Approximate Self-Attention Zhanpeng Zeng Sourav Pal Jeffery Kline G. Fung Vikas Singh 23 6 0 21 Jul 2022
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers Runsheng Xu Zhengzhong Tu Hao Xiang Wei Shao Bolei Zhou Jiaqi Ma 56 218 0 05 Jul 2022
LordNet: An Efficient Neural Network for Learning to Solve Parametric Partial Differential Equations without Simulated Data Xinquan Huang Wenlei Shi Xiaotian Gao Xinran Wei Jia Zhang Jiang Bian Mao Yang Tie-Yan Liu PINN 27 10 0 19 Jun 2022
WaveMix: A Resource-efficient Neural Network for Image Analysis Pranav Jeevan Kavitha Viswanathan S. AnanduA A. Sethi 20 20 0 28 May 2022
What Dense Graph Do You Need for Self-Attention? Yuxing Wang Chu-Tak Lee Qipeng Guo Zhangyue Yin Yunhua Zhou Xuanjing Huang Xipeng Qiu GNN 8 4 0 27 May 2022
EmoInHindi: A Multi-label Emotion and Intensity Annotated Dataset in Hindi for Emotion Recognition in Dialogues G. Singh Priyanshu Priya Mauajama Firdaus Asif Ekbal P. Bhattacharyya 14 12 0 27 May 2022
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT James Lee-Thorp Joshua Ainslie MoE 32 11 0 24 May 2022
FlexiBERT: Are Current Transformer Architectures too Homogeneous and Rigid? Shikhar Tuli Bhishma Dedhia Shreshth Tuli N. Jha 26 14 0 23 May 2022
FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting Tian Zhou Ziqing Ma Xue Wang Qingsong Wen Liang Sun Tao Yao Wotao Yin Rong Jin AI4TS 121 170 0 18 May 2022
Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers Arda Sahiner Tolga Ergen Batu Mehmet Ozturkler John M. Pauly Morteza Mardani Mert Pilanci 40 33 0 17 May 2022
FPSRS: A Fusion Approach for Paper Submission Recommendation System Son Huynh Nhi Dang Dac H. Nguyen Phong Huynh Binh T. Nguyen 11 2 0 12 May 2022
EigenNoise: A Contrastive Prior to Warm-Start Representations H. Heidenreich Jake Williams 13 1 0 09 May 2022
MAD: Self-Supervised Masked Anomaly Detection Task for Multivariate Time Series Yiwei Fu Feng Xue AI4TS 26 15 0 04 May 2022
To Know by the Company Words Keep and What Else Lies in the Vicinity Jake Williams H. Heidenreich 16 0 0 30 Apr 2022
Linear Complexity Randomized Self-attention Mechanism Lin Zheng Chong-Jun Wang Lingpeng Kong 22 31 0 10 Apr 2022
Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustness Giulio Lovisotto Nicole Finnie Mauricio Muñoz Chaithanya Kumar Mummadi J. H. Metzen AAML ViT 30 32 0 25 Mar 2022
Can we integrate spatial verification methods into neural-network loss functions for atmospheric science? Ryan Lagerquist I. Ebert‐Uphoff 46 11 0 21 Mar 2022
FAR: Fourier Aerial Video Recognition D. Kothandaraman Tianrui Guan Xijun Wang Sean Hu Ming-Shun Lin Tianyi Zhou 21 13 0 21 Mar 2022
Efficient Language Modeling with Sparse all-MLP Ping Yu Mikel Artetxe Myle Ott Sam Shleifer Hongyu Gong Ves Stoyanov Xian Li MoE 23 11 0 14 Mar 2022
IAE-Net: Integral Autoencoders for Discretization-Invariant Learning Yong Zheng Ong Zuowei Shen Haizhao Yang 18 15 0 10 Mar 2022
HyperMixer: An MLP-based Low Cost Alternative to Transformers Florian Mai Arnaud Pannatier Fabio Fehr Haolin Chen François Marelli F. Fleuret James Henderson 35 11 0 07 Mar 2022
WaveMix: Resource-efficient Token Mixing for Images Pranav Jeevan A. Sethi 17 10 0 07 Mar 2022
Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression A. B. Koyuncu Han Gao Atanas Boev Georgii Gaikov Elena Alshina Eckehard Steinbach ViT 39 68 0 04 Mar 2022
DCT-Former: Efficient Self-Attention with Discrete Cosine Transform Carmelo Scribano Giorgia Franchini M. Prato Marko Bertogna 18 21 0 02 Mar 2022
MLP-ASR: Sequence-length agnostic all-MLP architectures for speech recognition Jin Sakuma Tatsuya Komatsu Robin Scheibler 21 6 0 17 Feb 2022