FNet: Mixing Tokens with Fourier Transforms

9 May 2021

James Lee-Thorp

Joshua Ainslie

Ilya Eckstein

Santiago Ontanon

ArXiv PDF HTML

Papers citing "FNet: Mixing Tokens with Fourier Transforms"

50 / 251 papers shown

Title
ConvFormer: Revisiting Transformer for Sequential User Modeling Hao Wang Jianxun Lian Mingyang Wu Haoxuan Li Jiajun Fan Wanyue Xu Chaozhuo Li Xing Xie 19 3 0 05 Aug 2023
DRAW: Defending Camera-shooted RAW against Image Manipulation Xiaoxiao Hu Qichao Ying Zhenxing Qian Sheng Li Xinpeng Zhang AAML 42 7 0 31 Jul 2023
Improving BERT with Hybrid Pooling Network and Drop Mask Qian Chen Wen Wang Qinglin Zhang Chong Deng Ma Yukun Siqi Zheng 17 0 0 14 Jul 2023
FITS: Modeling Time Series with $10k$ Parameters Zhijian Xu Ailing Zeng Qiang Xu AI4TS 22 84 0 06 Jul 2023
Fourier-Mixed Window Attention: Accelerating Informer for Long Sequence Time-Series Forecasting Nhat Tran Jack Xin AI4TS 31 6 0 02 Jul 2023
WaveMixSR: A Resource-efficient Neural Network for Image Super-resolution Pranav Jeevan Akella Srinidhi Pasunuri Prathiba A. Sethi SupR 31 9 0 01 Jul 2023
MultiWave: Multiresolution Deep Architectures through Wavelet Decomposition for Multivariate Time Series Prediction I. Deznabi M. Fiterau AI4TS 35 5 0 16 Jun 2023
Recasting Self-Attention with Holographic Reduced Representations Mohammad Mahmudul Alam Edward Raff Stella Biderman Tim Oates James Holt 8 8 0 31 May 2023
A Quantitative Review on Language Model Efficiency Research Meng Jiang Hy Dang Lingbo Tong 25 0 0 28 May 2023
A Joint Time-frequency Domain Transformer for Multivariate Time Series Forecasting Yushu Chen Shengzhuo Liu Jinzhe Yang Hao Jing Wenlai Zhao Guang-Wu Yang AI4TS 24 15 0 24 May 2023
Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator Ziwei He Meng-Da Yang Minwei Feng Jingcheng Yin Xinbing Wang Jingwen Leng Zhouhan Lin ViT 35 13 0 24 May 2023
Recyclable Tuning for Continual Pre-training Yujia Qin Cheng Qian Xu Han Yankai Lin Huadong Wang Ruobing Xie Zhiyuan Liu Maosong Sun Jie Zhou CLL 26 11 0 15 May 2023
ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmaps Yanfang Li Huan Wang Muxia Sun LM&MA AI4TS AI4CE 29 46 0 10 May 2023
Toeplitz Neural Network for Sequence Modeling Zhen Qin Xiaodong Han Weixuan Sun Bowen He Dong Li Dongxu Li Yuchao Dai Lingpeng Kong Yiran Zhong AI4TS ViT 32 40 0 08 May 2023
Deep Learning-Based Multiband Signal Fusion for 3-D SAR Super-Resolution Josiah W. Smith Murat Torlak 29 3 0 03 May 2023
Sequence Modeling with Multiresolution Convolutional Memory Jiaxin Shi Ke Alexander Wang E. Fox 42 13 0 02 May 2023
Unlimiformer: Long-Range Transformers with Unlimited Length Input Amanda Bertsch Uri Alon Graham Neubig Matthew R. Gormley RALM 99 122 0 02 May 2023
SpectFormer: Frequency and Attention is what you need in a Vision Transformer Badri N. Patro Vinay P. Namboodiri Vijay Srinivas Agneeswaran ViT 35 47 0 13 Apr 2023
Cascaded Local Implicit Transformer for Arbitrary-Scale Super-Resolution Haoming Chen Yu-Syuan Xu Minui Hong Yi-Min Tsai Hsien-Kai Kuo Chun-Yi Lee OffRL 39 46 0 29 Mar 2023
TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference Shikhar Tuli N. Jha 38 5 0 27 Mar 2023
EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms Shikhar Tuli N. Jha 36 3 0 24 Mar 2023
Multiscale Attention via Wavelet Neural Operators for Vision Transformers Anahita Nekoozadeh M. Ahmadzadeh Zahra Mardani ViT 38 2 0 22 Mar 2023
TemporalMaxer: Maximize Temporal Context with only Max Pooling for Temporal Action Localization Tuan N. Tang Kwonyoung Kim Kwanghoon Sohn 24 29 0 16 Mar 2023
Resurrecting Recurrent Neural Networks for Long Sequences Antonio Orvieto Samuel L. Smith Albert Gu Anushan Fernando Çağlar Gülçehre Razvan Pascanu Soham De 88 268 0 11 Mar 2023
FFT-based Dynamic Token Mixer for Vision Yuki Tatsunami Masato Taki 45 20 0 07 Mar 2023
MITFAS: Mutual Information based Temporal Feature Alignment and Sampling for Aerial Video Action Recognition Ruiqi Xian Xijun Wang Tianyi Zhou 24 10 0 05 Mar 2023
AZTR: Aerial Video Action Recognition with Auto Zoom and Temporal Reasoning Xijun Wang Ruiqi Xian Tianrui Guan Celso M. de Melo Stephen M. Nogar Aniket Bera Tianyi Zhou 16 11 0 02 Mar 2023
Diagonal State Space Augmented Transformers for Speech Recognition G. Saon Ankit Gupta Xiaodong Cui AI4TS 35 26 0 27 Feb 2023
Magnification Invariant Medical Image Analysis: A Comparison of Convolutional Networks, Vision Transformers, and Token Mixers Pranav Jeevan Nikhil Cherian Kurian A. Sethi MedIm 32 1 0 22 Feb 2023
A Federated Approach for Hate Speech Detection Jay Gala Deep Gandhi Jash Mehta Zeerak Talat 21 4 0 18 Feb 2023
Efficiency 360: Efficient Vision Transformers Badri N. Patro Vijay Srinivas Agneeswaran 26 6 0 16 Feb 2023
Simple Hardware-Efficient Long Convolutions for Sequence Modeling Daniel Y. Fu Elliot L. Epstein Eric N. D. Nguyen A. Thomas Michael Zhang Tri Dao Atri Rudra Christopher Ré 16 52 0 13 Feb 2023
Fourier-RNNs for Modelling Noisy Physics Data Vignesh Gopakumar Stanislas Pamela L. Zanisi AI4TS 18 2 0 13 Feb 2023
A Unified View of Long-Sequence Models towards Modeling Million-Scale Dependencies Hongyu Hè Marko Kabić 25 2 0 13 Feb 2023
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment Jared Fernandez Jacob Kahn Clara Na Yonatan Bisk Emma Strubell FedML 33 10 0 13 Feb 2023
DNArch: Learning Convolutional Neural Architectures by Backpropagation David W. Romero Neil Zeghidour AI4CE 24 4 0 10 Feb 2023
MTS-Mixers: Multivariate Time Series Forecasting via Factorized Temporal and Channel Mixing Zhe Li Zhongwen Rao Lujia Pan Zenglin Xu AI4TS 33 60 0 09 Feb 2023
Efficient Joint Learning for Clinical Named Entity Recognition and Relation Extraction Using Fourier Networks: A Use Case in Adverse Drug Events A. Yazdani D. Proios H. Rouhizadeh Douglas Teodoro 21 7 0 08 Feb 2023
Scaling Up Computer Vision Neural Networks Using Fast Fourier Transform Siddharth Agrawal 15 0 0 02 Feb 2023
Out-of-distributional risk bounds for neural operators with applications to the Helmholtz equation Jose Antonio Lara Benitez Takashi Furuya F. Faucher Anastasis Kratsios X. Tricoche Maarten V. de Hoop 39 16 0 27 Jan 2023
WLD-Reg: A Data-dependent Within-layer Diversity Regularizer Firas Laakom Jenni Raitoharju Alexandros Iosifidis Moncef Gabbouj AI4CE 29 7 0 03 Jan 2023
Cramming: Training a Language Model on a Single GPU in One Day Jonas Geiping Tom Goldstein MoE 30 85 0 28 Dec 2022
Pretraining Without Attention Junxiong Wang J. Yan Albert Gu Alexander M. Rush 27 48 0 20 Dec 2022
First De-Trend then Attend: Rethinking Attention for Time-Series Forecasting Xiyuan Zhang Xiaoyong Jin Karthick Gopalswamy Gaurav Gupta Youngsuk Park Xingjian Shi Hongya Wang Danielle C. Maddix Yuyang Wang AI4TS 27 19 0 15 Dec 2022
Efficient Long Sequence Modeling via State Space Augmented Transformer Simiao Zuo Xiaodong Liu Jian Jiao Denis Xavier Charles Eren Manavoglu Tuo Zhao Jianfeng Gao 125 36 0 15 Dec 2022
Paraphrase Identification with Deep Learning: A Review of Datasets and Methods Chao Zhou Cheng Qiu Daniel Ernesto Acuna 32 25 0 13 Dec 2022
TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities Zhe Zhao Yudong Li Cheng-An Hou Jing-xin Zhao Rong Tian ... Xingwu Sun Zhanhui Kang Xiaoyong Du Linlin Shen Kimmo Yan VLM 41 23 0 13 Dec 2022
BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch? Joel Niklaus Daniele Giofré 33 11 0 30 Nov 2022
Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles Shuquan Ye Yujia Xie Dongdong Chen Yichong Xu Lu Yuan Chenguang Zhu Jing Liao VLM 24 11 0 29 Nov 2022
Lightweight Structure-Aware Attention for Visual Understanding Heeseung Kwon F. M. Castro M. Marín-Jiménez N. Guil Alahari Karteek 28 2 0 29 Nov 2022